New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn the link checker script into a web service #1

Open
seanh opened this Issue Oct 2, 2014 · 2 comments

Comments

Projects
None yet
1 participant
@seanh
Copy link
Contributor

seanh commented Oct 2, 2014

Current plan is to us Flask and APScheduler.

If it were an actual web app instead of just an API script the link checker could provide its own API for the site to use to request ad-hoc link checks in response to user interaction. Something like:

/check_these_links accepts a list of objects: ``[ {"id" : "...", "url": "..."}, ... ]`. The service will check the links and post the results back to the site as usual.

Authentication

The ad-hoc mode requires auth. I like @wardi's suggestion that the web service is configured with a list of site's that it works for, each time the service receives a request for ad-hoc link checks it contacts the site that the request claims to be from and asks if whether it made this request, before proceeding.

This means the link checker service doesn't need to handle authentication, it can just fall back on the site's to do it.

@seanh

This comment has been minimized.

Copy link
Contributor

seanh commented Oct 2, 2014

Link checker                                                     Client site
                 <----------------- Please check these links: -
                                         [<id_1>, <id_2> ...]
                 - 200 OK ------------------------------------>

                 - What's the URL for resource <id_1>? ------->
                 <---- The URL for resource <id_1> is <url_1> -  (Gets URL from db)
(Checks <url_1>) - Resource <id_1>'s URL is broken ----------->  (Saves result)

                 - What's the URL for resource <id_2>? ------->
                 <---- The URL for resource <id_2> is <url_2> -  (Gets URL from db)
(Checks <url_2>) - Resource <id_2>'s URL is broken ----------->  (Saves result)
                                ...
@seanh

This comment has been minimized.

Copy link
Contributor

seanh commented Oct 2, 2014

Use cases:

  1. Buttons the CKAN web UI: "Check this resource / dataset / organization / site for broken links now"
  2. Check a resource's link straight away when the user creates a new resource or changes a resource's URL

Note that once the link checker is a web service it will still be doing regular hourly link checks 90% of the time, the two ad-hoc use-cases above are the 10% nice-to-have cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment