catch link redirections

[bgruening]

No description provided.

[nsoranzo]

Maybe that's a bit too forgiving, I'll see if we can solve the issue with nature.com without allowing other broken redirects.

[nsoranzo]

requests doesn't have the issue seen in https://travis-ci.org/galaxyproject/tools-iuc/jobs/371740724 , maybe we can replace this function with something like the following (not tested):

     def validate_url(url, lint_ctx, user_agent=None):
        is_valid = True
        if user_agent:
            headers = {"User-Agent": user_agent, 'Accept': '*/*'}
        else:
            headers = None
        r = None
        try:
            r = requests.get(url, headers=headers, stream=True)
            r.raise_for_status()
            next(r.iter_content(1000))
        except Exception as e:
            if r and r.status_code == 429:
                # too many requests
                pass
            else:
                is_valid = False
                lint_ctx.error("Error '%s' accessing %s" % (e, url))
        if is_valid:
            lint_ctx.info("URL OK %s" % url)

[nsoranzo]

My above code works, BUT requests is HTTP-only, so FTP URLs fail with something like:

Applying linter tool_urls... FAIL
.. ERROR: Error 'No connection adapters were found for 'ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz'' accessing ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz

[bgruening]

Jupp, thats why I ended up with the more relaxed mode. Maybe the more relaxed one is ok, if we turn this into a warning?

[nsoranzo]

I think using requests for HTTP and urllib2 for FTP would be the best of both worlds, requests is so much better than Python standard library. I've implemented this in #809 .