Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Set a User-Agent for HTTP requests from :elinks #1417
Proposed User-Agent string:
In my own set of 1800 links, this change reduces the number of HTTP 403 Forbidden responses from 482 to 17. (Most of these are probably hosted by a single large provider like Cloudflare.) This could probably be reduced even further by faking a mainstream web browser’s User-Agent string. However, I’d rather live with a handful of blocked tests than start faking User-Agents.
Ooh, I like it. I think the User Agent that you propose is good, and I suppose it can be changed in the future if needed.
Could be useful to have a test for this, but probably not important. (I also realised that I haven’t really set up proper testing for the external links checker yet anyway…)