Should the external links check perform HEAD requests?

[dmathieu]

While I understand HEAD requests allow speeding up the checks, and the 405 HTTP response is handled, many web servers don't handle HEAD requests, and won't send a 405 either.

This basically makes the elinks check useless, as it can't be relied upon.

Steps to reproduce

1. Create a page with a link to https://redis.io/topics/protocol inside it.
2. Run nanoc check elinks

Expected behavior

This page does exist, and if users were to click the link, they wouldn't get a 404. The test should therefore not fail,

Actual behavior

Because the server doesn't handle HEAD requests, and doesn't return 405, the test fails, when the page actually exists.

Fixing this

I'm absolutely fine opening a PR fixing this, either by entirely removing the HEAD request, or any other solution proposed.
I want to make sure this kind of change would be approved though. And a quick search didn't show any other similar issue.

[denisdefreyne]

Makes sense!

An alternative could be to perform a HEAD request and fall back to GET if the HEAD returns a 404, but I don’t see the benefit of doing that over doing a GET in the first place.

I’ve opened PR #1179 to fix this.

[dmathieu]

Awesome!

If I may make another suggestion, how about setting an user agent?
Like nanoc/link_checker?

[denisdefreyne]

@dmathieu I don’t think that setting a custom user agent would bring much. Do you have a use case for it?

[dmathieu]

Not specifically. It's just a way to be a better citizen.
Thanks for the HEAD fix! 🌷