Fake User-Agent if response is 403?

[claudep]

I have some sites that return a 403 response for both HEAD and GET requests when the User-Agent is not in some whitelisted strings. Here's an example: https://www.cairn.info/revue-l-economie-politique-2005-3-page-60.htm Its probably a measure to avoid some bot traffic.

If you set the User-Agent as looking like a browser (e.g. 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0'), it returns a 200 status code.

Would it be acceptable (in the ethical sense) to try with a "fake" user agent in the case of a 403 response?

[claudep]

https://www.un.org/ is another example.

[timobrembeck]

Would it be acceptable (in the ethical sense) to try with a "fake" user agent in the case of a 403 response?

In my opinion, this would be justified. Since we try to minimize traffic by checking links only once no matter how often they occur in the content and by only checking URLs once every EXTERNAL_RECHECK_INTERVAL, this cannot be considered malicious behavior. In comparison to the website's regular visitors and all the traffic caused by enumeration tools (which usually also fake their user agent), our checks will probably not put excessive load on the servers.
Also, (at least in our use case) it's in the best interest of the website operators to be considered a valid link, otherwise our clients might remove the link from their content.

[claudep]

Thanks, I appreciate your point of view. It makes much sense.