User agent

[nomoregrapes]

What user agent will this crawler report as?

Please can there be a setting to specify information as to who is running the crawler, and put this in the useragent. This way, website owners can contact those responsible if they are being excessively crawled.

[mrwilson]

It looks from the code like the user-agent is being set as "AdsTxtCrawler/1.0; +https://github.com/InteractiveAdvertisingBureau/adstxtcrawler"

(https://github.com/InteractiveAdvertisingBureau/adstxtcrawler/blob/master/adstxt_crawler.py#L119)

But there's currently no feature to support customising the user-agent.

[nealrichter]

This is trivial to change in the code itself. One would expect any scaled ads company to do it...