Caching of redirected robots.txt may overwrite correct robots.txt rules

[sebastian-nagel]

Redirected robots.txt rules should be only cached for the target host if the URL path is /robots.txt otherwise the redirect may overwrite the correct robots rules, see NUTCH-2581.

[jnioche]

In the example you gave in NUTCH-2581, the target URL ends in /robots.txt so I'm not sure I understand the problem.

[sebastian-nagel]

It must be exactly /robots.txt, not /wyomingtheband/robots.txt or anything else. The problem is that the current logic assumes that the content returned for https://www.facebook.com/wyomingtheband/robots.txt is equivalent to that of https://www.facebook.com/robots.txt which is definitely not the case. Regarding the example: this redirect and many more mask the correct robots.txt and cause that even frequently the host www.facebook.com is considered to be allowed for crawling. Of course, there is some random what is fetched and cached first.

[jnioche]

Ok, got it, thanks Sebastian. Will fix it right now.

[sebastian-nagel]

Thanks, @jnioche!