Lighthouse unable to download robots.txt

[Krinkle]

To Reproduce
Steps to reproduce the behavior:

1. Go to https://lighthouse-dot-webdotdevsite.appspot.com//lh/html?url=https://www.wikipedia.org/#seo

Describe the bug

I'm getting a lowered SEO score when measuring via https://web.dev, with the following issue reported:

SEO: 90
robots.txt is not valid: Lighthouse was unable to download a robots.txt file

However, https://www.wikipedia.org/robots.txt responds without issue.

Perhaps it is getting blocked by something in the middleware, e.g. an Inspector rule of some kind in the way that web.dev/PSI configure Lighthouse? See also #10198 which might be similar.

[exterkamp]

I can replicate this in DevTools and CLI. So not a PSI/LR bug at least.

Edit: At a cursory glance it might be because https://wikipedia.org/robots.txt is returning 301's?

I can audit https://en.wikipedia.org and the robots.txt is found. We don't follow redirect/moved chains in following robots.txt. Possible feature?

[connorjclark]

strange, redirect for fetch options is set to follow by default, so it should follow redirects.

https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch

[churchthecat]

I have the same issue: robots.txt is not valid Lighthouse was unable to download a robots.txt file
https://tcckonsult.com/robots.txt

[churchthecat]

Found the issue for me, CSP policy was set to Header set Content-Security-Policy "default-src 'none'; changing it to Header set Content-Security-Policy "default-src 'self'; fixed it.

[martinburger]

Found the issue for me, CSP policy was set to Header set Content-Security-Policy "default-src 'none'; changing it to Header set Content-Security-Policy "default-src 'self'; fixed it.

Mozilla Observatory suggests that you deny by default, using default-src 'none' instead of default-src 'self'. Thus, you might following the suggestion at this answer at StackOverflow and use connect-src 'self' instead.

That way, you ensure a high(er) score at Mozilla Observatory and you have a passable workaround.

BTW, this seems to be related with issue #4386.