Status: Googlebot blocked by robots.txt

[the-s-a-m]

The following resources can not be loaded by googlebot because they are blocked by default by robots.txt

/system/assets/jquery/jquery-2.x.min.js
/user/plugins/form/assets/form-styles.css
/user/plugins/login/css/login.css

Thank you for fixing this error in the default configuration

[NicoHood]

The current robots is:

User-agent: *
Disallow: /backup/
Disallow: /bin/
Disallow: /cache/
Disallow: /grav/
Disallow: /logs/
Disallow: /system/
Disallow: /vendor/
Disallow: /user/
Allow: /user/pages/
Allow: /user/themes/
Allow: /user/images/
Allow: /

Maybe it would make sense to add Allow: /user/plugins/ to the list?
Why is it important to load the listed resources? Does google even care about javascript and css?

[the-s-a-m]

Google told me that the access is not possible therefore I changed it to

User-agent: *
Allow: /user/pages/
Allow: /user/themes/
Allow: /user/images/
Allow: /user/plugins/
Allow: /system/assets/jquery/jquery-2.x.min.js
Disallow: /backup/
Disallow: /bin/
Disallow: /cache/
Disallow: /grav/
Disallow: /logs/
Disallow: /system/
Disallow: /vendor/
Disallow: /user/
Allow: /

and now it is working

[rhukster]

Fixed by bc22c8d

[NicoHood]

It seems that the issue is not fixed for the builtin jquery:
/system/assets/jquery/jquery-2.x.min.js

The reason why this happens is unclear to me. It might be that Allow: *.js$ does not include script.min.js syntax (at least from my observations, as other .js scripts work. Searching around in google i found this post that suggests the following:

User-Agent: Googlebot
Allow: .css
Allow: .js

This also allows query parameters that are often used for cache busting: https://example.com/deep/style.css?something=1. And yes, User-Agent: Googlebot is important to use here! Another option is to explicitly allow the jquery path.

[NicoHood]

It turns out google is having random problems loading some assets. I am not yet sure if this is related to grav or not - I assume not. However the builtin jquery must be added with an additional rule, as Disallow: /system/ blocks it (i've tested that with the google robots.txt tool).

Allow: *.css$
Allow: *.js$
Allow: /system/*.js$

[rhukster]

That's pretty odd, as the allow's are coming after the disallows. But if it works in your testing, them i'm good with it. Thanks.

[NicoHood]

I've read, that the length of the rule matters for the googlebot. That makes totally sense why Allow: /system/*.js$ is required.

I also found out, why the other resources we unable to load: Google stops loading resources, if there are too many. It seems like a long standing crawling bug, I could work around this by enabling the assets manager. See https://support.google.com/webmasters/thread/4425254?hl=en

PR: #3129

[rhukster]

I had aready commited this to 1.7 branch: 14df5a6