Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update robots.txt #2632

Merged
merged 1 commit into from Aug 18, 2019

Conversation

@dsofeir
Copy link
Contributor

commented Aug 18, 2019

I have found that Bing/Yahoo/DuckDuckGo, Yandex and Google report crawl errors when using the default robots.txt. Specifically their bots will not crawl the the path '/' or any sub-paths. I agree that the current robots.txt should work and properly implements the specification. However it still does not work.

In my experience explicitly permitting the path '/' by adding the directive Allow: / resolves the issue.

More details can be found in a blog post about the issue here: https://www.dfoley.ie/blog/starting-with-the-indieweb

Update robots.txt
I have found that Bing/Yahoo/DuckDuckGo, Yandex and Google report crawl errors when using the default robots.txt. Specifically their bots will not crawl the the path '/' or any sub-paths. I agree that the current robots.txt should work and properly implements the specification. However it still does not work.

In my experience explicitly permitting the path '/' by adding the directive Allow: / resolves the issue.

More details can be found in a blog post about the issue here: https://www.dfoley.ie/blog/starting-with-the-indieweb

@rhukster rhukster merged commit ed87faa into getgrav:develop Aug 18, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.