-
Notifications
You must be signed in to change notification settings - Fork 25.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
next.angular.io is being indexed by google in spite of robots.txt disallowing it #21313
Comments
it seems that only the root url next.angular.io is being indexed. nothing else is: https://www.google.com/search?q=angular+site%3Anext.angular.io Maybe this is the expected behavior? |
I think I know what's going on. This note found in the noindex meta tag docs explains it:
We do block next.angular.io via robots.txt, so the robot does know that the url exists, but it's not allowed to index it - that's why there is no search snippet under the result ("No information is available for this page"). A better solution would be to remove the robots.txt restriction and instead use |
You mean remove the |
@gkalpak yes |
This does not yet appear resolved, although the search results for next.angular.io now has the Angular Elements page instead of the Language Service page. |
resolved - no longer a problem! closing! |
This issue has been automatically locked due to inactivity. Read more about our automatic conversation locking policy. This action has been performed automatically by a bot. |
example query: https://www.google.com/search?q=language+service+site%3Aangular.io
current result:
note: this issue was discovered as part of the investigation into #21272
The text was updated successfully, but these errors were encountered: