next.angular.io is being indexed by google in spite of robots.txt disallowing it

[IgorMinar]

example query: https://www.google.com/search?q=language+service+site%3Aangular.io

current result:

note: this issue was discovered as part of the investigation into #21272

[IgorMinar]

it seems that only the root url next.angular.io is being indexed. nothing else is: https://www.google.com/search?q=angular+site%3Anext.angular.io

Maybe this is the expected behavior?

[IgorMinar]

I think I know what's going on. This note found in the noindex meta tag docs explains it:

Important! For the noindex meta tag to be effective, the page must not be blocked by a robots.txt file. If the page is blocked by a robots.txt file, the crawler will never see the noindex tag, and the page can still appear in search results, for example if other pages link to it.

We do block next.angular.io via robots.txt, so the robot does know that the url exists, but it's not allowed to index it - that's why there is no search snippet under the result ("No information is available for this page").

A better solution would be to remove the robots.txt restriction and instead use noindex metatag on all pages. This solution is very much compatible with the 2. solution proposed to resolve the 404 pages being indexed issue.

[gkalpak]

You mean remove the noindex metatag on angular.io, but leave it on next.angular.io?

[IgorMinar]

@gkalpak yes

[petebacondarwin]

This does not yet appear resolved, although the search results for next.angular.io now has the Angular Elements page instead of the Language Service page.

[IgorMinar]

resolved - no longer a problem! closing!

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}