angular.io missing from google results for certain users/devices

[IgorMinar]

I was able to repro this only on my mobile phone but @StephenFluin and @bradlygreen reported seeing it on their desktop too:

[IgorMinar]

I've checked crawler stats and they look fine as far as I can tell. I also tried fetching the site as robot and it succeeded:

[IgorMinar]

I think we'll just monitor it for a bit first while watching analytics / crawler stats to understand the impact. It's unclear to me what could be responsible for this.

The only somewhat relevant changes we made recently were: #21665 and #21689.

There there are #21712 #21763 but those didn't make it to prod yet.

[IgorMinar]

I also got a notification that we now have new search console available for our site, but that seems unrelated.

[IgorMinar]

The index status is looking fine:

but we should note that the data processing for this graph appears to have latency of several days, so the graph ends with status as of Tuesday (3 days ago).

[IgorMinar]

EDIT: the data in this comment is wrong - I was looking at angularjs.org - I think the data is incomplete yet and that's why we see less traffic on that site. Ignore this comment. I'm leaving it here for archival purposes.

Sessions are 4-20% down for today (ignore the sharp drop - we don't have complete data for today yet)

sessions were already down by 4% down at 6am:

But the number of users is down sharply - 20-60% for today:

Started around 7pm PT yesterday:

[softplus]

It looks like these pages have a Googlebot noindex meta tag on them -- this prevents Google from indexing the page. Using JavaScript to remove limitations like these is not recommended, and can result in those pages being dropped from indexing before they're rendered.

[IgorMinar]

Thanks @softplus. In that case the culprit PR is #21665. I find it however very odd that the search console didn't warn us about this.

[petebacondarwin]

So we should remove the pre-emptive hard coded tag?

[IgorMinar]

@petebacondarwin yeah. I have a PR: #21816 out for review

[IgorMinar]

The fix in #21816 is now in production.

[petebacondarwin]

@IgorMinar - Is this now resolved?