It looks like angular.io may no longer be indexing

[StephenFluin]

I'm submitting a...

[X] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead see https://github.com/angular/angular/blob/master/CONTRIBUTING.md#question

Current behavior

Angular.io pages are not included or previewed in the google search index.

https://www.google.com/search?q=Angular+Language+Service+site%3Aangular.io&oq=Angular+Language+Service+site%3Aangular.io

Expected behavior

Angular.io pages are included and previewed in the google search index.

Minimal reproduction of the problem with instructions

Visit https://www.google.com/search?q=Angular+Language+Service+site%3Aangular.io

I've also noted this on several other sites, so it may be something on the Angular side of things.

https://www.google.com/search?q=site%3Afluin.io

Environment

Angular version: 5.x.x

[IgorMinar]

I checked webmaster tools and here is what I see:

On the surface there doesn't seem to be any problem - the number of documents hosted on angular.io that are in the index is about the same as before.

When I however try to fetch the language service doc as the google bot, the rendering fails with error:
[DocViewer]: Error preparing document 'guide/language-service'. and empty document viewer:

There is no more info about what went wrong.

It's as if the request to fetch the json doc failed, but in that case, we usually render a "network error" page, but that is not happening here. I tried several other urls and most of them suffer from this issue (which would explain the low quality search results).

Example urls that don't render in Google bot:

• https://angular.io/guide/language-service
• https://angular.io/guide/security
• https://angular.io/guide/architecture
• https://angular.io/api/animations/group
• https://angular.io/guide/quickstart

Example urls that do render in Google bot:

• https://angular.io
• https://angular.io/events
• https://angular.io/api
• https://angular.io/resources
• https://angular.io/docs
• https://angular.io/features
• https://angular.io/about?group=Angular

The only thing that pages that do vs do not work have in common is the number of path segments in the url. Single path segment seems to work, multiple don't work. I don't know if this is just a coincidence or not.

Just to be clear: all of these urls were indexable in the past, so something in aio or in google bot must have changed to cause this regression.

@StephenFluin's repro also surfaced two related issues that we should also address:

• next.angular.io is being indexed, when it shouldn't be (similarly v2.angular.io and v4.angular.io should not be indexed)
  • this is weird because we did setup robots.txt for next and v4 domains:
    • https://next.angular.io/robots.txt
    • https://v4.angular.io/robots.txt
  • v2.angular.io is missing robots.txt: https://v2.angular.io/robots.txt, we should fix that.
• the 404 pages for some old urls for the previous angular.io site are being indexed as regular documents (we should setup redirects for these + ensure that bogus urls that render as 404 do not end up in the index)

[IgorMinar]

I was thinking that maybe the change that introduced lazy-loaded components could be responsible for this, but /api page indexes just fine and it does use a lazy-loaded component as far as I know.

[arjunyel]

Would potentially using Rendertron be a good fix for the Googlebot? https://angularfirebase.com/lessons/seo-angular-part-1-rendertron-meta-tags/

[IgorMinar]

@arjunyel angular.io is our proving ground for ensuring that Angular apps work well out of the box, so we want to avoid setting up custom infrastructure to make SEO work. But thanks for the suggestion.

[IgorMinar]

Once we fix this we should change our error logging setup to send errors to some external storage when google bot is crawling the site. General instructions can be found here: https://developers.google.com/search/docs/guides/debug-rendering

[IgorMinar]

ok, @gkalpak's PR is now in and we have more info about the problem. Here is what I got:

[DocViewer] Error preparing document 'guide/quickstart': 
TypeError: Cannot read property 'toLowerCase' of undefined at 
https://angular.io/0.f2591f718907b6ae5a8f.chunk.js:1:196 at Array.forEach (native) at l 
(https://angular.io/0.f2591f718907b6ae5a8f.chunk.js:1:158) at r 
(https://angular.io/0.f2591f718907b6ae5a8f.chunk.js:1:484) at new t
 (https://angular.io/0.f2591f718907b6ae5a8f.chunk.js:1:31662) at Ke 
(https://angular.io/main.bec31ad514fe5bf71717.bundle.js:1:83560) at qe
 (https://angular.io/main.bec31ad514fe5bf71717.bundle.js:1:82013) at In 
(https://angular.io/main.bec31ad514fe5bf71717.bundle.js:1:96674) at xn 
(https://angular.io/main.bec31ad514fe5bf71717.bundle.js:1:95462) at Object.Gn [as createRootView] 
(https://angular.io/main.bec31ad514fe5bf71717.bundle.js:1:102128)

[IgorMinar]

This stacktrace maps back to:

angular/aio/src/app/shared/attribute-utils.ts

Lines 6 to 17 in cdc66f6

	/**
	* Get attribute map from element or ElementRef `attributes`.
	* Attribute map keys are forced lowercase for case-insensitive lookup.
	* @param el The source of the attributes
	*/
	export function getAttrs(el: HTMLElement | ElementRef): StringMap {
	const attrs: NamedNodeMap = el instanceof ElementRef ? el.nativeElement.attributes : el.attributes;
	const attrMap = {};
	Object.keys(attrs).forEach(key =>
	attrMap[(attrs[key] as Attr).name.toLowerCase()] = (attrs[key] as Attr).value);
	return attrMap;
	}

Specifically line:

angular/aio/src/app/shared/attribute-utils.ts

Line 15 in cdc66f6

attrMap[(attrs[key] as Attr).name.toLowerCase()] = (attrs[key] as Attr).value);

[IgorMinar]

And this path gets invoked when we try to instantiate an embedded component with attributes like:

angular/aio/content/guide/security.md

Lines 101 to 103 in cdc66f6

	<code-example path="security/src/app/inner-html-binding.component.html" title="src/app/inner-html-binding.component.html">
	</code-example>

Which explains why certain pages work and others don't. Only those that don't have embedded components with attributes work and this are typically the top level pages like most marketing pages.