ng add @angular/seo

[IgorMinar]

Bug Report or Feature Request (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request

Area

- [ ] devkit
- [x] schematics

Versions

v6

Desired functionality

I'd like to be able to invoke ng add @angular/seo and have the CLI make my application search engine friendly. Initially I'd like to see the following feature provided by the schematics:

• generate robots.txt with no rules but a simple instructions/link on how to setup blacklist rules
• generate sitemap.xml with similar documentation/links as in robots.txt
• add code that would feature detect and conditionally load polyfills needed by the crawlers
• setup error logging via global error handler with an option to redirect the output to a remote server
• provide facilities to easily add and remove the noindex metadata to emulate the HTTP 404 and 500 status codes

Mention any other details that might be useful

[hansl]

Why not make this as part of the ng new experience directly?

provide facilities to easily add and remove the noindex metadata to emulate the HTTP 404 and 500 status codes

That implies a router, no? We don't have a router by default (maybe we should).

[clydin]

There are classes of applications that do not need/want to worry about SEO: internal applications, applications exclusively behind a login, etc. For those cases the files and logic could manually be removed, I suppose.

However, some of those items (e.g., conditional polyfills) are things that could be elevated beyond SEO though.

[IgorMinar]

I think we need to experiment with this stuff more before we roll it out to everyone. This would be a safe way to iterate and find a the right solution for most apps without breaking things.

…

On Wed, Apr 18, 2018, 10:47 AM clydin ***@***.***> wrote: There are classes of applications that do not need/want to worry about SEO: internal applications, applications exclusively behind a login, etc. For those cases the files and logic could manually be removed, I suppose. However, some of those items (e.g., conditional polyfills) are things that could be elevated beyond SEO though. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#729 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANM6IGYVfpeP7yBHWm1MYviQ0wdnBHVks5tp24SgaJpZM4TZyPX> .

[tobi-or-not-tobi]

If it's only a set of relatively static files that can be generated initially, it wouldn't matter to much whether its part of ng new or done in a schematics, besides the note of @IgorMinar. But I'd imagine that the real value goes into some dynamically data that will change over time, i.e. updates to the sitemap based on the router. That would require a repeatable activity and would be used while building and maintaining the app, and then initial creation only wouldn't be enough.

[tobi-or-not-tobi]

Another aspect is that part of the generated files might need to be extended by users based on dynamic data, which are not known in the app. This is a separate flow and we should be careful do allow for such manual additions.
We could generate a sitemap index file during the initial creation of the app (i.e. ng new --sitemap) and build specific sitemaps while the app grows (i..e ng add @angular/sitemap-routes). The later would generate specific links which are simply not available initially, and write them in a separate sitemap.xml files.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex>
  <sitemap>
    <loc>http://www.we-need-a-domain.com/static-routes.xml</loc>
    [...]
  </sitemap>
<!-- developers could add specific sitemaps to the sitemapindex file. 
  <sitemap>
    <loc>http://www.we-need-a-domain.com/products.xml</loc>
    [...]
  </sitemap>
</sitemapindex>

Developers can create other manual sub sitemap files and include them into the main sitemap (which should not be generated again).

[donoso-eth]

Great Idea, as i built my site, it would had helped me, if at least, I would have known the which basics I have to take care.
My take on sitemap.xml:
The Sitemap is very dynamic (last-mod, frequency and prority) for each URL, therefore for sitemaps we could ask what is worth to offer to the devellopper?

1. Create an empty sitemap.xml with xml structure
2. Based upon a json config and a string[] with routes create the sitemap.xml
   config= {
   "host":"www.mywebsite.com"
   "frecuency":"weekfly",
   .......
   }
3. Following the idea from @tobi-or-not-tobi of @schematics/routes we could develop a small crawler that on request craw all internal "routerLink" attributes and create a list of "urls", which merged with a config file will create the string[] with the urls.

Probably the trade off effort/results is to go for the number 2.
The number 3 is the interesting one, for ssr I can imagine how could it be done, for client side rendering I dont´t have a clue so far....

[CharlBest]

I think this is a really cool idea and didn't know about any of this so adding this with one command potentially with some granularity/options would be awesome!

[kylecordes]

It seems to me that in the general case, for a dynamic application, sitemap.xml would be generated by server-side code that accesses the same data as the Angular application.

Therefore - perhaps the sitemap idea fits in well with Universal, but not with client-only Angular?

Universal is becoming more popular; perhaps angular/seo ought depend on /universal.

[tobi-or-not-tobi]

@kylecordes agreed that this would make a lot of sense in universal, but as @IgorMinar pointed out at ngconf, he's in favor to see a solution in non-universal driven websites. Ideally we try to achieve the same for both in a generic way.

[hiepxanh]

I believe SEO is the most important think. More than speed, comfort, simple code, etc....
Without SEO, my best beautiful fastest ever website is just nothing, it just like a business without marketing...
I think we have two problems:

• Problem 1: Developer should create sitemap.xml and robots.txt but he should create an automation way to automatic update that file. ( when?)
• Problem 2: How we can update that without touching that file? That is the client side problem.

[hiepxanh]

About the price he said. I think IgorMinar is right. More money, more SEO... It is a truth and it is acceptable. Assume that your website is dynamic content (just like news, or blog or something with auto generated content).
About current situation:

1. People need push more money and effort to make angular SEO able.

• money: every website with dynamic content need a server. ( but, as Loki said "we have firebase functions" 😄)
• effort: every website with dynamic content need few setup. (our case is universal).
  if they choose that. how we can do it painlessly?

2. if using client only. I heard that they have Rendertron but I think we can provide something better. or just integrate Rendertron to universal?

[BenjaminHofstetter]

A great idea.... and maybe some kind of support for JSON lD (https://json-ld.org/)

[donoso-eth]

@IgorMinar diving into the angular.io index.html to see which best practies are you using.
I came across following script // Dynamically, pre-emptively, addnoindex, which will be removed when the doc is ready and valid tag = document.createElement('meta'); tag.name = 'robots'; tag.content = 'noindex'; document.head.appendChild(tag); which as it is written add a noindex tag whih will be removed once the app is bootstrapped.
My question is then, for search bots other than googlebot, is this tactic not penalyzing the index as these bots will only see the tag noindex?
Thanks for your awesome job and contribution

[alexeagle]

This issue was moved to angular/angular-cli#12120