Set a canonical URL on all documentation pages that points to the latest docs

[daveverwer]

I was sure we had an issue open for having documentation pages missing from the Google index, but I can't find it.

After a rather confusing investigation with Google Search Console, I believe our missing documentation pages are being skipped during indexing or excluded from the Google index because we have multiple versions of every documentation set hosted and available.

We should set a canonical URL on every documentation page and point to the latest stable/beta/branch release using the same logic as we have for where the “Documentation” button links to.

[daveverwer]

Found the "issue" in Discussions:

• Add documentation pages to the Sitemap #2078 (comment)
• Add documentation pages to the Sitemap #2078 (comment)
• Add documentation pages to the Sitemap #2078 (comment)

[daveverwer]

OK so the first attempt at a fix (#2360) is live for this and results are being tracked in this spreadsheet:

https://docs.google.com/spreadsheets/d/1PZDxI4jRJh3Mc2qCQmVq4Il26HslAzWheffZF3FGruU/edit?usp=sharing

Tracking five packages at their root and at a specific documentation page and I'll check in every Tuesday to see when they all hopefully turn green 🤞

[daveverwer]

This is not working very well. Google has picked up the canonical URLs on the root of each documentation version but is still selecting the URL without the version in as the canonical (which means that it doesn't index anything as that "canonical" link it's selecting is just a redirect.

We have two possible options next:

1. The simple option, change the documentation URL on the package page to be the canonical and de-emphasise that redirect.
2. Proper sitemaps for documentation content, as discussed in Add documentation pages to the Sitemap #2078.

[tayloraswift]

google prefers stable URLs when selecting canonicals. by definition "latest release" cannot possibly be stable because there will always be new releases.

this is a very hard problem.