rustdoc: add doc_link_canonical feature

[lolbinarycat]

proposing this as a t-rustdoc experiment (do we have a process for those?), implementation was surprisingly simple so it's basically a complete implementation of my design (although ideally it would be integrated into cargo in the future using --crate-attr), will likely need FCP then RFC for stabilization, may also need perf testing to make sure added allocations have a negligible impact, not sure.

fixes #143139

r? @GuillaumeGomez

Problem Statement

There are two main usecases for this feature:

1. crates that wish to have their canonical documentation site be somewhere other than docs.rs.
2. crates that want to have docs.rs be their canonical documentation site, but their crate docs are frequently included in doc bundles hosted by a third party.

In both these cases, this can cause search engines to duplicate results or show the wrong (non-canonical) page in results.

Solution

A new crate-level doc attribute is added, html_link_canonical, which adds a link rel="canonical" element to the head of every documentation page (pages such as help and settings are excluded due to not belonging to any crate in particular), leveraging the existing system search engines use for de-duplicating equivalent pages.

Future Work

1. When html_link_canonical is used with no value, it will use the value of html_root_url, or --extern-html-root-url, if present.
2. cargo flag that implies -Zrustdoc-map and passes --crate-attr='doc(html_link_canonical)' to each rustdoc invocation. This will cause all crates that do not manually specify html_root_url or html_link_canonical to use the docs.rs page as the canonical page. Alternatively, instead of implying -Zrustdoc-map, it could simply reuse its logic, passing the value that would be passed to --extern-html-root-url to be passed via --crate-attr='doc(html_link_canonical="...")'.

[rustbot]

Some changes occurred in compiler/rustc_passes/src/check_attr.rs

cc @jdonszelmann

[lolbinarycat]

The one feature that isn't currently implemented is having the default value be the same as from html_root_url if you just specify doc(html_link_canonical) with no value.

[GuillaumeGomez]

Took a look at the implementation, it seems correct. However, I'm not sure it solves it the correct way, would need to actually completely understand the issue (don't have time right now but didn't want to leave the PR pending with no feedback).

So for experimental features, we generally go through an FCP. Now, to make a PR able to go through FCP, the first comment needs to contain a lot more information: describe exactly the problem it tries to solve, how it solved it and why it solved it this way.

[lolbinarycat]

@GuillaumeGomez did my best to describe the experiment, let me know if there's anything I can give more details on or write better.