Run rustdoc --check in CI to prevent broken intra-doc links

[sharnoff]

I recently discovered that rustdoc has a --check flag to avoid generating documentation, and just produce warnings/errors. We currently have a number of broken internal doc links, so (after fixing them) it'd be great to automatically prevent more from being added.

Problems

Overall, running this right now might be a bit hacky:

• There's no cargo doc --check command (see: Feature request: cargo check for documentation rust-lang/cargo#10025, Add --check for rustdoc rust-lang/cargo#8859)
• rustdoc --check requires nightly (see: Rustdoc check option rust-lang/rust#78984)
• cargo rustdoc ... can only be run for one crate at a time (not too bad; we can just loop through all of them)

Sample command

Currently, the following correctly runs, but requires nightly:

cargo +nightly rustdoc -p pageserver --lib -- -Z unstable-options --check

Notes:

• -Z unstable-options is required for --check.
• -p pageserver is required because rustdoc expects only a single crate
• --lib is required because pageserver has both lib and bin targets, and rustdoc needs just one
• (not shown) We have to also have to either pass --deny rustdoc::all or --deny rustdoc::$lint (for each lint) in order to cause rustdoc to exit with a non-zero status code

In general, requiring an extra nightly version in CI might be too much. It's "just" checking the code for dependencies + target crate, so not nearly as much as a full compilation -- it takes ~1.4 minutes on my machine (8 core/16 thread) to run the command above. It's then ~1 second for running on an additional crate (e.g., safekeeper).

In terms of possible nightly issues: I think we can mostly avoid stability issues by pinning to a particular version. We're also not compiling anything, so there's a lot of potential instability avoided there as well. It also might be possible to do something extraordinarily hacky and use nightly features with our stable tools (e.g, as in https://fasterthanli.me/articles/why-is-my-rust-build-so-slow#how-much-time-are-we-spending-on-these-steps). The really hacky version looks like:

RUSTC_BOOTSTRAP=1 cargo rustdoc -p pageserver --lib -- -Z unstable-options --check

It performs an initial run much faster (17.5s).

[arpad-m]

I have filed #4711 to fix the warnings generated by rustdoc. As explained in that PR, running rustdoc currently requires nightly due to dependencies, but I think after update of opentelemetry, it should be possible to run cargo doc in CI without a --check param and then just not do anything with the result.