Rustdoc integration

[jyn514]

It would be great to add deadlinks to rustdoc itself! That would increase discoverability a ton, and I expect it would uncover a bunch more bugs in rustdoc too. It would also allow getting rid of the linkchecker in the rust repo.

As a bonus, rustdoc does know how it generates the links, so this might allow fixing #14 (comment) at the same time.

• Add support for ignored links (Allow ignoring specific broken links #133)
• Clean up the API and publish 1.0
• Add deadlinks to rustdoc
  • (Optional) Add proper error messages to tie the dead link in the HTML to the original page in the source

[jyn514]

Hmm, unfortunately some of the licenses in lol_html's dependencies are not MIT or Apache:

tidy error: invalid license `MPL-2.0` in `cssparser 0.25.9 (registry+https://github.com/rust-lang/crates.io-index)`
tidy error: invalid license `MPL-2.0` in `cssparser-macros 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)`
tidy error: invalid license `MPL-2.0` in `dtoa-short 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)`
tidy error: dependency `fuchsia-cprng 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)` does not define a license expression
tidy error: invalid license `BSD-3-Clause` in `lol_html 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)`
tidy error: invalid license `ISC` in `rdrand 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)`
tidy error: invalid license `MPL-2.0` in `selectors 0.21.0 (registry+https://github.com/rust-lang/crates.io-index)`
tidy error: invalid license `MPL-2.0` in `thin-slice 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)`

[jyn514]

Zulip thread: https://rust-lang.zulipchat.com/#narrow/stream/266220-rustdoc/topic/Is.20there.20an.20approved.20HTML.20parser.3F

[jyn514]

WIP implementation: https://github.com/jyn514/rust/tree/linkcheck

[jyn514]

Note that is blocked on rust-lang/rust#80527, but I expect figuring out the dependencies will take long enough that will be merged before I have to worry about it.

[jyn514]

I guess as a last resort I could make deadlinks generic over the parser and slot in https://github.com/rust-lang/rust/blob/94e6ea9fc97845b336067bffb3d103780474f6f5/src/tools/linkchecker/main.rs#L406 somehow, but that really seems like a hack, parsing HTML with regex is never a good idea.

[jyn514]

Oh I'm being silly - html5ever meets all the licenses. So I can make deadlinks generic over the parser and slot in HTML5ever for rust-lang/rust.

[jyn514]

Finally got a response about licensing, other tools already use the licenses so it shouldn't be a problem: https://rust-lang.zulipchat.com/#narrow/stream/266220-rustdoc/topic/Is.20there.20an.20approved.20HTML.20parser.3F/near/312418703

Mark did suggest that this doesn't necessarily need to be part of rustdoc, and that a cargo fix lint to migrate from manual links to intra-doc links might be more helpful. I'm going to try and make a crater run with https://github.com/rust-lang/rust/compare/master...jyn514:rust:linkcheck?expand=1 to see how useful it will be in practice, but I'm probably inclined to agree.