Tracking issue for RFC 1946 - intra-rustdoc links

[nrc]

RFC: https://github.com/rust-lang/rfcs/blob/master/text/1946-intra-rustdoc-links.md
RFC PR: rust-lang/rfcs#1946

Todo:

• Land #47046
• Make it work with cross-crate imports
  • Blocked on #47046 (comment)
• Make it warn more on resolution errors (#49542)
• Make warnings/errors use better spans than the first line of docs
  • first step: attribute (#51111)
  • point to actual link in doc comment (#51391)
  • Add lint for link resolution failure (#51382)
• Implied Shortcut Reference Links (#48335)
  • blocked on #44229 and raphlinus/pulldown-cmark#121
• Enum variants and UFCS/associated methods (#47701)
• Primitives (#49459)
• Use the right resolution scopes for inner doc comments on things that aren't modules (#55364)
• Handle fields and variants (#49512)
  • Handle fields inside struct variants
• Support Self (#49583)
• Stabilization (current impl is gated to only run on nightly, but does not require a feature gate)

Other related issues:

• provide option to turn intra-link resolution errors into proper errors (#50082)

[QuietMisdreavus]

Preliminary implementation notes:

Getting just the links from the markdown is relatively straightforward. You'll need to do this twice, to capture the behavior in both the Hoedown and Pulldown markdown renderers. Thankfully, each has a way of capturing links as they are parsed from the document. (While capturing the link you can also look for the "struct/enum/mod/macro/static/etc" marker to aid in resolution down the line.)

In Hoedown, the render function in src/librustdoc/html/markdown.rs captures all the events as they are captured. Hoedown uses a system of registering callbacks on the renderer configuration to capture each event separately. The link event and its related linkfn function pointer signature are already present in the file, so you would need to create and use that callback to capture and modify links as they appear.

For Pulldown, the parser is surfaced as an Iterator of events. There are already a handful of iterator wrappers present in html/markdown.rs, so adding another one (and using it in the impl fmt::Display for Markdown farther down) shouldn't be a big deal. I'm... honestly not that familiar with how Pulldown represents link definitions in its Event/Tag enums, so more research would be needed to effectively get link text for that.

Now, with the links in hand, you need to do two things, neither of which i know offhand since they involve interfacing with the compiler:

• Attempt to parse the link as a path (if it's not a valid Rust path, just emit the URL as-is and skip the rest of this)
• Resolve the path in the "current" module (emit a warning if resolution fails)

Presumably there's a way to make the resolution come up with a DefId? Even after cleaning the AST, rustdoc still deals in those, so if that's the case there are ways to map that to the proper relative link.

[Manishearth]

I've got a WIP going on in https://github.com/manishearth/rust/tree/correct-resolver-fail , which works off of Misdreavus' WIP (they did most of the work).

[QuietMisdreavus]

Note that that branch is now live as #47046, which i'm polishing and finishing up.

[QuietMisdreavus]

Note for people watching this thread but not #47046: We're trying to wind that PR down to get a minimal version landed, so an initial implementation is fairly close.

[sgrif]

One thing that appears to not work in #47046 (and I don't think is ever explicitly addressed in the RFC) is how to link to primitives like u32 or bool.

[Manishearth]

Yeah I have to investigate that

[Manishearth]

@killercup @QuietMisdreavus should we turn the error into a warning ? I don't like the idea of crate docs breaking because someone introduced a name into a glob import

[killercup]

@Manishearth does rustdoc have some idea of "lint levels"? What are the current deprecation warnings? Just println!s?

[Manishearth]

We can just struct_span_warn instead. rustc can emit warnings not tied to actual warning names. There are no lint levels.

[killercup]

Alright, let's do that for now. In the end I'd really like to have a way to check the docs for errors but that's a more involved/fragile process I imagine. Manish Goregaokar <notifications@github.com> schrieb am Mi. 24. Jan. 2018 um 09:17:

…

We can just struct_span_warn instead. rustc can emit warnings not tied to actual warning names. There are no lint levels. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#43466 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABOXw-H_Wyi-zoTpfv_fqi-ovlRv5a9ks5tNucqgaJpZM4OiBhy> .

[Manishearth]

#47701

[Manishearth]

Shortcut links being implemented at #48335

[cramertj]

The issue @MajorBreakfast is hitting in #43466 (comment) is still happening and is a serious blocker for me-- even when building with --no-deps, I'm getting warnings and errors from upstream documentation resolution failures.

[cramertj]

Example log

[cramertj]

(am unblocked through the magic of RUSTDOC env var, shell scripts, and the --cap-lints setting, but still this should get fixed XD)

[QuietMisdreavus]

[dependencies]
futures-preview = "0.3.0-alpha.7"

#![deny(warnings)]

extern crate futures;

This is enough to trigger a failure right now. (I'll need to narrow it down, but using a crate with some specific use of intra-links will work.) I thought i'd fixed this kind of erroneous error with #53162 (comment) but it turns out it wasn't good enough. I'm building/testing a fix right now that refuses to run link collection on non-local items. It may be a bit drastic (and breaks docs which actually work in this situation) but it finally silence this kind of issue until we can actually support them.

[QuietMisdreavus]

Narrowed it down:

// inner crate

use std::fmt::Debug;

pub trait SomeTrait {}

/// this is an impl for [SomeTrait]
impl<T: Debug> SomeTrait for T {}

I bet the PR i linked doesn't touch this impl because it's generic, so it could apply to any number of local items.

[SergioBenitez]

In Rocket, we have two libraries in a workspace: a core library, rocket, and a contrib library rocket_contrib. rocket_contrib is not a dependency of rocket, but rocket refers to rocket_contrib in its documentation often.

Because rocket_contrib is not a dependency of rocket, cross-crate links are not resolving:

This is [rocket_contrib], a broken link.

Is there any workaround to this? I've tried adding --extern rocket_contrib=path/to.rmeta manually via RUSTDOCFLAGS, as well as using -Z unstable-options --extern-html-root-url rocket_contrib=https://rocket-contrib.docs, but neither work. Even if the latter worked, it isn't a great solution as it'll produce absolute links, partially defeating the purpose.

In general, I think cargo and/or rustdoc should automatically add all crates in the same workspace to the resolution domain for inter-crate links, making something like [rocket_contrb] work out of the box.

[killercup]

@SergioBenitez, have you tried adding a docs feature to the rocket lib that you set when building docs and that enables an optional dependency on rocket_contrib?

(Sorry to respond here without having followed this issue or impl much!)

[Nemo157]

automatically add all crates in the same workspace to the resolution domain for inter-crate links, making something like [rocket_contrb] work out of the box.

This will cause issues with the docs when built outside the workspace, e.g. when someone runs cargo doc -p rocket --open to get locally built docs in their project. Changing RUSTDOCFLAGS or having an optional feature has similar problems.

Earlier I suggested including dev-dependencies during doc builds to solve the same issue, or maybe there could be an additional doc-dependencies section?

[SergioBenitez]

@killercup I considered that, but unfortunately that would introduce a cyclic dependency.

@Nemo157 I instinctively tried that - placing the dependency in [dev-dependencies] - actually. Although I secretly hoped it didn't work as I really don't want to build/check contrib when working on core. A special doc-dependencies section would resolve this particular issue if it allowed cyclic dependencies.

[Nemo157]

Currently it's possible to link to fields of structs via a form of psuedo-path:

/// [`Foo::bar`]
pub struct Foo { pub bar: () }

Similarly it would be useful if it were possible to link to fields of a variant of an enum, probably via the same sort of psuedo-path:

/// [`Bar::Baz::quux`]
pub enum Bar {
    Baz { quux: () }
}

[dekellum]

I'm trying to ascertain the readiness of this feature for release use, and was hoping the tracking issue might help. While it is working great for me on recent nightly builds, on the just released stable rust 1.30.0 (da5f414 2018-10-24), I'm surprised to find that the feature still isn't complete: some links using rust paths aren't resolved to html links.

Is this feature intended to be nightly only? Or is there flags or feature's to be set in order to use it with the rust stable channel? Should I file a new issue for the missed links on 1.30?

[Manishearth]

There's a big hole in the current implementation, documentation for re-exported items doesn't work. Re-exports are used a lot in the ecosystem so this is a stabilization blocker. It's nontrivial to implement since it requires some architectural changes. Also, yeah, file issues for missing links, cc me.

…

On Thu, Oct 25, 2018, 6:12 PM David Kellum ***@***.***> wrote: I'm trying to ascertain the readiness of this feature for release use, and was hoping the tracking issue might help. While it is working great for me on recent nightly builds, on the just released stable rust 1.30.0 (da5f414 <da5f414> 2018-10-24), I'm surprised to find that the feature still isn't complete: some links using rust paths aren't resolved to html links. Is this feature intended to be nightly only? Or is there flags or feature's to be set in order to use it with the rust stable channel? Should I file a new issue for the missed links on 1.30? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#43466 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABivSBJ2j47jDYuOKpSrRxK79UEKGTQvks5uofEJgaJpZM4OiBhy> .

[wagenet]

@Manishearth I'm not privy to how Rust tends to handle these sorts of things, so I realize this might go against established practice, but it seems to me like it would be useful to have what currently exists in stable, but just not promoted as something that actually works. It's not dangerous and is useful in its current state. I guess the risk is that nobody ever finishes the support for re-exports and we're stuck with a half working feature?

[Manishearth]

The problem is more that the brokenness of re-exports is subtle. If folks start using this in documentation, they may not notice that it's broken in re-exports, which will end up showing ugly documentation. I'm not going to push back strongly though, if the docs team feels that early stabilization is okay that sounds good to me.

…

On Thu, Oct 25, 2018, 6:37 PM Peter Wagenet ***@***.***> wrote: @Manishearth <https://github.com/Manishearth> I'm not privy to how Rust tends to handle these sorts of things, so I realize this might go against established practice, but it seems to me like it would be useful to have what currently exists in stable, but just not promoted as something that actually works. It's not dangerous and is useful in its current state. I guess the risk is that nobody ever finishes the support for re-exports and we're stuck with a half working feature? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#43466 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABivSNWV6v8XoU8stua09siBxVMwoeRuks5uofbKgaJpZM4OiBhy> .

[wagenet]

@Manishearth I've found it super useful in its current form, but I can see the worry around things being subtly broken.

[dekellum]

I misinterpreted what I was initially seeing on 1.30: all, not just some, rust path style links (this feature) are not resolved on this latest (and prior) stable releases. Besides that not being explained anywhere I can find, the output is slightly confusing because the implicit type links are shown as obviously broken, e.g. "[SomeStruct]" where as links explicitly using a rust path are included verbatim, e.g. <a href="mod1::SomeStruct">, as would be expected without this feature being present.

As it seems now the intent for this was to be nightly-only until further along, shouldn't this tracking issue be edited to include the following, as it might save the next prospective user some confusion (@nrc)?

• Stabilization PR (current implementation is nightly only, but not feature gated)

I agree that this is value add as found on current nightly. Authors can just avoid using the new link style for re-exports, as well as links to other crates which are not dependencies, until these are resolved.

But as it stands, its a bit difficult to justify using the feature when it will not generate correct links for a user-local cargo doc even on the latest stable, not to mention the minimum stable rust version. Please consider if keeping this off for stable, with no option or flag, is now more likely to cause trouble with broken links (as this starts getting used with nightly) vs any regressions stabilizing might otherwise cause, given that the new style link targets would never have worked without the feature?

[killercup]

Random idea: If it is planned to keep this semi-usable* for a few more months, it might interesting to see if we can make nightly rustdoc optionally print out the generated relative links (with obvious issues when docs are rendered in more than one place). I don't think we have precendence for this, but if you someone wants to have a fun project, they might even try to integrate this with rustfix (i.e., yield 'suggestion-like' output where you appen \n/// [$a]: $b for each link).

* You can use it right now, with the obvious missing features mentioned above. Since docs.rs uses a nightly compiler version where the intra doc links work, a lot of people reading the docs for your crate will see the correct links. People trying to build the docs will be disappointed, though.

[QuietMisdreavus]

@dekellum You are correct in your observations: The feature is currently written so that links will only ever resolve when running on a nightly (or locally-compiled) rustdoc. In any other situation, they will be passed through in the way you observed.

The current implementation of intra-doc links is not only incomplete, as @Manishearth mentioned, but is also bug-ridden: There have long been issues of spurious "dead links" being found in crates which aren't using intra-doc links at all. This, combined with the fact that we emit a proper warning lint for these resolution errors, led to situations where innocent crates suddenly broke because they had #![deny(warnings)] set.

To be honest, running this without a proper feature gate is shaky, in terms of how unstable features work in the rest of the compiler and rustdoc, but until recently i was uncertain if we could properly introduce a #![feature] gate to the compiler without a matching use in rustc itself. (All the #![feature] attributes that rustdoc currently uses are tied to attributes, so their check code is also in the compiler.) I would be willing to introduce a feature gate for this, since it would also stamp down on the false positive rate across the ecosystem.

However, it leads to similar broken-link situations. Imagine crate A which uses intra-doc links, complete with having the feature set. Now imagine crate B, which re-exports items from crate A to inline them into its own documentation, but doesn't use intra-doc links itself. Because crate B doesn't have the feature set, now those links from the crate A items won't resolve, even if they were accidentally correct because those items were re-exported into the right positions!

(This situation exists today; these crates are futures and hyper, though the items from futures that hyper re-exports don't have any links on them.)

I'm very hesitant to "partially stabilize" these links, because of the subtleties with re-exports that @Manishearth mentioned. It's too easy to use any part of this feature and get it to a position where it doesn't work. And the major hurdle for making that work requires a deep compiler refactor that is hard to find a champion for. If it comes down to it, if there is another Rust All-Hands early next year like the one in Berlin earlier this year, i may try getting the right people in a room together to design this problem out. However, that won't be able to happen for another few months.

[euclio]

I wonder if the standard library crates will need special treatment by this feature? For example, the link to the REPLACEMENT_CHAR constant in String docs needs to resolve to std::char::REPLACEMENT_CHAR or core::char::REPLACEMENT_CHAR depending if the String docs are being rendered as part of the libstd or liballoc docs.