Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for RFC 1946 - intra-rustdoc links #43466

Closed
nrc opened this issue Jul 25, 2017 · 159 comments
Closed

Tracking issue for RFC 1946 - intra-rustdoc links #43466

nrc opened this issue Jul 25, 2017 · 159 comments

Comments

@nrc
Copy link
Member

@nrc nrc commented Jul 25, 2017

RFC: https://github.com/rust-lang/rfcs/blob/master/text/1946-intra-rustdoc-links.md
RFC PR: rust-lang/rfcs#1946

Todo:

  • Land #47046
  • Make it work with cross-crate imports (#65983, #47046 (comment))
  • Make it work for proc macros #73173
  • Make it warn more on resolution errors (#49542)
  • Make warnings/errors use better spans than the first line of docs
    • first step: attribute (#51111)
    • point to actual link in doc comment (#51391)
    • Add lint for link resolution failure (#51382)
  • Implied Shortcut Reference Links (#48335)
  • Enum variants and UFCS/associated methods (#47701)
  • Primitives (#49459)
  • Use the right resolution scopes for inner doc comments on things that aren't modules (#55364)
  • Handle fields and variants (#49512)
    • Handle fields inside struct variants (#66704)
  • Support Self (#49583)
  • Write documentation (#66000)
  • Stabilization (#74430 - current impl is gated to only run on nightly, but does not require a feature gate)
  • Handle methods from traits implementations
  • Support anchors (#66675)
  • Support missing primitive types (such as slices) (#63351)
  • Allow referring to Self::item (for cmp on Ord)
  • Fix current crate linking when no module is provided in the path (for example, Foo won't link to the crate's Foo type for some reason)
  • doc(hidden) reexports can break intra-doc links (#73363)
  • Support associated items in contexts other than documenting a trait method (#74489)

Other related issues:

  • Suppress privacy errors with document-private-types (#74134, fix at #74147)
  • provide option to turn intra-link resolution errors into proper errors (#50082)
@QuietMisdreavus
Copy link
Member

@QuietMisdreavus QuietMisdreavus commented Sep 19, 2017

Preliminary implementation notes:

Getting just the links from the markdown is relatively straightforward. You'll need to do this twice, to capture the behavior in both the Hoedown and Pulldown markdown renderers. Thankfully, each has a way of capturing links as they are parsed from the document. (While capturing the link you can also look for the "struct/enum/mod/macro/static/etc" marker to aid in resolution down the line.)

In Hoedown, the render function in src/librustdoc/html/markdown.rs captures all the events as they are captured. Hoedown uses a system of registering callbacks on the renderer configuration to capture each event separately. The link event and its related linkfn function pointer signature are already present in the file, so you would need to create and use that callback to capture and modify links as they appear.

For Pulldown, the parser is surfaced as an Iterator of events. There are already a handful of iterator wrappers present in html/markdown.rs, so adding another one (and using it in the impl fmt::Display for Markdown farther down) shouldn't be a big deal. I'm... honestly not that familiar with how Pulldown represents link definitions in its Event/Tag enums, so more research would be needed to effectively get link text for that.

Now, with the links in hand, you need to do two things, neither of which i know offhand since they involve interfacing with the compiler:

  • Attempt to parse the link as a path (if it's not a valid Rust path, just emit the URL as-is and skip the rest of this)
  • Resolve the path in the "current" module (emit a warning if resolution fails)

Presumably there's a way to make the resolution come up with a DefId? Even after cleaning the AST, rustdoc still deals in those, so if that's the case there are ways to map that to the proper relative link.

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Dec 28, 2017

I've got a WIP going on in https://github.com/manishearth/rust/tree/correct-resolver-fail , which works off of Misdreavus' WIP (they did most of the work).

@Manishearth Manishearth mentioned this issue Dec 28, 2017
6 of 12 tasks complete
@QuietMisdreavus
Copy link
Member

@QuietMisdreavus QuietMisdreavus commented Dec 28, 2017

Note that that branch is now live as #47046, which i'm polishing and finishing up.

@QuietMisdreavus
Copy link
Member

@QuietMisdreavus QuietMisdreavus commented Jan 8, 2018

Note for people watching this thread but not #47046: We're trying to wind that PR down to get a minimal version landed, so an initial implementation is fairly close.

bors added a commit that referenced this issue Jan 22, 2018
…Gomez,QuietMisdreavus,Manishearth

Implement RFC 1946 - intra-rustdoc links

rust-lang/rfcs#1946 #43466

Note for reviewers: The plain line counts are a little inflated because of how the markdown link parsing was done. [Read the file diff with "whitespace only" changes removed](https://github.com/rust-lang/rust/pull/47046/files?w=1) to get a better view of what actually changed there.

This pulls the name/path resolution mechanisms out of the compiler and runs it on the markdown in a crate's docs, so that links can be made to `SomeStruct` directly rather than finding the folder path to `struct.SomeStruct.html`. Check the `src/test/rustdoc/intra-paths.rs` test in this PR for a demo. The change was... a little invasive, but unlocks a really powerful mechanism for writing documentation that doesn't care about where an item was written to on the hard disk.

Items included:

 - [x] Make work with the hoedown renderer
 - [x] Handle relative paths
 - [x] Parse out the "path ambiguities" qualifiers (`[crate foo]`, `[struct Foo]`, `[foo()]`, `[static FOO]`, `[foo!]`, etc)
 - [x] Resolve foreign macros
 - [x] Resolve local macros
 - [x] Handle the use of inner/outer attributes giving different resolution scopes (handling for non-modules pushed to different PR)

Items not included:

 - [ ] Make sure cross-crate inlining works (blocked on refactor described in #47046 (comment))
 - [ ] Implied Shortcut Reference Links (where just doing `[::std::iter::Iterator][]` without a reference anchor will resolve using the reference name rather than the link target) (requires modifying the markdown parser - blocked on Hoedown/Pulldown switch and raphlinus/pulldown-cmark#121)
 - [ ] Handle enum variants and UFCS methods (Enum variants link to the enum page, associated methods don't link at all)
 - [ ] Emit more warnings/errors when things fail to resolve (linking to a value-namespaced item without a qualifier will emit an error, otherwise the link is just treated as a url, not a rust path)
 - [ ] Give better spans for resolution errors (currently the span for the first doc comment is used)
 - [ ] Check for inner doc comments on things that aren't modules

I'm making the PR, but it should be noted that most of the work was done by Misdreavus 😄

(Editor's note: This has become a lie, check that commit log, Manish did a ton of work after this PR was opened `>_>`)
bors added a commit that referenced this issue Jan 22, 2018
…Gomez,QuietMisdreavus,Manishearth,killercup

Implement RFC 1946 - intra-rustdoc links

rust-lang/rfcs#1946 #43466

Note for reviewers: The plain line counts are a little inflated because of how the markdown link parsing was done. [Read the file diff with "whitespace only" changes removed](https://github.com/rust-lang/rust/pull/47046/files?w=1) to get a better view of what actually changed there.

This pulls the name/path resolution mechanisms out of the compiler and runs it on the markdown in a crate's docs, so that links can be made to `SomeStruct` directly rather than finding the folder path to `struct.SomeStruct.html`. Check the `src/test/rustdoc/intra-paths.rs` test in this PR for a demo. The change was... a little invasive, but unlocks a really powerful mechanism for writing documentation that doesn't care about where an item was written to on the hard disk.

Items included:

 - [x] Make work with the hoedown renderer
 - [x] Handle relative paths
 - [x] Parse out the "path ambiguities" qualifiers (`[crate foo]`, `[struct Foo]`, `[foo()]`, `[static FOO]`, `[foo!]`, etc)
 - [x] Resolve foreign macros
 - [x] Resolve local macros
 - [x] Handle the use of inner/outer attributes giving different resolution scopes (handling for non-modules pushed to different PR)

Items not included:

 - [ ] Make sure cross-crate inlining works (blocked on refactor described in #47046 (comment))
 - [ ] Implied Shortcut Reference Links (where just doing `[::std::iter::Iterator][]` without a reference anchor will resolve using the reference name rather than the link target) (requires modifying the markdown parser - blocked on Hoedown/Pulldown switch and raphlinus/pulldown-cmark#121)
 - [ ] Handle enum variants and UFCS methods (Enum variants link to the enum page, associated methods don't link at all)
 - [ ] Emit more warnings/errors when things fail to resolve (linking to a value-namespaced item without a qualifier will emit an error, otherwise the link is just treated as a url, not a rust path)
 - [ ] Give better spans for resolution errors (currently the span for the first doc comment is used)
 - [ ] Check for inner doc comments on things that aren't modules

I'm making the PR, but it should be noted that most of the work was done by Misdreavus 😄

(Editor's note: This has become a lie, check that commit log, Manish did a ton of work after this PR was opened `>_>`)
bors added a commit that referenced this issue Jan 23, 2018
…Gomez,QuietMisdreavus,Manishearth

Implement RFC 1946 - intra-rustdoc links

rust-lang/rfcs#1946 #43466

Note for reviewers: The plain line counts are a little inflated because of how the markdown link parsing was done. [Read the file diff with "whitespace only" changes removed](https://github.com/rust-lang/rust/pull/47046/files?w=1) to get a better view of what actually changed there.

This pulls the name/path resolution mechanisms out of the compiler and runs it on the markdown in a crate's docs, so that links can be made to `SomeStruct` directly rather than finding the folder path to `struct.SomeStruct.html`. Check the `src/test/rustdoc/intra-paths.rs` test in this PR for a demo. The change was... a little invasive, but unlocks a really powerful mechanism for writing documentation that doesn't care about where an item was written to on the hard disk.

Items included:

 - [x] Make work with the hoedown renderer
 - [x] Handle relative paths
 - [x] Parse out the "path ambiguities" qualifiers (`[crate foo]`, `[struct Foo]`, `[foo()]`, `[static FOO]`, `[foo!]`, etc)
 - [x] Resolve foreign macros
 - [x] Resolve local macros
 - [x] Handle the use of inner/outer attributes giving different resolution scopes (handling for non-modules pushed to different PR)

Items not included:

 - [ ] Make sure cross-crate inlining works (blocked on refactor described in #47046 (comment))
 - [ ] Implied Shortcut Reference Links (where just doing `[::std::iter::Iterator][]` without a reference anchor will resolve using the reference name rather than the link target) (requires modifying the markdown parser - blocked on Hoedown/Pulldown switch and raphlinus/pulldown-cmark#121)
 - [ ] Handle enum variants and UFCS methods (Enum variants link to the enum page, associated methods don't link at all)
 - [ ] Emit more warnings/errors when things fail to resolve (linking to a value-namespaced item without a qualifier will emit an error, otherwise the link is just treated as a url, not a rust path)
 - [ ] Give better spans for resolution errors (currently the span for the first doc comment is used)
 - [ ] Check for inner doc comments on things that aren't modules

I'm making the PR, but it should be noted that most of the work was done by Misdreavus 😄

(Editor's note: This has become a lie, check that commit log, Manish did a ton of work after this PR was opened `>_>`)
@sgrif
Copy link
Contributor

@sgrif sgrif commented Jan 23, 2018

One thing that appears to not work in #47046 (and I don't think is ever explicitly addressed in the RFC) is how to link to primitives like u32 or bool.

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Jan 24, 2018

Yeah I have to investigate that

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Jan 24, 2018

@killercup @QuietMisdreavus should we turn the error into a warning ? I don't like the idea of crate docs breaking because someone introduced a name into a glob import

@killercup
Copy link
Member

@killercup killercup commented Jan 24, 2018

@Manishearth does rustdoc have some idea of "lint levels"? What are the current deprecation warnings? Just println!s?

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Jan 24, 2018

We can just struct_span_warn instead. rustc can emit warnings not tied to actual warning names. There are no lint levels.

@killercup
Copy link
Member

@killercup killercup commented Jan 24, 2018

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Jan 24, 2018

lawliet89 added a commit to lawliet89/rocket_cors that referenced this issue Feb 14, 2018
@Manishearth
Copy link
Member

@Manishearth Manishearth commented Feb 18, 2018

Shortcut links being implemented at #48335

Manishearth added a commit to Manishearth/rust that referenced this issue Feb 19, 2018
…Misdreavus

Implement implied shortcut links for intra-rustdoc-links

cc rust-lang#43466

Needs raphlinus/pulldown-cmark#126

r? @QuietMisdreavus
Manishearth added a commit to Manishearth/rust that referenced this issue Feb 19, 2018
…Misdreavus

Implement implied shortcut links for intra-rustdoc-links

cc rust-lang#43466

Needs raphlinus/pulldown-cmark#126

r? @QuietMisdreavus
GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this issue Feb 21, 2018
…Misdreavus

Implement implied shortcut links for intra-rustdoc-links

cc rust-lang#43466

Needs raphlinus/pulldown-cmark#126

r? @QuietMisdreavus
@Manishearth
Copy link
Member

@Manishearth Manishearth commented Jul 9, 2020

@ollie27 We have had a functional implementation for three years now. It has been used in the stdlib docs for ages, and is used across the ecosystem. The ability for reexports to work correctly was a blocker to stabilization, but it is not a huge feature that will be used a lot. The reason stabilization was blocked on reexports was that we determined that the bug fix could be complicated and might require rearchitecting the feature, which made us reluctant to stabilize earlier.

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Jul 9, 2020

seeing if it can fix the issues it was designed for (#32129, #32130)

I hadn't seen these issues earlier, and we should absolutely fix them using intra doc links, but I don't want to block stabilization on that. We've had this feature mostly working for ages.

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Jul 15, 2020

Alright, so the current status of this is that we have a stack of PRs blocking all remaining major issues (#65983, #73363): #74347 -> #73566 -> #73101 -> #73365

Between @GuillaumeGomez, @jyn514, and myself, we hope to stabilize this shortly thereafter. @GuillaumeGomez will be filing an FCP but as we're currently mostly waiting for the final chips to fall in place, I wanted to front-load some of the discussions that may happen during that FCP. Hopefully when we make that FCP it is simply some rubber-stamping if we have these discussions earlier.

@rust-lang/rustdoc What are your thoughts on stabilizing this shortly after the last few patches land? I recall @ollie27 having some concerns. I don't recall if @kinnison has said anything on this point.

My position on this (I think this is shared by Guillaume and Joshua) is that this feature has been cooking on nightly for three years now. The cross-crate import issue was a blocker not because it was a major piece of functionality, but rather because there was a chance we wouldn't be able to implement it in a neat manner and would paint ourselves into a corner by committing to it. It's clear that that's not the case. If we're worried about the code for cross-crate imports itself being broken, that's what the release train schedule is for -- we can undo these patches and stabilization for twelve weeks if this happens.

@kinnison
Copy link
Contributor

@kinnison kinnison commented Jul 16, 2020

I think that once the remaining stack lands, we're in good shape to stabilise. There're bound to be corner cases we've not found etc. and documentation which could be improved, but getting this out and usable by the general developer community will be immensely useful.

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Jul 17, 2020

#65983 has landed, so now we're ready for stabilization. #73363 should land soon

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Jul 17, 2020

Stabilization PR is ready #74430

@jyn514
Copy link
Member

@jyn514 jyn514 commented Aug 19, 2020

It should be noted that in the RFC discussion it was determined that exact knowledge of the item type should not be necessary; only knowing the namespace should suffice. It is acceptable that the tool resolving the links allows (and successfully resolves) a link with the wrong prefix that is in the same namespace. E.g., given an struct Foo, it may be possible to link to it using [enum Foo], or, given a mod bar, it may be possible to link to that using [struct bar].

After #75079 this is no longer the case. Should the RFC be updated?

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Aug 20, 2020

@jyn514 go for it

@runiq
Copy link

@runiq runiq commented Sep 4, 2020

Triage: #74489 has been merged, so its mark can be checked, I believe?

Dylan-DPC added a commit to Dylan-DPC/rust that referenced this issue Sep 24, 2020
…Manishearth

Stabilize intra-doc links

Fixes rust-lang#43466

Thanks to the great work of @jyn514 in getting the [cross-crate reexport issue](rust-lang#65983) in intra-rustdoc links fixed, I think we're now in a position to stabilize this feature.

The tracking issue currently has two unresolved issues:

 - <s>behavior around doc(hidden): This is fixed in rust-lang#73365, which is just waiting for CI and should land tomorrow. It's also a pretty niche bug so while I expect it to land soon I don't think we need to block stabilization on it anyway.</s>
 - Non-identifier primitive types like slices: This was not a part of the original RFC anyway, and is a pretty niche use case

The feature itself, sans rust-lang#65983, has been shipped on nightly for three years now, with people using it on docs.rs. rust-lang#65983 itself is not an overwhelmingly central bit of functionality; the reason we elected to block stabilization on it was that back in 2017 it was not possible to fix the issue without some major refactorings of resolve, and we did not want to stabilize something that had such a potentially unfixable bug.

Given that we've fixed it, I see no reason to delay stabilization on this long awaited feature. It's possible that the latest patches have problems, however we _have_ done crater runs of some of the crucial parts. Furthermore, that's what the release trains are for, we will have a solid three months to let it ride the trains before it actually hits the stable compiler.

r? @rust-lang/rustdoc
@bors bors closed this in 78a0894 Sep 24, 2020
v1olen added a commit to v1olen/rocket_cors that referenced this issue Mar 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.