Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue: rfc 1990 - add external doc attribute to rustc #44732

Open
nrc opened this Issue Sep 21, 2017 · 58 comments

Comments

Projects
None yet
@nrc
Copy link
Member

commented Sep 21, 2017

RFC PR: rust-lang/rfcs#1990
RFC text: https://github.com/rust-lang/rfcs/blob/master/text/1990-external-doc-attribute.md

Current documentation


Summary of how to use this:

  1. Add #![feature(external_doc)] to your crate.
  2. Add a file "src/some-docs.md" with some docs to your crate.
  3. Add an attribute #[doc(include = "some-docs.md")] to something that needs some docs. The file path is relative to lib.rs, so if you want a doc folder to live alongside src, then all your paths inside the doc(include) attributes need to begin with ../doc.

Summary of current status:

  • Initial implementation landed in #44781, on 2017-11-22.
  • Updates based on conversation in this thread landed in #46858, on 2017-12-21.
  • Please use this in your own projects and comment in this thread if something goes wrong! (Or even if it goes well! Knowing it's working as intended is great!)

Current tasks:

  • Land #44781
  • (promote the "failed to load file" warnings to proper lints?) (Make them hard errors instead, per the RFC #46858)
  • Ensure line number info is properly preserved in doctest errors

QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Sep 22, 2017

allow loading external files in documentation
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732

QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Sep 23, 2017

allow loading external files in documentation
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732

QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Sep 23, 2017

allow loading external files in documentation
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732
@matklad

This comment has been minimized.

Copy link
Member

commented Oct 16, 2017

cc @joshtriplett

At today's Cargo meeting, and interesting question was raised how Cargo should check if it should rebuild the docs when an external file with docs changes.

Here's some info on how dependency-tracking works today:

When doing cargo build, Cargo passes an --emit=deb-info flag to rustc, which causes it to emit dep-info files with .d extension, which lists what files were used for compilation (remember, only crate roots are explicitly mentioned in Cargo.toml). These files then used in fingerprint.rs to compute the fingerprint.

Curiously, cargo doc does not emit dep-info at all, and fingerpint.rs does not use depinfo when doing doc. Instead it asks Source for fingerprint which for the local repository boils down to listing all files in a directory with Cargo.toml.

TL;DR: touch Readme.md today causes cargo doc to rebuild docs for the current crate, which might not be the most efficient thing to do, but which should work with external documentation, unless it is fetched from a place outside of Cargo package.

@joshtriplett

This comment has been minimized.

Copy link
Member

commented Oct 16, 2017

@matklad So, it sounds like the current behavior is correct but could be made more efficient. That's better than being incorrect.

@matklad

This comment has been minimized.

Copy link
Member

commented Oct 17, 2017

So, it sounds like the current behavior is correct but could be made more efficient.

There's an edge case where it might not do rebuild when it should, if in src/lib.rs you have something like

#[path="../../bar.rs"]
pub mod bar;

Or

include!("../../bar.rs")

That is, if you touch files outside of the Cargo package directory.

Other than that yes, current behavior is correct!.

QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 9, 2017

allow loading external files in documentation
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732

QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 11, 2017

allow loading external files in documentation
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732

QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 11, 2017

allow loading external files in documentation
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732

QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 11, 2017

allow loading external files in documentation
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732

QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 21, 2017

allow loading external files in documentation
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732

QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 21, 2017

allow loading external files in documentation
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732

bors added a commit that referenced this issue Nov 22, 2017

Auto merge of #44781 - QuietMisdreavus:doc-include, r=GuillaumeGomez
rustdoc: include external files in documentation (RFC 1990)

Part of rust-lang/rfcs#1990 (needs work on the error reporting, which i'm deferring to after this initial PR)

cc #44732

Also fixes #42760, because the prep work for the error reporting made it easy to fix that at the same time.

bors added a commit that referenced this issue Nov 22, 2017

Auto merge of #44781 - QuietMisdreavus:doc-include, r=GuillaumeGomez
rustdoc: include external files in documentation (RFC 1990)

Part of rust-lang/rfcs#1990 (needs work on the error reporting, which i'm deferring to after this initial PR)

cc #44732

Also fixes #42760, because the prep work for the error reporting made it easy to fix that at the same time.
@luser

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2017

This probably won't cause any issues in the real world, but related to the discussion above about dependency tracking, I noticed that #[doc(include="file")] doesn't wind up in the dependency info output by rustc --emit=dep-info, which doesn't seem right. (In practice, since doc comments presumably don't affect the compiler output, it probably wouldn't cause any actual correctness problems).

Compare vs. include_str! for example:

luser@eye7:/tmp$ rustc +nightly --version
rustc 1.24.0-nightly (560a5da9f 2017-11-27)
luser@eye7:/tmp$ cat > test1.rs <<EOF
> #[allow(unused)]
> const s: &str = include_str!("other-file");
> EOF
luser@eye7:/tmp$ cat > other-file <<EOF
> hello!
> EOF
luser@eye7:/tmp$ rustc +nightly test1.rs --emit=dep-info -o test1.d
luser@eye7:/tmp$ cat test1.d
test1.d: test1.rs other-file

test1.rs:
other-file:
luser@eye7:/tmp$ cat > test2.rs <<EOF
> #![feature(external_doc)]
> #![doc(include="other-file")]
> EOF
luser@eye7:/tmp$ rustc +nightly test2.rs --emit=dep-info -o test2.d
cluser@eye7:/tmp$ cat test2.d
test2.d: test2.rs

test2.rs:

It looks like include_str! et. al. simply call cx.codemap().new_filemap_and_lines(&filename, &contents) to make this happen:

// Add this input file to the code map to make it available as

so presumably doing the same thing inside the implementation of #[doc(include="..")] would fix this (probably here, I'm guessing):

let include_info = vec![

@luser

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2017

I neglected to mention, but it looks like there's even a test to ensure that files used in include_str! and include_bytes! wind up in the dep-info: https://github.com/rust-lang/rust/tree/b1363a73ede57ae595f3a1be2bb75d308ba4f7f6/src/test/run-make/include_bytes_deps

@QuietMisdreavus

This comment has been minimized.

Copy link
Member

commented Dec 6, 2017

Humorously enough, i asked about that when i was writing it:

<misdreavus> hmm, should i add files loaded by doc(include) into the codemap? include_str does
<jseyfried> I would lean against

Judging from the above discussion in this thread, it may be prudent to add it, but it also may not be necessary. I'd defer to @rust-lang/compiler for that, i guess.

@joshtriplett

This comment has been minimized.

Copy link
Member

commented Dec 6, 2017

I personally would find it quite unfortunate if changing the included file doesn't lead to rebuilding the code. (One day, incremental compilation should make that a very fast rebuild.) I realize that usually it won't affect the generated code, but I could imagine a syntax extension of some kind processing doc comments (after all documentation has been normalized to the equivalent of a doc attribute containing a string), and it shouldn't be the job of that syntax extension to handle dependencies on included files.

@jonhoo

This comment has been minimized.

Copy link
Contributor

commented Dec 13, 2017

The current implementation seems to interact poorly with #![deny_missing]:

$ cat test.md
foo
$ cat test.rs
#![doc(include = "test.md")]
#![feature(external_doc)]
#![deny(missing_docs)]

fn main() {}
$ rustc +nightly test.rs
error: missing documentation for crate
 --> test.rs:1:1
  |
1 | / #![doc(include = "test.md")]
2 | | #![feature(external_doc)]
3 | | #![deny(missing_docs)]
4 | |
5 | | fn main() {}
  | |____________^
  |
note: lint level defined here
 --> test.rs:3:9
  |
3 | #![deny(missing_docs)]
  |         ^^^^^^^^^^^^

error: aborting due to previous error
@euclio

This comment has been minimized.

Copy link
Contributor

commented Nov 28, 2018

I have a branch pending that improves the diagnostics for this feature, but it's based on #56258, so I'm going to let that PR get merged before opening another PR.

pietroalbini added a commit to pietroalbini/rust that referenced this issue Dec 11, 2018

Rollup merge of rust-lang#56679 - euclio:external-doc-parse, r=estebank
overhaul external doc attribute diagnostics

This PR improves the error handling and spans for the external doc attribute. Many cases that silently failed before now emit errors, spans are tightened, and the errors have help and suggestions.

I tried to address all the cases that users ran into in the tracking issue.

cc rust-lang#44732

r? @QuietMisdreavus

kennytm added a commit to kennytm/rust that referenced this issue Dec 12, 2018

Rollup merge of rust-lang#56679 - euclio:external-doc-parse, r=estebank
overhaul external doc attribute diagnostics

This PR improves the error handling and spans for the external doc attribute. Many cases that silently failed before now emit errors, spans are tightened, and the errors have help and suggestions.

I tried to address all the cases that users ran into in the tracking issue.

cc rust-lang#44732

r? @QuietMisdreavus

pietroalbini added a commit to pietroalbini/rust that referenced this issue Dec 12, 2018

Rollup merge of rust-lang#56679 - euclio:external-doc-parse, r=estebank
overhaul external doc attribute diagnostics

This PR improves the error handling and spans for the external doc attribute. Many cases that silently failed before now emit errors, spans are tightened, and the errors have help and suggestions.

I tried to address all the cases that users ran into in the tracking issue.

cc rust-lang#44732

r? @QuietMisdreavus

pietroalbini added a commit to pietroalbini/rust that referenced this issue Dec 13, 2018

Rollup merge of rust-lang#56679 - euclio:external-doc-parse, r=estebank
overhaul external doc attribute diagnostics

This PR improves the error handling and spans for the external doc attribute. Many cases that silently failed before now emit errors, spans are tightened, and the errors have help and suggestions.

I tried to address all the cases that users ran into in the tracking issue.

cc rust-lang#44732

r? @QuietMisdreavus

kennytm added a commit to kennytm/rust that referenced this issue Dec 13, 2018

Rollup merge of rust-lang#56679 - euclio:external-doc-parse, r=estebank
overhaul external doc attribute diagnostics

This PR improves the error handling and spans for the external doc attribute. Many cases that silently failed before now emit errors, spans are tightened, and the errors have help and suggestions.

I tried to address all the cases that users ran into in the tracking issue.

cc rust-lang#44732

r? @QuietMisdreavus

kennytm added a commit to kennytm/rust that referenced this issue Dec 14, 2018

Rollup merge of rust-lang#56679 - euclio:external-doc-parse, r=estebank
overhaul external doc attribute diagnostics

This PR improves the error handling and spans for the external doc attribute. Many cases that silently failed before now emit errors, spans are tightened, and the errors have help and suggestions.

I tried to address all the cases that users ran into in the tracking issue.

cc rust-lang#44732

r? @QuietMisdreavus

kennytm added a commit to kennytm/rust that referenced this issue Dec 14, 2018

Rollup merge of rust-lang#56679 - euclio:external-doc-parse, r=estebank
overhaul external doc attribute diagnostics

This PR improves the error handling and spans for the external doc attribute. Many cases that silently failed before now emit errors, spans are tightened, and the errors have help and suggestions.

I tried to address all the cases that users ran into in the tracking issue.

cc rust-lang#44732

r? @QuietMisdreavus

pietroalbini added a commit to pietroalbini/rust that referenced this issue Dec 14, 2018

Rollup merge of rust-lang#56679 - euclio:external-doc-parse, r=estebank
overhaul external doc attribute diagnostics

This PR improves the error handling and spans for the external doc attribute. Many cases that silently failed before now emit errors, spans are tightened, and the errors have help and suggestions.

I tried to address all the cases that users ran into in the tracking issue.

cc rust-lang#44732

r? @QuietMisdreavus

pietroalbini added a commit to pietroalbini/rust that referenced this issue Dec 15, 2018

Rollup merge of rust-lang#56679 - euclio:external-doc-parse, r=estebank
overhaul external doc attribute diagnostics

This PR improves the error handling and spans for the external doc attribute. Many cases that silently failed before now emit errors, spans are tightened, and the errors have help and suggestions.

I tried to address all the cases that users ran into in the tracking issue.

cc rust-lang#44732

r? @QuietMisdreavus
@abonander

This comment has been minimized.

Copy link
Contributor

commented Jan 23, 2019

It seems like a really underpromoted possibility of this feature (though it obviously hasn't gone unnoticed) is the ability to have your README automatically doctested. I think on that basis alone it should be possible to generate a lot of interest and engineering effort toward getting this working well and stabilized.

@abonander

This comment has been minimized.

Copy link
Contributor

commented Jan 23, 2019

I've got a proof-of-concept project for using this feature to test code examples in README.md: https://github.com/abonander/readme-doctest-poc

I chose to apply the attribute to a private typedef since no code gets generated for those. Fixing the file/line numbers for test failures in included files is definitely important though; I can see it being a real pain trying to fix failures if you have a lot of examples (it's not exactly common to add a lot of context to asserts in examples since they're usually self-evident).

@abonander

This comment has been minimized.

Copy link
Contributor

commented Jan 23, 2019

Another thought: this would be significantly cooler if the Rustdoc CSS had a class that hides its contents so you could just reuse your README as your crate root docs and just hide stuff that doesn't necessarily need to be in the docs like license information or contributing instructions or CI build status.

That use-case is unfortunately a non-starter for most while this feature is unstable anyway. I don't want to require nightly to build documentation (or just sometimes not have crate root docs).

@jonhoo

This comment has been minimized.

Copy link
Contributor

commented Jan 23, 2019

@abonander the way I've usually gone about this is to use cargo-readme to generate the README from the crate-level docs, but that has the issue that relative links won't work.

@Nemo157

This comment has been minimized.

Copy link
Contributor

commented Jan 23, 2019

The paths being relative to the root module's file still seems like the wrong choice to me (at least for the use-cases I have). @yoshuawuyts and @SimonSapin appear to agree and @mgattozzi's last comment seems to be ok with changing it (and I haven't seen any other opinions noted since implementation). Is there any chance of re-discussing this now that there's some usage data to base it off?

@mgattozzi

This comment has been minimized.

Copy link
Member

commented Jan 24, 2019

I'm sure @QuietMisdreavus has a more informed opinion of this now then I would regarding this. I'm of the opinion of the least amount of surprise meaning consistency with what Rust has done before, whether it's relative to the file or the crate root (I'm not sure which it is tbh).

@SimonSapin

This comment has been minimized.

Copy link
Contributor

commented Jan 24, 2019

What Rust has done before is that the include!, include_byte!, and include_str! macros take a path relative to that of the source file that contains the macro invocation. This is consistent with how links work in HTML, or URLs on the web in general.

@QuietMisdreavus

This comment has been minimized.

Copy link
Member

commented Jan 24, 2019

I have the same opinion about source file directories as in my comment from last year:

The reason that lib.rs was chosen was because rustdoc has no concept of Cargo projects, and also because the path behavior of include! and friends is considered a mistake for hygiene reasons. Moving a source file which uses one of these macros changes its behavior. At least, that's the explanation i remember seeing.

I would not be opposed to adding a parameter to rustdoc to set this root arbitrarily, with the expectation that Cargo could use this for cargo doc invocations. The only issue would be that passing this CLI argument would be a pain until Cargo supported it. And i'm not sure how Cargo would be convinced to handle passing the -Z unstable-features flag for such a feature while the flag itself was unstable.

@swfsql

This comment has been minimized.

Copy link

commented Feb 26, 2019

Thanks! I didn't know if this feature.
I plan on having a README for the library, and also for each module which has its own directory.
This use case benefits from source-file relative path, and I think this structuring would be the cleanest regarding GitHub projects.

@jonas-schievink

This comment has been minimized.

Copy link
Member

commented Feb 26, 2019

I don't really get the reasoning behind the discrepancy in relative paths for #[doc(include)]. It seems to me like this is just needlessly confusing for people trying this feature for the first time. All other parts of Rust that currently act on files at compile time (#[path] on a module declaration, and the include_x! macros) base their paths off of the containing file.

The point that "rustdoc has no concept of Cargo projects" doesn't make much sense to me considering that rustc also doesn't know anything about Cargo project structure (other than module structure, obviously, but rustdoc also knows that).

The existing behavior of #[path] and include_x! might have been a mistake, but I think it would also be a mistake to introduce a similar-looking feature with different behavior without a concrete plan to move everything to that behavior. I guess #[path] and include_x! would be fixable in another edition, since all the include_x! macros are built-in, so changing their behavior based on the edition of the crate that names the macro would be possible.

@jonas-schievink jonas-schievink referenced this issue Mar 5, 2019

Open

Write docs and examples #8

1 of 7 tasks complete
@shepmaster

This comment has been minimized.

Copy link
Member

commented Mar 11, 2019

Is there some way of using this feature with some kind of fallback for stable Rust? I'd love to use this, but completely losing docs for any stable users would be a real deal killer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.