Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue: rfc 1990 - add external doc attribute to rustc #44732

Closed
2 of 3 tasks
nrc opened this issue Sep 21, 2017 · 85 comments
Closed
2 of 3 tasks

Tracking issue: rfc 1990 - add external doc attribute to rustc #44732

nrc opened this issue Sep 21, 2017 · 85 comments

Comments

@nrc
Copy link
Member

@nrc nrc commented Sep 21, 2017

RFC PR: rust-lang/rfcs#1990
RFC text: https://github.com/rust-lang/rfcs/blob/master/text/1990-external-doc-attribute.md

Current documentation


Summary of how to use this:

  1. Add #![feature(external_doc)] to your crate.
  2. Add a file "src/some-docs.md" with some docs to your crate.
  3. Add an attribute #[doc(include = "some-docs.md")] to something that needs some docs. The file path is relative to lib.rs, so if you want a doc folder to live alongside src, then all your paths inside the doc(include) attributes need to begin with ../doc.

Summary of current status:

  • Initial implementation landed in #44781, on 2017-11-22.
  • Updates based on conversation in this thread landed in #46858, on 2017-12-21.
  • Please use this in your own projects and comment in this thread if something goes wrong! (Or even if it goes well! Knowing it's working as intended is great!)

Current tasks:

  • Land #44781
  • (promote the "failed to load file" warnings to proper lints?) (Make them hard errors instead, per the RFC #46858)
  • Ensure line number info is properly preserved in doctest errors
QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Sep 22, 2017
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732
QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Sep 23, 2017
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732
QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Sep 23, 2017
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732
@matklad
Copy link
Member

@matklad matklad commented Oct 16, 2017

cc @joshtriplett

At today's Cargo meeting, and interesting question was raised how Cargo should check if it should rebuild the docs when an external file with docs changes.

Here's some info on how dependency-tracking works today:

When doing cargo build, Cargo passes an --emit=deb-info flag to rustc, which causes it to emit dep-info files with .d extension, which lists what files were used for compilation (remember, only crate roots are explicitly mentioned in Cargo.toml). These files then used in fingerprint.rs to compute the fingerprint.

Curiously, cargo doc does not emit dep-info at all, and fingerpint.rs does not use depinfo when doing doc. Instead it asks Source for fingerprint which for the local repository boils down to listing all files in a directory with Cargo.toml.

TL;DR: touch Readme.md today causes cargo doc to rebuild docs for the current crate, which might not be the most efficient thing to do, but which should work with external documentation, unless it is fetched from a place outside of Cargo package.

@joshtriplett
Copy link
Member

@joshtriplett joshtriplett commented Oct 16, 2017

@matklad So, it sounds like the current behavior is correct but could be made more efficient. That's better than being incorrect.

@matklad
Copy link
Member

@matklad matklad commented Oct 17, 2017

So, it sounds like the current behavior is correct but could be made more efficient.

There's an edge case where it might not do rebuild when it should, if in src/lib.rs you have something like

#[path="../../bar.rs"]
pub mod bar;

Or

include!("../../bar.rs")

That is, if you touch files outside of the Cargo package directory.

Other than that yes, current behavior is correct!.

QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 9, 2017
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732
QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 11, 2017
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732
QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 11, 2017
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732
QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 11, 2017
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732
QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 21, 2017
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732
QuietMisdreavus added a commit to QuietMisdreavus/rust that referenced this issue Nov 21, 2017
Partial implementation of rust-lang/rfcs#1990
(needs error reporting work)

cc rust-lang#44732
bors added a commit that referenced this issue Nov 22, 2017
rustdoc: include external files in documentation (RFC 1990)

Part of rust-lang/rfcs#1990 (needs work on the error reporting, which i'm deferring to after this initial PR)

cc #44732

Also fixes #42760, because the prep work for the error reporting made it easy to fix that at the same time.
bors added a commit that referenced this issue Nov 22, 2017
rustdoc: include external files in documentation (RFC 1990)

Part of rust-lang/rfcs#1990 (needs work on the error reporting, which i'm deferring to after this initial PR)

cc #44732

Also fixes #42760, because the prep work for the error reporting made it easy to fix that at the same time.
@luser
Copy link
Contributor

@luser luser commented Dec 6, 2017

This probably won't cause any issues in the real world, but related to the discussion above about dependency tracking, I noticed that #[doc(include="file")] doesn't wind up in the dependency info output by rustc --emit=dep-info, which doesn't seem right. (In practice, since doc comments presumably don't affect the compiler output, it probably wouldn't cause any actual correctness problems).

Compare vs. include_str! for example:

luser@eye7:/tmp$ rustc +nightly --version
rustc 1.24.0-nightly (560a5da9f 2017-11-27)
luser@eye7:/tmp$ cat > test1.rs <<EOF
> #[allow(unused)]
> const s: &str = include_str!("other-file");
> EOF
luser@eye7:/tmp$ cat > other-file <<EOF
> hello!
> EOF
luser@eye7:/tmp$ rustc +nightly test1.rs --emit=dep-info -o test1.d
luser@eye7:/tmp$ cat test1.d
test1.d: test1.rs other-file

test1.rs:
other-file:
luser@eye7:/tmp$ cat > test2.rs <<EOF
> #![feature(external_doc)]
> #![doc(include="other-file")]
> EOF
luser@eye7:/tmp$ rustc +nightly test2.rs --emit=dep-info -o test2.d
cluser@eye7:/tmp$ cat test2.d
test2.d: test2.rs

test2.rs:

It looks like include_str! et. al. simply call cx.codemap().new_filemap_and_lines(&filename, &contents) to make this happen:

// Add this input file to the code map to make it available as

so presumably doing the same thing inside the implementation of #[doc(include="..")] would fix this (probably here, I'm guessing):

let include_info = vec![

@luser
Copy link
Contributor

@luser luser commented Dec 6, 2017

I neglected to mention, but it looks like there's even a test to ensure that files used in include_str! and include_bytes! wind up in the dep-info: https://github.com/rust-lang/rust/tree/b1363a73ede57ae595f3a1be2bb75d308ba4f7f6/src/test/run-make/include_bytes_deps

@QuietMisdreavus
Copy link
Member

@QuietMisdreavus QuietMisdreavus commented Dec 6, 2017

Humorously enough, i asked about that when i was writing it:

<misdreavus> hmm, should i add files loaded by doc(include) into the codemap? include_str does
<jseyfried> I would lean against

Judging from the above discussion in this thread, it may be prudent to add it, but it also may not be necessary. I'd defer to @rust-lang/compiler for that, i guess.

@joshtriplett
Copy link
Member

@joshtriplett joshtriplett commented Dec 6, 2017

I personally would find it quite unfortunate if changing the included file doesn't lead to rebuilding the code. (One day, incremental compilation should make that a very fast rebuild.) I realize that usually it won't affect the generated code, but I could imagine a syntax extension of some kind processing doc comments (after all documentation has been normalized to the equivalent of a doc attribute containing a string), and it shouldn't be the job of that syntax extension to handle dependencies on included files.

@jonhoo
Copy link
Contributor

@jonhoo jonhoo commented Dec 13, 2017

The current implementation seems to interact poorly with #![deny_missing]:

$ cat test.md
foo
$ cat test.rs
#![doc(include = "test.md")]
#![feature(external_doc)]
#![deny(missing_docs)]

fn main() {}
$ rustc +nightly test.rs
error: missing documentation for crate
 --> test.rs:1:1
  |
1 | / #![doc(include = "test.md")]
2 | | #![feature(external_doc)]
3 | | #![deny(missing_docs)]
4 | |
5 | | fn main() {}
  | |____________^
  |
note: lint level defined here
 --> test.rs:3:9
  |
3 | #![deny(missing_docs)]
  |         ^^^^^^^^^^^^

error: aborting due to previous error

bors added a commit to rust-lang-ci/rust that referenced this issue Dec 10, 2020
Accept arbitrary expressions in key-value attributes at parse time

Continuation of rust-lang#77271.

We now support arbitrary expressions in values of key-value attributes at parse time.
```
#[my_attr = EXPR]
```
Previously only unsuffixed literals and interpolated expressions (`$expr`) were accepted.

There are two immediate motivational cases for this:
- External doc strings (`#[doc = include_str!("my_doc.md")]`, eliminating the need in rust-lang#44732) and expanding macros in this position in general. Currently such macro expansions are supported in this position in interpolated `$expr`s (the `#[doc = $doc]` idiom).
- Paths (`#[namespace = foo::bar] extern "C++" { ... }`) like proposed in rust-lang#76734.

If the attribute in question survives expansion, then the value is still restricted to unsuffixed literals by a semantic check.
This restriction doesn't prevent the use cases listed above, so this PR keeps it in place for now.

Closes rust-lang#52607.
Previous attempt - rust-lang#67121.
Some more detailed write up on internals - https://internals.rust-lang.org/t/macro-expansion-points-in-attributes/11455.
Tracking issue - rust-lang#78835.
@petrochenkov
Copy link
Contributor

@petrochenkov petrochenkov commented Dec 10, 2020

@jyn514 is planning to move this forward (to removal) by getting rid of existing uses of doc(include) (#78835), so unassigning myself.

@petrochenkov petrochenkov removed their assignment Dec 10, 2020
@jyn514 jyn514 self-assigned this Dec 10, 2020
@trevyn
Copy link
Contributor

@trevyn trevyn commented Jan 11, 2021

For reference, #[doc = include_str!("my_doc.md")] is now stable in 1.50 via-- #78837

Edit: Ooh, how embarrassing. Still behind #![feature(extended_key_value_attributes)] feature gate. (tracking #78835) Thanks @jyn514

@jyn514
Copy link
Member

@jyn514 jyn514 commented Jan 11, 2021

@trevyn include_str is not stable, it's still behind a feature gate.

@camelid
Copy link
Member

@camelid camelid commented Feb 25, 2021

To be clear, include_str! is stable, #[doc = include_str!] is not.

EDIT: clarified

@jyn514

This comment has been hidden.

@camelid

This comment has been hidden.

@jyn514
Copy link
Member

@jyn514 jyn514 commented Feb 26, 2021

@jyn514 is planning to move this forward (to removal) by getting rid of existing uses of doc(include) (#78835)

I opened #82539 deprecating doc(include). Assuming that makes it past FCP, I'll give it a few months then remove the feature altogether.

Dylan-DPC added a commit to Dylan-DPC/rust that referenced this issue May 18, 2021
…=petrochenkov

Stabilize extended_key_value_attributes

Closes rust-lang#44732. Closes rust-lang#78835. Closes rust-lang#82768 (by making it irrelevant).

 # Stabilization report

 ## Summary

This stabilizes using macro expansion in key-value attributes, like so:

 ```rust
 #[doc = include_str!("my_doc.md")]
 struct S;

 #[path = concat!(env!("OUT_DIR"), "/generated.rs")]
 mod m;
 ```

See Petrochenkov's excellent blog post [on internals](https://internals.rust-lang.org/t/macro-expansion-points-in-attributes/11455)
for alternatives that were considered and rejected ("why accept no more and no less?")

This has been available on nightly since 1.50 with no major issues.

## Notes

### Accepted syntax

The parser accepts arbitrary Rust expressions in this position, but any expression other than a macro invocation will ultimately lead to an error because it is not expected by the built-in expression forms (e.g., `#[doc]`).  Note that decorators and the like may be able to observe other expression forms.

### Expansion ordering

Expansion of macro expressions in "inert" attributes occurs after decorators have executed, analogously to macro expressions appearing in the function body or other parts of decorator input.

There is currently no way for decorators to accept macros in key-value position if macro expansion must be performed before the decorator executes (if the macro can simply be copied into the output for later expansion, that can work).

## Test cases

 - https://github.com/rust-lang/rust/blob/master/src/test/ui/attributes/key-value-expansion-on-mac.rs
 - https://github.com/rust-lang/rust/blob/master/src/test/rustdoc/external-doc.rs

The feature has also been dogfooded extensively in the compiler and
standard library:

- rust-lang#83329
- rust-lang#83230
- rust-lang#82641
- rust-lang#80534

## Implementation history

- Initial proposal: rust-lang#55414 (comment)
- Experiment to see how much code it would break: rust-lang#67121
- Preliminary work to restrict expansion that would conflict with this
feature: rust-lang#77271
- Initial implementation: rust-lang#78837
- Fix for an ICE: rust-lang#80563

## Unresolved Questions

~~rust-lang#83366 (comment) listed some concerns, but they have been resolved as of this final report.~~

 ## Additional Information

 There are two workarounds that have a similar effect for `#[doc]`
attributes on nightly. One is to emulate this behavior by using a limited version of this feature that was stabilized for historical reasons:

```rust
macro_rules! forward_inner_docs {
    ($e:expr => $i:item) => {
        #[doc = $e]
        $i
    };
}

forward_inner_docs!(include_str!("lib.rs") => struct S {});
```

This also works for other attributes (like `#[path = concat!(...)]`).
The other is to use `doc(include)`:

```rust
 #![feature(external_doc)]
 #[doc(include = "lib.rs")]
 struct S {}
```

The first works, but is non-trivial for people to discover, and
difficult to read and maintain. The second is a strange special-case for
a particular use of the macro. This generalizes it to work for any use
case, not just including files.

I plan to remove `doc(include)` when this is stabilized
(rust-lang#82539). The `forward_inner_docs`
workaround will still compile without warnings, but I expect it to be
used less once it's no longer necessary.
@bors bors closed this in 3c99dcd May 19, 2021
isislovecruft added a commit to isislovecruft/subtle that referenced this issue Jul 13, 2021
The latest rust nightlies and beta (at stable version 1.55) include a
change from

    #![cfg_attr(feature = "nightly", feature(external_doc))]
    #![cfg_attr(feature = "nightly", doc(include = "../README.md"))]

to removing `feature(external_doc)` and also changing the syntax of
the second line to

    #![cfg_attr(feature = "nightly", doc = include_str!("../README.md"))]

However.  `include_str!` is stable currently, but the syntax of `doc =
` is expressly disallowed.  This gives me four options:

 1. Don't build documentation (bad)
 2. Copy-pasta README.md into src/lib.rs (also bad)
 3. Support only beta/nightly but not stable (completely untennable
    for a crate with ~14 million downloads)
 4. Support only stable but not beta/nightly (also untennable)

Further, waiting for this to be "fixed" by its inclusion in stable
Rust in about a week means that our MSRV increases from 1.41 to 1.56,
with no changes to actual code (other than how to build documentation)
at all, which seems quite unfriendly to downstream dependents who are
pinning their rust versions for whatever reason.

So, sadly, it seems the most friendly fix is to copy-pasta our
README.md into the codebase.

(cf. rust-lang/rust#44732,
      rust-lang/rust#82539)
isislovecruft added a commit to isislovecruft/subtle that referenced this issue Jul 13, 2021
The latest rust nightlies and beta (at stable version 1.55) include a
change from

    #![cfg_attr(feature = "nightly", feature(external_doc))]
    #![cfg_attr(feature = "nightly", doc(include = "../README.md"))]

to removing `feature(external_doc)` and also changing the syntax of
the second line to

    #![cfg_attr(feature = "nightly", doc = include_str!("../README.md"))]

However.  `include_str!` is stable currently, but the syntax of `doc =
` is expressly disallowed.  This gives me four options:

 1. Don't build documentation (bad)
 2. Copy-pasta README.md into src/lib.rs (also bad)
 3. Support only beta/nightly but not stable (completely untennable
    for a crate with ~14 million downloads)
 4. Support only stable but not beta/nightly (also untennable)

Further, waiting for this to be "fixed" by its inclusion in stable
Rust in about a week means that our MSRV increases from 1.41 to 1.56,
with no changes to actual code (other than how to build documentation)
at all, which seems quite unfriendly to downstream dependents who are
pinning their rust versions for whatever reason.

So, sadly, it seems the most friendly fix is to copy-pasta our
README.md into the codebase.

(cf. rust-lang/rust#44732,
      rust-lang/rust#82539)
thedodd added a commit to thedodd/ybc that referenced this issue Jul 26, 2021
@gilescope
Copy link
Contributor

@gilescope gilescope commented Jul 28, 2021

That relative paths niggle: Has anyone written a macro that fixes the relative links?
#[doc = deepen_relatives!(include_str("../README.md"), 1)]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.