Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustdoc: add option to calculate "documentation coverage" #58626

Merged
merged 14 commits into from Mar 9, 2019

Conversation

Projects
None yet
5 participants
@QuietMisdreavus
Copy link
Member

QuietMisdreavus commented Feb 21, 2019

This PR adds a new flag to rustdoc, --show-coverage. When passed, this flag will make rustdoc count the number of items in a crate with documentation instead of generating docs. This count will be output as a table of each file in the crate, like this (when run on my crate egg-mode):

+-------------------------------------+------------+------------+------------+
| File                                | Documented |      Total | Percentage |
+-------------------------------------+------------+------------+------------+
| src/auth.rs                         |         16 |         16 |     100.0% |
| src/common/mod.rs                   |          1 |          1 |     100.0% |
| src/common/response.rs              |          9 |          9 |     100.0% |
| src/cursor.rs                       |         24 |         24 |     100.0% |
| src/direct/fun.rs                   |          6 |          6 |     100.0% |
| src/direct/mod.rs                   |         41 |         41 |     100.0% |
| src/entities.rs                     |         50 |         50 |     100.0% |
| src/error.rs                        |         27 |         27 |     100.0% |
| src/lib.rs                          |          1 |          1 |     100.0% |
| src/list/fun.rs                     |         19 |         19 |     100.0% |
| src/list/mod.rs                     |         22 |         22 |     100.0% |
| src/media/mod.rs                    |         27 |         27 |     100.0% |
| src/place/fun.rs                    |          8 |          8 |     100.0% |
| src/place/mod.rs                    |         35 |         35 |     100.0% |
| src/search.rs                       |         26 |         26 |     100.0% |
| src/service.rs                      |         74 |         74 |     100.0% |
| src/stream/mod.rs                   |         49 |         49 |     100.0% |
| src/tweet/fun.rs                    |         15 |         15 |     100.0% |
| src/tweet/mod.rs                    |         73 |         73 |     100.0% |
| src/user/fun.rs                     |         24 |         24 |     100.0% |
| src/user/mod.rs                     |         87 |         87 |     100.0% |
+-------------------------------------+------------+------------+------------+
| Total                               |        634 |        634 |     100.0% |
+-------------------------------------+------------+------------+------------+

Trait implementations are not counted because by default they "inherit" the docs from the trait, even though an impl can override those docs. Similarly, inherent impl blocks are not counted at all, because for the majority of cases such docs are not useful. (The usual pattern for inherent impl blocks is to throw all the methods on a type into a single impl block. Any docs you would put on that block would be better served on the type itself.)

In addition, --show-coverage can be combined with --document-private-items to get the coverage counts for everything in the crate, not just public items.

The coverage calculation is implemented as a late pass and two new sets of passes which strip out most of the work that rustdoc otherwise does when generating docs. The is because after the new pass is executed, rustdoc immediately closes instead of going on to generate documentation.

Many examples of coverage calculations have been included as rustdoc-ui tests.

r? @rust-lang/rustdoc

@QuietMisdreavus

This comment has been minimized.

Copy link
Member Author

QuietMisdreavus commented Feb 21, 2019

cc @nikomatsakis This is the "doc coverage" feature i've been working on. I don't remember who else in T-compiler was interested in seeing this, though.

@QuietMisdreavus

This comment has been minimized.

Copy link
Member Author

QuietMisdreavus commented Feb 21, 2019

One thing i forgot to mention: One sizable hole in this feature right now is the fact that items that came from macros are fully counted. This means things like #[derive(Serialize)], whose implementation is mostly out of the crate author's control, is considered an undocumented trait impl. This is another major reason why trait impls were counted separately. I was able to get built-in derive macros filtered out, because they all tag their impls with #[automatically_derived]. Anything from a proc-macro doesn't have this luxury, though.

@Centril

This comment has been minimized.

Copy link
Contributor

Centril commented Feb 21, 2019

This seems nice. However... as a user of rustdoc, I think I would approach coverage by folder and file rather than by Rust construct. Knowing how many traits are covered compared to how many structs are isn't all that interesting to me... Rather, I would like to know whether a certain module lacks documentation and how substantial the documentation is. A tree structure with folders and files (use a heuristic to decide how much info to show) akin to as seen in https://github.com/cgag/loc with --files would I think be helpful.

@GuillaumeGomez

This comment has been minimized.

Copy link
Member

GuillaumeGomez commented Feb 22, 2019

Another I'd add would be a sentence about adding #![deny(missing_docs)] so users can see where it's missing exactly.

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Feb 27, 2019

☔️ The latest upstream changes (presumably #58777) made this pull request unmergeable. Please resolve the merge conflicts.

@QuietMisdreavus QuietMisdreavus force-pushed the QuietMisdreavus:doc-coverage branch from 6df07d9 to 74cf1ad Feb 28, 2019

@QuietMisdreavus

This comment has been minimized.

Copy link
Member Author

QuietMisdreavus commented Feb 28, 2019

Updated and force-pushed. (I was hoping that pushing the commits first before rebasing would preserve their history, but apparently not. >_>) The last two commits are the ones that fix review comments. I've updated the PR description with the new table layout, but i have not updated the tests. I probably need to scrap most of them now that i'm not counting items separately.

@QuietMisdreavus

This comment has been minimized.

Copy link
Member Author

QuietMisdreavus commented Feb 28, 2019

Pushed an update to the tests. Two things of note:

  1. Since we print the filenames in the output, these tests might be flaky since they're made relative to the current directory if possible. However, since we cut off the name if it's longer than 35 bytes, it may not be a problem, since the names of these files after src/ are all longer than 35 bytes. We'll see how they fare in travis.
  2. Primitives and keywords don't get the right span information when they're cleaned? The exotic.rs test reports two items from the file <anon>, which corresponds to the primitive and keyword which are documented in the test. I'm not sure this is a problem? Since these are only ever used by the standard library, it's not likely to be seen outside of this repo.
Show resolved Hide resolved src/librustdoc/config.rs Outdated
@@ -391,7 +396,14 @@ fn main_args(args: &[String]) -> isize {
let diag_opts = (options.error_format,
options.debugging_options.treat_err_as_bug,
options.debugging_options.ui_testing);
let show_coverage = options.show_coverage;
rust_input(options, move |out| {

This comment has been minimized.

@GuillaumeGomez

GuillaumeGomez Mar 5, 2019

Member

Is it necessary to call rust_input in show_coverage case?

This comment has been minimized.

@QuietMisdreavus

QuietMisdreavus Mar 5, 2019

Author Member

rust_input is what actually calls the core::run documentation loading code, so i wanted to go through that to make sure that it used the same cleaning/filtering process that regular docs loading did.

This comment has been minimized.

@GuillaumeGomez

GuillaumeGomez Mar 5, 2019

Member

Seems a bit overkill but let's go along for now...

@GuillaumeGomez

This comment has been minimized.

Copy link
Member

GuillaumeGomez commented Mar 5, 2019

Apart from my comments, looks good to me!

@GuillaumeGomez

This comment has been minimized.

Copy link
Member

GuillaumeGomez commented Mar 5, 2019

Thanks! Once CI passed, r=me.

@QuietMisdreavus

This comment has been minimized.

Copy link
Member Author

QuietMisdreavus commented Mar 5, 2019

@bors r=GuillaumeGomez

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Mar 5, 2019

📌 Commit 3df0b89 has been approved by GuillaumeGomez

Centril added a commit to Centril/rust that referenced this pull request Mar 9, 2019

Rollup merge of rust-lang#58626 - QuietMisdreavus:doc-coverage, r=Gui…
…llaumeGomez

rustdoc: add option to calculate "documentation coverage"

This PR adds a new flag to rustdoc, `--show-coverage`. When passed, this flag will make rustdoc count the number of items in a crate with documentation instead of generating docs. This count will be output as a table of each file in the crate, like this (when run on my crate `egg-mode`):

```
+-------------------------------------+------------+------------+------------+
| File                                | Documented |      Total | Percentage |
+-------------------------------------+------------+------------+------------+
| src/auth.rs                         |         16 |         16 |     100.0% |
| src/common/mod.rs                   |          1 |          1 |     100.0% |
| src/common/response.rs              |          9 |          9 |     100.0% |
| src/cursor.rs                       |         24 |         24 |     100.0% |
| src/direct/fun.rs                   |          6 |          6 |     100.0% |
| src/direct/mod.rs                   |         41 |         41 |     100.0% |
| src/entities.rs                     |         50 |         50 |     100.0% |
| src/error.rs                        |         27 |         27 |     100.0% |
| src/lib.rs                          |          1 |          1 |     100.0% |
| src/list/fun.rs                     |         19 |         19 |     100.0% |
| src/list/mod.rs                     |         22 |         22 |     100.0% |
| src/media/mod.rs                    |         27 |         27 |     100.0% |
| src/place/fun.rs                    |          8 |          8 |     100.0% |
| src/place/mod.rs                    |         35 |         35 |     100.0% |
| src/search.rs                       |         26 |         26 |     100.0% |
| src/service.rs                      |         74 |         74 |     100.0% |
| src/stream/mod.rs                   |         49 |         49 |     100.0% |
| src/tweet/fun.rs                    |         15 |         15 |     100.0% |
| src/tweet/mod.rs                    |         73 |         73 |     100.0% |
| src/user/fun.rs                     |         24 |         24 |     100.0% |
| src/user/mod.rs                     |         87 |         87 |     100.0% |
+-------------------------------------+------------+------------+------------+
| Total                               |        634 |        634 |     100.0% |
+-------------------------------------+------------+------------+------------+
```

Trait implementations are not counted because by default they "inherit" the docs from the trait, even though an impl can override those docs. Similarly, inherent impl blocks are not counted at all, because for the majority of cases such docs are not useful. (The usual pattern for inherent impl blocks is to throw all the methods on a type into a single impl block. Any docs you would put on that block would be better served on the type itself.)

In addition, `--show-coverage` can be combined with `--document-private-items` to get the coverage counts for everything in the crate, not just public items.

The coverage calculation is implemented as a late pass and two new sets of passes which strip out most of the work that rustdoc otherwise does when generating docs. The is because after the new pass is executed, rustdoc immediately closes instead of going on to generate documentation.

Many examples of coverage calculations have been included as `rustdoc-ui` tests.

r? @rust-lang/rustdoc

bors added a commit that referenced this pull request Mar 9, 2019

Auto merge of #59043 - Centril:rollup, r=Centril
Rollup of 14 pull requests

Successful merges:

 - #58518 (Use early unwraps instead of bubbling up errors just to unwrap in the end)
 - #58626 (rustdoc: add option to calculate "documentation coverage")
 - #58629 (rust-lldb: fix crash when printing empty string)
 - #58660 (MaybeUninit: add read_initialized, add examples)
 - #58670 (fixes #52482)
 - #58676 (look for python2 symlinks before bootstrap python)
 - #58679 (Refactor passes and pass execution to be more parallel)
 - #58743 (Add tests for several E-needstest issues)
 - #58750 (Make `Unique::as_ptr`, `NonNull::dangling` and `NonNull::cast` const)
 - #58762 (Mention `unwind(aborts)` in diagnostics for `#[unwind]`)
 - #58900 (Ignore should_panic tests on android)
 - #58924 (Add as_slice() to slice::IterMut and vec::Drain)
 - #58990 (Actually publish miri in the manifest)
 - #59018 (std: Delete a by-definition spuriously failing test)

Failed merges:

r? @ghost

Centril added a commit to Centril/rust that referenced this pull request Mar 9, 2019

Rollup merge of rust-lang#58626 - QuietMisdreavus:doc-coverage, r=Gui…
…llaumeGomez

rustdoc: add option to calculate "documentation coverage"

This PR adds a new flag to rustdoc, `--show-coverage`. When passed, this flag will make rustdoc count the number of items in a crate with documentation instead of generating docs. This count will be output as a table of each file in the crate, like this (when run on my crate `egg-mode`):

```
+-------------------------------------+------------+------------+------------+
| File                                | Documented |      Total | Percentage |
+-------------------------------------+------------+------------+------------+
| src/auth.rs                         |         16 |         16 |     100.0% |
| src/common/mod.rs                   |          1 |          1 |     100.0% |
| src/common/response.rs              |          9 |          9 |     100.0% |
| src/cursor.rs                       |         24 |         24 |     100.0% |
| src/direct/fun.rs                   |          6 |          6 |     100.0% |
| src/direct/mod.rs                   |         41 |         41 |     100.0% |
| src/entities.rs                     |         50 |         50 |     100.0% |
| src/error.rs                        |         27 |         27 |     100.0% |
| src/lib.rs                          |          1 |          1 |     100.0% |
| src/list/fun.rs                     |         19 |         19 |     100.0% |
| src/list/mod.rs                     |         22 |         22 |     100.0% |
| src/media/mod.rs                    |         27 |         27 |     100.0% |
| src/place/fun.rs                    |          8 |          8 |     100.0% |
| src/place/mod.rs                    |         35 |         35 |     100.0% |
| src/search.rs                       |         26 |         26 |     100.0% |
| src/service.rs                      |         74 |         74 |     100.0% |
| src/stream/mod.rs                   |         49 |         49 |     100.0% |
| src/tweet/fun.rs                    |         15 |         15 |     100.0% |
| src/tweet/mod.rs                    |         73 |         73 |     100.0% |
| src/user/fun.rs                     |         24 |         24 |     100.0% |
| src/user/mod.rs                     |         87 |         87 |     100.0% |
+-------------------------------------+------------+------------+------------+
| Total                               |        634 |        634 |     100.0% |
+-------------------------------------+------------+------------+------------+
```

Trait implementations are not counted because by default they "inherit" the docs from the trait, even though an impl can override those docs. Similarly, inherent impl blocks are not counted at all, because for the majority of cases such docs are not useful. (The usual pattern for inherent impl blocks is to throw all the methods on a type into a single impl block. Any docs you would put on that block would be better served on the type itself.)

In addition, `--show-coverage` can be combined with `--document-private-items` to get the coverage counts for everything in the crate, not just public items.

The coverage calculation is implemented as a late pass and two new sets of passes which strip out most of the work that rustdoc otherwise does when generating docs. The is because after the new pass is executed, rustdoc immediately closes instead of going on to generate documentation.

Many examples of coverage calculations have been included as `rustdoc-ui` tests.

r? @rust-lang/rustdoc

bors added a commit that referenced this pull request Mar 9, 2019

Auto merge of #59046 - Centril:rollup, r=Centril
Rollup of 13 pull requests

Successful merges:

 - #58518 (Use early unwraps instead of bubbling up errors just to unwrap in the end)
 - #58626 (rustdoc: add option to calculate "documentation coverage")
 - #58629 (rust-lldb: fix crash when printing empty string)
 - #58660 (MaybeUninit: add read_initialized, add examples)
 - #58670 (fixes #52482)
 - #58676 (look for python2 symlinks before bootstrap python)
 - #58679 (Refactor passes and pass execution to be more parallel)
 - #58743 (Add tests for several E-needstest issues)
 - #58750 (Make `Unique::as_ptr`, `NonNull::dangling` and `NonNull::cast` const)
 - #58762 (Mention `unwind(aborts)` in diagnostics for `#[unwind]`)
 - #58924 (Add as_slice() to slice::IterMut and vec::Drain)
 - #58990 (Actually publish miri in the manifest)
 - #59018 (std: Delete a by-definition spuriously failing test)

Failed merges:

r? @ghost

Centril added a commit to Centril/rust that referenced this pull request Mar 9, 2019

Rollup merge of rust-lang#58626 - QuietMisdreavus:doc-coverage, r=Gui…
…llaumeGomez

rustdoc: add option to calculate "documentation coverage"

This PR adds a new flag to rustdoc, `--show-coverage`. When passed, this flag will make rustdoc count the number of items in a crate with documentation instead of generating docs. This count will be output as a table of each file in the crate, like this (when run on my crate `egg-mode`):

```
+-------------------------------------+------------+------------+------------+
| File                                | Documented |      Total | Percentage |
+-------------------------------------+------------+------------+------------+
| src/auth.rs                         |         16 |         16 |     100.0% |
| src/common/mod.rs                   |          1 |          1 |     100.0% |
| src/common/response.rs              |          9 |          9 |     100.0% |
| src/cursor.rs                       |         24 |         24 |     100.0% |
| src/direct/fun.rs                   |          6 |          6 |     100.0% |
| src/direct/mod.rs                   |         41 |         41 |     100.0% |
| src/entities.rs                     |         50 |         50 |     100.0% |
| src/error.rs                        |         27 |         27 |     100.0% |
| src/lib.rs                          |          1 |          1 |     100.0% |
| src/list/fun.rs                     |         19 |         19 |     100.0% |
| src/list/mod.rs                     |         22 |         22 |     100.0% |
| src/media/mod.rs                    |         27 |         27 |     100.0% |
| src/place/fun.rs                    |          8 |          8 |     100.0% |
| src/place/mod.rs                    |         35 |         35 |     100.0% |
| src/search.rs                       |         26 |         26 |     100.0% |
| src/service.rs                      |         74 |         74 |     100.0% |
| src/stream/mod.rs                   |         49 |         49 |     100.0% |
| src/tweet/fun.rs                    |         15 |         15 |     100.0% |
| src/tweet/mod.rs                    |         73 |         73 |     100.0% |
| src/user/fun.rs                     |         24 |         24 |     100.0% |
| src/user/mod.rs                     |         87 |         87 |     100.0% |
+-------------------------------------+------------+------------+------------+
| Total                               |        634 |        634 |     100.0% |
+-------------------------------------+------------+------------+------------+
```

Trait implementations are not counted because by default they "inherit" the docs from the trait, even though an impl can override those docs. Similarly, inherent impl blocks are not counted at all, because for the majority of cases such docs are not useful. (The usual pattern for inherent impl blocks is to throw all the methods on a type into a single impl block. Any docs you would put on that block would be better served on the type itself.)

In addition, `--show-coverage` can be combined with `--document-private-items` to get the coverage counts for everything in the crate, not just public items.

The coverage calculation is implemented as a late pass and two new sets of passes which strip out most of the work that rustdoc otherwise does when generating docs. The is because after the new pass is executed, rustdoc immediately closes instead of going on to generate documentation.

Many examples of coverage calculations have been included as `rustdoc-ui` tests.

r? @rust-lang/rustdoc

bors added a commit that referenced this pull request Mar 9, 2019

Auto merge of #59050 - Centril:rollup, r=Centril
Rollup of 13 pull requests

Successful merges:

 - #58518 (Use early unwraps instead of bubbling up errors just to unwrap in the end)
 - #58626 (rustdoc: add option to calculate "documentation coverage")
 - #58629 (rust-lldb: fix crash when printing empty string)
 - #58660 (MaybeUninit: add read_initialized, add examples)
 - #58670 (fixes #52482)
 - #58676 (look for python2 symlinks before bootstrap python)
 - #58679 (Refactor passes and pass execution to be more parallel)
 - #58750 (Make `Unique::as_ptr`, `NonNull::dangling` and `NonNull::cast` const)
 - #58762 (Mention `unwind(aborts)` in diagnostics for `#[unwind]`)
 - #58924 (Add as_slice() to slice::IterMut and vec::Drain)
 - #58990 (Actually publish miri in the manifest)
 - #59018 (std: Delete a by-definition spuriously failing test)
 - #59045 (Expose new_sub_parser_from_file)

Failed merges:

r? @ghost

bors added a commit that referenced this pull request Mar 9, 2019

Auto merge of #59050 - Centril:rollup, r=Centril
Rollup of 13 pull requests

Successful merges:

 - #58518 (Use early unwraps instead of bubbling up errors just to unwrap in the end)
 - #58626 (rustdoc: add option to calculate "documentation coverage")
 - #58629 (rust-lldb: fix crash when printing empty string)
 - #58660 (MaybeUninit: add read_initialized, add examples)
 - #58670 (fixes #52482)
 - #58676 (look for python2 symlinks before bootstrap python)
 - #58679 (Refactor passes and pass execution to be more parallel)
 - #58750 (Make `Unique::as_ptr`, `NonNull::dangling` and `NonNull::cast` const)
 - #58762 (Mention `unwind(aborts)` in diagnostics for `#[unwind]`)
 - #58924 (Add as_slice() to slice::IterMut and vec::Drain)
 - #58990 (Actually publish miri in the manifest)
 - #59018 (std: Delete a by-definition spuriously failing test)
 - #59045 (Expose new_sub_parser_from_file)

Failed merges:

r? @ghost

@bors bors merged commit 3df0b89 into rust-lang:master Mar 9, 2019

@QuietMisdreavus QuietMisdreavus deleted the QuietMisdreavus:doc-coverage branch Mar 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.