From 41920f7106150a783336046e8d58633f38a961a2 Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Wed, 5 Nov 2025 20:25:39 +0200 Subject: [PATCH 1/8] sembr src/rustdoc.md --- src/rustdoc.md | 128 ++++++++++++++++++++++++------------------------- 1 file changed, 64 insertions(+), 64 deletions(-) diff --git a/src/rustdoc.md b/src/rustdoc.md index aba80131d..9e2fe07ea 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -1,28 +1,26 @@ # Rustdoc overview -`rustdoc` lives in-tree with the -compiler and standard library. This chapter is about how it works. +`rustdoc` lives in-tree with the compiler and standard library. +This chapter is about how it works. For information about Rustdoc's features and how to use them, see the [Rustdoc book](https://doc.rust-lang.org/nightly/rustdoc/). -For more details about how rustdoc works, see the -["Rustdoc internals" chapter][Rustdoc internals]. +For more details about how rustdoc works, see the ["Rustdoc internals" chapter][Rustdoc internals]. [Rustdoc internals]: ./rustdoc-internals.md `rustdoc` uses `rustc` internals (and, of course, the standard library), so you will have to build the compiler and `std` once before you can build `rustdoc`. -Rustdoc is implemented entirely within the crate [`librustdoc`][rd]. It runs -the compiler up to the point where we have an internal representation of a -crate (HIR) and the ability to run some queries about the types of items. [HIR] -and [queries] are discussed in the linked chapters. +Rustdoc is implemented entirely within the crate [`librustdoc`][rd]. +It runs the compiler up to the point where we have an internal representation of a +crate (HIR) and the ability to run some queries about the types of items. +[HIR] and [queries] are discussed in the linked chapters. [HIR]: ./hir.md [queries]: ./query.md [rd]: https://github.com/rust-lang/rust/tree/HEAD/src/librustdoc -`librustdoc` performs two major steps after that to render a set of -documentation: +`librustdoc` performs two major steps after that to render a set of documentation: * "Clean" the AST into a form that's more suited to creating documentation (and slightly more resistant to churn in the compiler). @@ -31,9 +29,9 @@ documentation: Naturally, there's more than just this, and those descriptions simplify out lots of details, but that's the high-level overview. -(Side note: `librustdoc` is a library crate! The `rustdoc` binary is created -using the project in [`src/tools/rustdoc`][bin]. Note that literally all that -does is call the `main()` that's in this crate's `lib.rs`, though.) +(Side note: `librustdoc` is a library crate! +The `rustdoc` binary is created using the project in [`src/tools/rustdoc`][bin]. +Note that literally all that does is call the `main()` that's in this crate's `lib.rs`, though.) [bin]: https://github.com/rust-lang/rust/tree/HEAD/src/tools/rustdoc @@ -47,15 +45,14 @@ does is call the `main()` that's in this crate's `lib.rs`, though.) rustdoc you can run on other projects. * Add `library/test` to be able to use `rustdoc --test`. * Run `rustup toolchain link stage2 build/host/stage2` to add a - custom toolchain called `stage2` to your rustup environment. After - running that, `cargo +stage2 doc` in any directory will build with + custom toolchain called `stage2` to your rustup environment. + After running that, `cargo +stage2 doc` in any directory will build with your locally-compiled rustdoc. * Use `./x doc library` to use this rustdoc to generate the standard library docs. * The completed docs will be available in `build/host/doc` (under `core`, `alloc`, and `std`). * If you want to copy those docs to a webserver, copy all of - `build/host/doc`, since that's where the CSS, JS, fonts, and landing - page are. + `build/host/doc`, since that's where the CSS, JS, fonts, and landing page are. * For frontend debugging, disable the `rust.docs-minification` option in [`bootstrap.toml`]. * Use `./x test tests/rustdoc*` to run the tests using a stage1 rustdoc. @@ -65,7 +62,8 @@ does is call the `main()` that's in this crate's `lib.rs`, though.) ### JavaScript CI checks -Rustdoc’s JavaScript and TypeScript are checked during CI by `eslint`, `es-check`, and `tsc` (not by compiletest). These run as part of the `tidy` job. +Rustdoc’s JavaScript and TypeScript are checked during CI by `eslint`, `es-check`, and `tsc` (not by compiletest). +These run as part of the `tidy` job. ```bash ./x.py test tidy --extra-checks=js @@ -103,22 +101,24 @@ See [Rustdoc tests suites](tests/compiletest.md#rustdoc-test-suites) for more de ## Constraints -We try to make rustdoc work reasonably well with JavaScript disabled, and when -browsing local files. We support +We try to make rustdoc work reasonably well with JavaScript disabled, and when browsing local files. +We support [these browsers](https://rust-lang.github.io/rfcs/1985-tiered-browser-support.html#supported-browsers). Supporting local files (`file:///` URLs) brings some surprising restrictions. Certain browser features that require secure origins, like `localStorage` and -Service Workers, don't work reliably. We can still use such features but we -should make sure pages are still usable without them. +Service Workers, don't work reliably. +We can still use such features but we should make sure pages are still usable without them. Rustdoc [does not type-check function bodies][platform-specific docs]. This works by [overriding the built-in queries for typeck][override queries], by [silencing name resolution errors], and by [not resolving opaque types]. This comes with several caveats: in particular, rustdoc *cannot* run any parts of the compiler that -require type-checking bodies; for example it cannot generate `.rlib` files or run most lints. +require type-checking bodies; +for example it cannot generate `.rlib` files or run most lints. We want to move away from this model eventually, but we need some alternative for -[the people using it][async-std]; see [various][zulip stop accepting broken code] +[the people using it][async-std]; +see [various][zulip stop accepting broken code] [previous][rustdoc meeting 2024-07-08] [Zulip][compiler meeting 2023-01-26] [discussion][notriddle rfc]. For examples of code that breaks if this hack is removed, see [`tests/rustdoc-ui/error-in-impl-trait`]. @@ -136,9 +136,9 @@ For examples of code that breaks if this hack is removed, see ## Multiple runs, same output directory -Rustdoc can be run multiple times for varying inputs, with its output set to the -same directory. That's how cargo produces documentation for dependencies of the -current crate. It can also be done manually if a user wants a big +Rustdoc can be run multiple times for varying inputs, with its output set to the same directory. +That's how cargo produces documentation for dependencies of the current crate. +It can also be done manually if a user wants a big documentation bundle with all of the docs they care about. HTML is generated independently for each crate, but there is some cross-crate @@ -147,68 +147,68 @@ information that we update as we add crates to the output directory: - `crates.js` holds a list of all crates in the output directory. - `search-index.js` holds a list of all searchable items. - For each trait, there is a file under `implementors/.../trait.TraitName.js` - containing a list of implementors of that trait. The implementors may be in - different crates than the trait, and the JS file is updated as we discover - new ones. + containing a list of implementors of that trait. + The implementors may be in + different crates than the trait, and the JS file is updated as we discover new ones. ## Use cases -There are a few major use cases for rustdoc that you should keep in mind when -working on it: +There are a few major use cases for rustdoc that you should keep in mind when working on it: ### Standard library docs -These are published at as part of the Rust release -process. Stable releases are also uploaded to specific versioned URLs like -. Beta and nightly docs are published to +These are published at as part of the Rust release process. +Stable releases are also uploaded to specific versioned URLs like +. +Beta and nightly docs are published to and . The docs are uploaded with the [promote-release -tool](https://github.com/rust-lang/promote-release) and served from S3 with -CloudFront. +tool](https://github.com/rust-lang/promote-release) and served from S3 with CloudFront. -The standard library docs contain five crates: alloc, core, proc_macro, std, and -test. +The standard library docs contain five crates: alloc, core, proc_macro, std, and test. ### docs.rs When crates are published to crates.io, docs.rs automatically builds -and publishes their documentation, for instance at -. It always builds with the current nightly +and publishes their documentation, for instance at . +It always builds with the current nightly rustdoc, so any changes you land in rustdoc are "insta-stable" in that they will -have an immediate public effect on docs.rs. Old documentation is only sometimes rebuilt, so -you will see some variation in UI when browsing old releases in docs.rs. Crate -authors can request rebuilds, which will be run with the latest rustdoc. +have an immediate public effect on docs.rs. +Old documentation is only sometimes rebuilt, so +you will see some variation in UI when browsing old releases in docs.rs. +Crate authors can request rebuilds, which will be run with the latest rustdoc. Docs.rs performs some transformations on rustdoc's output in order to save -storage and display a navigation bar at the top. In particular, certain static +storage and display a navigation bar at the top. +In particular, certain static files, like main.js and rustdoc.css, may be shared across multiple invocations -of the same version of rustdoc. Others, like crates.js and sidebar-items.js, are -different for different invocations. Still others, like fonts, will never -change. These categories are distinguished using the `SharedResource` enum in +of the same version of rustdoc. +Others, like crates.js and sidebar-items.js, are different for different invocations. +Still others, like fonts, will never change. +These categories are distinguished using the `SharedResource` enum in `src/librustdoc/html/render/write_shared.rs` Documentation on docs.rs is always generated for a single crate at a time, so -the search and sidebar functionality don't include dependencies of the current -crate. +the search and sidebar functionality don't include dependencies of the current crate. ### Locally generated docs -Crate authors can run `cargo doc --open` in crates they have checked -out locally to see the docs. This is useful to check that the docs they -are writing are useful and display correctly. It can also be useful for -people to view documentation on crates they aren't authors of, but want to -use. In both cases, people may use `--document-private-items` Cargo flag to +Crate authors can run `cargo doc --open` in crates they have checked out locally to see the docs. +This is useful to check that the docs they are writing are useful and display correctly. +It can also be useful for people to view documentation on crates they aren't authors of, but want to +use. +In both cases, people may use `--document-private-items` Cargo flag to see private methods, fields, and so on, which are normally not displayed. -By default `cargo doc` will generate documentation for a crate and all of its -dependencies. That can result in a very large documentation bundle, with a large -(and slow) search corpus. The Cargo flag `--no-deps` inhibits that behavior and -generates docs for just the crate. +By default `cargo doc` will generate documentation for a crate and all of its dependencies. +That can result in a very large documentation bundle, with a large (and slow) search corpus. +The Cargo flag `--no-deps` inhibits that behavior and generates docs for just the crate. ### Self-hosted project docs -Some projects like to host their own documentation. For example: -. This is easy to do by locally generating docs, and -simply copying them to a web server. Rustdoc's HTML output can be extensively -customized by flags. Users can add a theme, set the default theme, and inject -arbitrary HTML. See `rustdoc --help` for details. +Some projects like to host their own documentation. +For example: . +This is easy to do by locally generating docs, and simply copying them to a web server. +Rustdoc's HTML output can be extensively customized by flags. +Users can add a theme, set the default theme, and inject arbitrary HTML. +See `rustdoc --help` for details. From 59fc8f9e099a2db188bac2c66d145bbe6c50c4d4 Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Wed, 5 Nov 2025 20:32:19 +0200 Subject: [PATCH 2/8] use sentence case for titles --- src/rustdoc-internals.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md index 55b08e889..927bffc59 100644 --- a/src/rustdoc-internals.md +++ b/src/rustdoc-internals.md @@ -1,11 +1,11 @@ -# Rustdoc Internals +# Rustdoc internals This page describes [`rustdoc`]'s passes and modes. For an overview of `rustdoc`, see the ["Rustdoc overview" chapter](./rustdoc.md). [`rustdoc`]: https://github.com/rust-lang/rust/tree/HEAD/src/tools/rustdoc -## From Crate to Clean +## From crate to clean In [`core.rs`] are two central items: the [`rustdoc::core::DocContext`] `struct`, and the [`rustdoc::core::run_global_ctxt`] function. The latter is @@ -153,7 +153,7 @@ itself. [`librustdoc/passes`]: https://github.com/rust-lang/rust/tree/HEAD/src/librustdoc/passes [`stripper`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/passes/stripper/index.html -## From Clean To HTML +## From clean to HTML This is where the "second phase" in `rustdoc` begins. This phase primarily lives in the [`librustdoc/formats`] and [`librustdoc/html`] folders, and it all starts with @@ -247,7 +247,7 @@ generation, as well as just keeping things organized: [didn't used to be the case]: https://github.com/rust-lang/rust/pull/80090 [`SharedContext`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/html/render/context/struct.SharedContext.html -## Other Tricks Up Its Sleeve +## Other tricks up its sleeve All this describes the process for generating `HTML` documentation from a Rust crate, but there are couple other major modes that `rustdoc` runs in. It can also @@ -268,7 +268,7 @@ in `test.rs` is the function `make_test`, which is where hand-written Some extra reading about `make_test` can be found [here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/). -## Testing Locally +## Testing locally Some features of the generated `HTML` documentation might require local storage to be used across pages, which doesn't work well without an `HTTP` @@ -284,7 +284,7 @@ $ python3 -m http.server -d build/[YOUR ARCH]/doc Now you can browse your documentation just like you would if it was hosted on the internet. For example, the url for `std` will be `rust/std/`. -## See Also +## See also - The [`rustdoc` api docs] - [An overview of `rustdoc`](./rustdoc.md) From b443c9b7238d769c149f007320bb9f247c57a37a Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Wed, 5 Nov 2025 20:56:16 +0200 Subject: [PATCH 3/8] some text improvements --- src/rustdoc.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/rustdoc.md b/src/rustdoc.md index 9e2fe07ea..1be0ae369 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -102,24 +102,23 @@ See [Rustdoc tests suites](tests/compiletest.md#rustdoc-test-suites) for more de ## Constraints We try to make rustdoc work reasonably well with JavaScript disabled, and when browsing local files. -We support -[these browsers](https://rust-lang.github.io/rfcs/1985-tiered-browser-support.html#supported-browsers). +We have [a list of supported browsers]. Supporting local files (`file:///` URLs) brings some surprising restrictions. Certain browser features that require secure origins, like `localStorage` and Service Workers, don't work reliably. -We can still use such features but we should make sure pages are still usable without them. +We can still use such features, but we should make sure pages are still usable without them. Rustdoc [does not type-check function bodies][platform-specific docs]. This works by [overriding the built-in queries for typeck][override queries], by [silencing name resolution errors], and by [not resolving opaque types]. This comes with several caveats: in particular, rustdoc *cannot* run any parts of the compiler that require type-checking bodies; -for example it cannot generate `.rlib` files or run most lints. +for example, it cannot generate `.rlib` files or run most lints. We want to move away from this model eventually, but we need some alternative for [the people using it][async-std]; see [various][zulip stop accepting broken code] -[previous][rustdoc meeting 2024-07-08] [Zulip][compiler meeting 2023-01-26] [discussion][notriddle rfc]. +[previous][rustdoc meeting 2024-07-08] [Zulip][compiler meeting 2023-01-26] [discussions][notriddle rfc]. For examples of code that breaks if this hack is removed, see [`tests/rustdoc-ui/error-in-impl-trait`]. @@ -133,6 +132,7 @@ For examples of code that breaks if this hack is removed, see [zulip stop accepting broken code]: https://rust-lang.zulipchat.com/#narrow/stream/266220-rustdoc/topic/stop.20accepting.20broken.20code [notriddle rfc]: https://rust-lang.zulipchat.com/#narrow/channel/266220-t-rustdoc/topic/Pre-RFC.3A.20stop.20accepting.20broken.20code [`tests/rustdoc-ui/error-in-impl-trait`]: https://github.com/rust-lang/rust/tree/163cb4ea3f0ae3bc7921cc259a08a7bf92e73ee6/tests/rustdoc-ui/error-in-impl-trait +[a list of supported browsers]: https://rust-lang.github.io/rfcs/1985-tiered-browser-support.html#supported-browsers ## Multiple runs, same output directory @@ -200,7 +200,7 @@ use. In both cases, people may use `--document-private-items` Cargo flag to see private methods, fields, and so on, which are normally not displayed. -By default `cargo doc` will generate documentation for a crate and all of its dependencies. +By default, `cargo doc` will generate documentation for a crate and all of its dependencies. That can result in a very large documentation bundle, with a large (and slow) search corpus. The Cargo flag `--no-deps` inhibits that behavior and generates docs for just the crate. From 1d07bf1d4f3cce96639a5a407d636a63c8857007 Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Wed, 5 Nov 2025 20:56:59 +0200 Subject: [PATCH 4/8] that redirects to docs.rs --- src/rustdoc.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/rustdoc.md b/src/rustdoc.md index 1be0ae369..0e673bf4f 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -206,8 +206,7 @@ The Cargo flag `--no-deps` inhibits that behavior and generates docs for just th ### Self-hosted project docs -Some projects like to host their own documentation. -For example: . +Some projects host their own documentation. This is easy to do by locally generating docs, and simply copying them to a web server. Rustdoc's HTML output can be extensively customized by flags. Users can add a theme, set the default theme, and inject arbitrary HTML. From 3708d54f6d9ffe6e036cbdd441a502def6e3bfb6 Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Wed, 5 Nov 2025 21:02:44 +0200 Subject: [PATCH 5/8] sembr src/rustdoc-internals.md --- src/rustdoc-internals.md | 141 +++++++++++++++++++++------------------ 1 file changed, 75 insertions(+), 66 deletions(-) diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md index 927bffc59..199a75e99 100644 --- a/src/rustdoc-internals.md +++ b/src/rustdoc-internals.md @@ -1,22 +1,23 @@ # Rustdoc internals -This page describes [`rustdoc`]'s passes and modes. For an overview of `rustdoc`, -see the ["Rustdoc overview" chapter](./rustdoc.md). +This page describes [`rustdoc`]'s passes and modes. +For an overview of `rustdoc`, see the ["Rustdoc overview" chapter](./rustdoc.md). [`rustdoc`]: https://github.com/rust-lang/rust/tree/HEAD/src/tools/rustdoc ## From crate to clean In [`core.rs`] are two central items: the [`rustdoc::core::DocContext`] -`struct`, and the [`rustdoc::core::run_global_ctxt`] function. The latter is -where `rustdoc` calls out to `rustc` to compile a crate to the point where -`rustdoc` can take over. The former is a state container used when crawling -through a crate to gather its documentation. +`struct`, and the [`rustdoc::core::run_global_ctxt`] function. +The latter is where `rustdoc` calls out to `rustc` to compile a crate to the point where +`rustdoc` can take over. +The former is a state container used when crawling through a crate to gather its documentation. The main process of crate crawling is done in [`clean/mod.rs`] through several -functions with names that start with `clean_`. Each function accepts an `hir` -or `ty` data structure, and outputs a `clean` structure used by `rustdoc`. For -example, [this function for converting lifetimes]: +functions with names that start with `clean_`. +Each function accepts an `hir` +or `ty` data structure, and outputs a `clean` structure used by `rustdoc`. +For example, [this function for converting lifetimes]: ```rust,ignore fn clean_lifetime<'tcx>(lifetime: &hir::Lifetime, cx: &mut DocContext<'tcx>) -> Lifetime { @@ -34,17 +35,19 @@ fn clean_lifetime<'tcx>(lifetime: &hir::Lifetime, cx: &mut DocContext<'tcx>) -> ``` Also, `clean/mod.rs` defines the types for the "cleaned" [Abstract Syntax Tree -(`AST`)][ast] used later to render documentation pages. Each usually accompanies a +(`AST`)][ast] used later to render documentation pages. +Each usually accompanies a `clean_*` function that takes some [`AST`][ast] or [High-Level Intermediate -Representation (`HIR`)][hir] type from `rustc` and converts it into the -appropriate "cleaned" type. "Big" items like modules or associated items may +Representation (`HIR`)][hir] type from `rustc` and converts it into the appropriate "cleaned" type. +"Big" items like modules or associated items may have some extra processing in its `clean` function, but for the most part these -`impl`s are straightforward conversions. The "entry point" to this module is +`impl`s are straightforward conversions. +The "entry point" to this module is [`clean::utils::krate`][ck0], which is called by [`run_global_ctxt`]. The first step in [`clean::utils::krate`][ck1] is to invoke -[`visit_ast::RustdocVisitor`] to process the module tree into an intermediate -[`visit_ast::Module`]. This is the step that actually crawls the +[`visit_ast::RustdocVisitor`] to process the module tree into an intermediate [`visit_ast::Module`]. +This is the step that actually crawls the [`rustc_hir::Crate`], normalizing various aspects of name resolution, such as: * handling `#[doc(inline)]` and `#[doc(no_inline)]` @@ -57,14 +60,15 @@ The first step in [`clean::utils::krate`][ck1] is to invoke they're defined as a reexport or not After this step, `clean::krate` invokes [`clean_doc_module`], which actually -converts the `HIR` items to the cleaned [`AST`][ast]. This is also the step where cross- +converts the `HIR` items to the cleaned [`AST`][ast]. +This is also the step where cross- crate inlining is performed, which requires converting `rustc_middle` data structures into the cleaned [`AST`][ast]. The other major thing that happens in `clean/mod.rs` is the collection of doc comments and `#[doc=""]` attributes into a separate field of the [`Attributes`] -`struct`, present on anything that gets hand-written documentation. This makes it -easier to collect this documentation later in the process. +`struct`, present on anything that gets hand-written documentation. +This makes it easier to collect this documentation later in the process. The primary output of this process is a [`clean::types::Crate`] with a tree of [`Item`]s which describe the publicly-documentable items in the target crate. @@ -90,13 +94,14 @@ which describe the publicly-documentable items in the target crate. ### Passes Anything But a Gas Station (or: [Hot Potato](https://www.youtube.com/watch?v=WNFBIt5HxdY)) Before moving on to the next major step, a few important "passes" occur over -the cleaned [`AST`][ast]. Several of these passes are `lint`s and reports, but some of -them mutate or generate new items. +the cleaned [`AST`][ast]. +Several of these passes are `lint`s and reports, but some of them mutate or generate new items. These are all implemented in the [`librustdoc/passes`] directory, one file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping private/hidden items can be bypassed by passing -`--document-private-items` to `rustdoc`. Note that unlike the previous set of [`AST`][ast] +`--document-private-items` to `rustdoc`. +Note that unlike the previous set of [`AST`][ast] transformations, the passes are run on the _cleaned_ crate. Here is the list of passes as of March 2023: @@ -105,8 +110,7 @@ Here is the list of passes as of March 2023: flag. - `check-doc-test-visibility` runs `doctest` visibility–related `lint`s. This pass - runs before `strip-private`, which is why it needs to be separate from - `run-lints`. + runs before `strip-private`, which is why it needs to be separate from `run-lints`. - `collect-intra-doc-links` resolves [intra-doc links](https://doc.rust-lang.org/nightly/rustdoc/write-documentation/linking-to-items-by-name.html). @@ -121,8 +125,8 @@ Here is the list of passes as of March 2023: - `bare_urls` detects links that are not linkified, e.g., in Markdown such as `Go to https://example.com/.` It suggests wrapping the link with angle brackets: - `Go to .` to linkify it. This is the code behind the `rustdoc::bare_urls` `lint`. + `Go to .` to linkify it. + This is the code behind the `rustdoc::bare_urls` `lint`. - `check_code_block_syntax` validates syntax inside Rust code blocks (```rust) @@ -131,33 +135,37 @@ Here is the list of passes as of March 2023: in doc comments. - `strip-hidden` and `strip-private` strip all `doc(hidden)` and private items - from the output. `strip-private` implies `strip-priv-imports`. Basically, the - goal is to remove items that are not relevant for public documentation. This - pass is skipped when `--document-hidden-items` is passed. + from the output. + `strip-private` implies `strip-priv-imports`. + Basically, the goal is to remove items that are not relevant for public documentation. + This pass is skipped when `--document-hidden-items` is passed. - `strip-priv-imports` strips all private import statements (`use`, `extern - crate`) from a crate. This is necessary because `rustdoc` will handle *public* + crate`) from a crate. + This is necessary because `rustdoc` will handle *public* imports by either inlining the item's documentation to the module or creating - a "Reexports" section with the import in it. The pass ensures that all of - these imports are actually relevant to documentation. It is technically - only run when `--document-private-items` is passed, but `strip-private` + a "Reexports" section with the import in it. + The pass ensures that all of these imports are actually relevant to documentation. + It is technically only run when `--document-private-items` is passed, but `strip-private` accomplishes the same thing. - `strip-private` strips all private items from a crate which cannot be seen - externally. This pass is skipped when `--document-private-items` is passed. + externally. + This pass is skipped when `--document-private-items` is passed. There is also a [`stripper`] module in `librustdoc/passes`, but it is a -collection of utility functions for the `strip-*` passes and is not a pass -itself. +collection of utility functions for the `strip-*` passes and is not a pass itself. [`librustdoc/passes`]: https://github.com/rust-lang/rust/tree/HEAD/src/librustdoc/passes [`stripper`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/passes/stripper/index.html ## From clean to HTML -This is where the "second phase" in `rustdoc` begins. This phase primarily lives +This is where the "second phase" in `rustdoc` begins. +This phase primarily lives in the [`librustdoc/formats`] and [`librustdoc/html`] folders, and it all starts with -[`formats::renderer::run_format`]. This code is responsible for setting up a type that +[`formats::renderer::run_format`]. +This code is responsible for setting up a type that `impl FormatRenderer`, which for `HTML` is [`Context`]. This structure contains methods that get called by `run_format` to drive the @@ -168,8 +176,8 @@ doc rendering, which includes: * `after_krate` generates other global resources like `all.html` In `item`, the "page rendering" occurs, via a mixture of [Askama] templates -and manual `write!()` calls, starting in [`html/layout.rs`]. The parts that have -not been converted to templates occur within a series of `std::fmt::Display` +and manual `write!()` calls, starting in [`html/layout.rs`]. +The parts that have not been converted to templates occur within a series of `std::fmt::Display` implementations and functions that pass around a `&mut std::fmt::Formatter`. The parts that actually generate `HTML` from the items and documentation start @@ -183,11 +191,13 @@ pieces like "how should I print a where clause as part of some other item". Whenever `rustdoc` comes across an item that should print hand-written documentation alongside, it calls out to [`html/markdown.rs`] which interfaces -with the Markdown parser. This is exposed as a series of types that wrap a -string of Markdown, and implement `fmt::Display` to emit `HTML` text. It takes -special care to enable certain features like footnotes and tables and add +with the Markdown parser. +This is exposed as a series of types that wrap a +string of Markdown, and implement `fmt::Display` to emit `HTML` text. +It takes special care to enable certain features like footnotes and tables and add syntax highlighting to Rust code blocks (via `html/highlight.rs`) before -running the Markdown parser. There's also a function [`find_codes`] which is +running the Markdown parser. +There's also a function [`find_codes`] which is called by `find_testable_codes` that specifically scans for Rust code blocks so the test-runner code can find all the `doctest`s in the crate. @@ -208,11 +218,11 @@ the test-runner code can find all the `doctest`s in the crate. [video]: https://www.youtube.com/watch?v=hOLAGYmUQV0 It's important to note that `rustdoc` can ask the compiler for type information -directly, even during `HTML` generation. This [didn't used to be the case], and +directly, even during `HTML` generation. +This [didn't used to be the case], and a lot of `rustdoc`'s architecture was designed around not doing that, but a `TyCtxt` is now passed to `formats::renderer::run_format`, which is used to -run generation for both `HTML` and the -(unstable as of March 2023) JSON format. +run generation for both `HTML` and the (unstable as of March 2023) JSON format. This change has allowed other changes to remove data from the "clean" [`AST`][ast] that can be easily derived from `TyCtxt` queries, and we'll usually accept @@ -222,18 +232,17 @@ is complicated from two other constraints that `rustdoc` runs under: * Docs can be generated for crates that don't actually pass type checking. This is used for generating docs that cover mutually-exclusive platform configurations, such as `libstd` having a single package of docs that - cover all supported operating systems. This means `rustdoc` has to be able - to generate docs from `HIR`. + cover all supported operating systems. + This means `rustdoc` has to be able to generate docs from `HIR`. * Docs can inline across crates. Since crate metadata doesn't contain `HIR`, it must be possible to generate inlined docs from the `rustc_middle` data. -The "clean" [`AST`][ast] acts as a common output format for both input formats. There -is also some data in clean that doesn't correspond directly to `HIR`, such as -synthetic `impl`s for auto traits and blanket `impl`s generated by the -`collect-trait-impls` pass. +The "clean" [`AST`][ast] acts as a common output format for both input formats. +There is also some data in clean that doesn't correspond directly to `HIR`, such as +synthetic `impl`s for auto traits and blanket `impl`s generated by the `collect-trait-impls` pass. -Some additional data is stored in -`html::render::context::{Context, SharedContext}`. These two types serve as +Some additional data is stored in `html::render::context::{Context, SharedContext}`. +These two types serve as ways to segregate `rustdoc`'s data for an eventual future with multithreaded doc generation, as well as just keeping things organized: @@ -250,19 +259,20 @@ generation, as well as just keeping things organized: ## Other tricks up its sleeve All this describes the process for generating `HTML` documentation from a Rust -crate, but there are couple other major modes that `rustdoc` runs in. It can also -be run on a standalone Markdown file, or it can run `doctest`s on Rust code or -standalone Markdown files. For the former, it shortcuts straight to +crate, but there are couple other major modes that `rustdoc` runs in. +It can also be run on a standalone Markdown file, or it can run `doctest`s on Rust code or +standalone Markdown files. +For the former, it shortcuts straight to `html/markdown.rs`, optionally including a mode which inserts a Table of Contents to the output `HTML`. For the latter, `rustdoc` runs a similar partial-compilation to get relevant documentation in `test.rs`, but instead of going through the full clean and -render process, it runs a much simpler crate walk to grab *just* the -hand-written documentation. Combined with the aforementioned +render process, it runs a much simpler crate walk to grab *just* the hand-written documentation. +Combined with the aforementioned "`find_testable_code`" in `html/markdown.rs`, it builds up a collection of -tests to run before handing them off to the test runner. One notable location -in `test.rs` is the function `make_test`, which is where hand-written +tests to run before handing them off to the test runner. +One notable location in `test.rs` is the function `make_test`, which is where hand-written `doctest`s get transformed into something that can be executed. Some extra reading about `make_test` can be found @@ -271,9 +281,8 @@ Some extra reading about `make_test` can be found ## Testing locally Some features of the generated `HTML` documentation might require local -storage to be used across pages, which doesn't work well without an `HTTP` -server. To test these features locally, you can run a local `HTTP` server, like -this: +storage to be used across pages, which doesn't work well without an `HTTP` server. +To test these features locally, you can run a local `HTTP` server, like this: ```bash $ ./x doc library @@ -281,8 +290,8 @@ $ ./x doc library $ python3 -m http.server -d build/[YOUR ARCH]/doc ``` -Now you can browse your documentation just like you would if it was hosted -on the internet. For example, the url for `std` will be `rust/std/`. +Now you can browse your documentation just like you would if it was hosted on the internet. +For example, the url for `std` will be `rust/std/`. ## See also From 63e2cbf8938e5eabe84c3bd98edb59088b011414 Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Wed, 5 Nov 2025 21:09:24 +0200 Subject: [PATCH 6/8] some text improvements --- src/rustdoc-internals.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md index 199a75e99..40bc4aeb2 100644 --- a/src/rustdoc-internals.md +++ b/src/rustdoc-internals.md @@ -61,9 +61,8 @@ This is the step that actually crawls the After this step, `clean::krate` invokes [`clean_doc_module`], which actually converts the `HIR` items to the cleaned [`AST`][ast]. -This is also the step where cross- -crate inlining is performed, which requires converting `rustc_middle` data -structures into the cleaned [`AST`][ast]. +This is also the step where cross-crate inlining is performed, +which requires converting `rustc_middle` data structures into the cleaned [`AST`][ast]. The other major thing that happens in `clean/mod.rs` is the collection of doc comments and `#[doc=""]` attributes into a separate field of the [`Attributes`] @@ -101,7 +100,7 @@ These are all implemented in the [`librustdoc/passes`] directory, one file per p By default, all of these passes are run on a crate, but the ones regarding dropping private/hidden items can be bypassed by passing `--document-private-items` to `rustdoc`. -Note that unlike the previous set of [`AST`][ast] +Note that, unlike the previous set of [`AST`][ast] transformations, the passes are run on the _cleaned_ crate. Here is the list of passes as of March 2023: From cbac426f6a938b36ff1c2b379c434ef49782754f Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Wed, 5 Nov 2025 21:10:44 +0200 Subject: [PATCH 7/8] rustdoc json format remains unstable today --- src/rustdoc-internals.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md index 40bc4aeb2..0b75660a0 100644 --- a/src/rustdoc-internals.md +++ b/src/rustdoc-internals.md @@ -221,7 +221,7 @@ directly, even during `HTML` generation. This [didn't used to be the case], and a lot of `rustdoc`'s architecture was designed around not doing that, but a `TyCtxt` is now passed to `formats::renderer::run_format`, which is used to -run generation for both `HTML` and the (unstable as of March 2023) JSON format. +run generation for both `HTML` and the (unstable as of Nov 2025) JSON format. This change has allowed other changes to remove data from the "clean" [`AST`][ast] that can be easily derived from `TyCtxt` queries, and we'll usually accept From 43e99b1b9c2d8973556af6611ae6c1b548440735 Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Wed, 5 Nov 2025 21:17:47 +0200 Subject: [PATCH 8/8] fix code block markers --- src/rustdoc-internals.md | 2 +- src/rustdoc.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md index 0b75660a0..989a77290 100644 --- a/src/rustdoc-internals.md +++ b/src/rustdoc-internals.md @@ -283,7 +283,7 @@ Some features of the generated `HTML` documentation might require local storage to be used across pages, which doesn't work well without an `HTTP` server. To test these features locally, you can run a local `HTTP` server, like this: -```bash +```console $ ./x doc library # The documentation has been generated into `build/[YOUR ARCH]/doc`. $ python3 -m http.server -d build/[YOUR ARCH]/doc diff --git a/src/rustdoc.md b/src/rustdoc.md index 0e673bf4f..47b18f4e7 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -65,7 +65,7 @@ Note that literally all that does is call the `main()` that's in this crate's `l Rustdoc’s JavaScript and TypeScript are checked during CI by `eslint`, `es-check`, and `tsc` (not by compiletest). These run as part of the `tidy` job. -```bash +```console ./x.py test tidy --extra-checks=js ```