Implement Rustdoc #167

HeroicKatora · 2021-01-23T14:36:25Z

Adds a new include type (as an url scheme rustdoc) that runs cargo doc on a target crate with json output then processes that document. Currently only supports local absolute paths.

Example

<!-- File: `examples/rustdoc.md` -->

![](rustdoc:crate)

Relative path resolution (rustdoc:crate) resolves document relative
Absolute path (rustdoc:/crate) resolves project relative
Absolute domain (rustdoc:///path/crate) resolves an absolute file path

Inline documentation

Is rendered with the correct header level, starting with level 3. (The individual items from the documentation have level 2).

The latex article backend only supports up to level 3 headers out of the box. Added 3 more without numbering to support the nesting required for embedding typical rustdoc markdown.
no_run, compile_fail, text should be treated differently when appearing as the language annotation of a source code block.

Source locations

The rustdoc output includes source files and span information, where convenient. We do not make use of it. The two possible future uses are:
a) somehow linking to it. This might be quite beneficial if we were to pack the source files into the output document via some embedding. (Polyglot as the PoC||GTFO journal does would be cool but is quite zip+pdf specific).
b) to refer to the item in the diagnostic output, when something fails.

Implementation progress

Maybe use the crate provided through: rust-lang/rust#81287

closes: #154

Fixes infinite recursion when we hit a specific case of an itertools iterator, a few visual glitches (too many or few line breaks), and adds the struct type distinction.

Some deficiencies mean we don't currently get all information. For example, qualifiers such as extern ABI and unsafe are dropped from methods and the information has_body is true even for provided methods.

Trait bounds and generics are still missing. Unifies the way that impls are scheduled such that we might group them soon.

They appear as ExternCrateItem 😂 That is obviously wrong and in particular this loses any information on the factual item name, on the crate origin, the path, and possibly more. We nevertheless at least try to render them here which works decently except for globs.

Also gates header levels to those that the backend actually supports. There is no subsubsubsubsection.

We had to make one slight adjustment in the order of Union and Struct, since they appear to have the same items now. This means that union can match union, with the type as an extra field that is ignored!

Uses the header adjustment strategy previously introduced to the code to ensure that scoping is correct. Most doc comments contain an h1 instead of the proper header level from the context. Note that in particular this will put some pressure on the backend to allow for slightly deeper nested headers, at least to 6. We already emulate them by a weighted font in one case.

HeroicKatora · 2021-02-13T01:28:45Z

@oberien I belive, as far as architectural changes are concerned, this is very much ready for a review. The basic set of features work although the style is anything but clean. Note the automatic header nesting on includes which is desirable for other includes as well.

Current state translating image:

```config
document_type = "article"
lang = "en"
titlepage = false
title = "Test Markdown File"
subtitle = "Showing heradoc's Features"
author = "Me"
date = "\\today"
publisher = "My Publisher"
advisor = "My Advisor"
supervisor = "My Supervisor"
citestyle = "ieee"
header_includes = ["\\usepackage{lipsum}", "\\lstalias{text}{rust}", "\\lstalias{no_run}{rust}", "\\lstalias{compile_fail}{rust}"]
geometry.margin = "2cm"
```

![](rustdoc:///home/andreas/code/projects/image)

rustdoc.pdf

It's important that this is the first step. This avoid any potential error that would come from trying to serialize a field that has an incompatible type, which would lead to the incorrect diagnosis that rust has output broken documentation.

oberien

Quite a hefty PR. I really like it, as it also shows that our concept of decoupling the frontend, generator and backend somewhat works!

oberien · 2021-02-01T09:33:00Z

src/backend/mod.rs

+    cfg: &'a Config, backend: impl Backend<'a>, arena: &'a Arena<String>, what_cargo: String,
+    out: impl Write, stderr: Arc<Mutex<StandardStream>>,
+) -> FatalResult<()> {
+    todo!()


Is this intended, or is the PR still in a draft state?

oberien · 2021-02-01T10:57:05Z

src/frontend/rustdoc/mod.rs

+                        .emit();
+                    return Err(Fatal::Output(std::io::Error::new(
+                        std::io::ErrorKind::InvalidData,
+                        "Metadata call failed",


Suggested change

"Metadata call failed",

"couldn't get crate metadata from cargo",

oberien · 2021-02-01T11:06:16Z

src/frontend/rustdoc/mod.rs

+                let krate = meta.workspace_members[0].split(' ').next().unwrap();
+
+                let format = Command::new("cargo")
+                    .args(&["+nightly", "rustdoc", "-p"])


Is nightly required? If yes, can we display a better error message if the user doesn't have nightly?

oberien · 2021-02-01T11:10:59Z

src/frontend/rustdoc/mod.rs

+                target.push({
+                    // FIXME: support actually renamed library targets?
+                    let lib_name = format!("{}.json", krate);
+                    lib_name.replace("-", "_")
+                });


Suggested change

target.push({

// FIXME: support actually renamed library targets?

let lib_name = format!("{}.json", krate);

lib_name.replace("-", "_")

});

// FIXME: support actually renamed library targets?

target.push(format!("{}.json", krate).replace("-", "_"));

oberien · 2021-02-01T11:11:40Z

src/frontend/rustdoc/mod.rs

+                    Ok(krate) => Ok(krate),
+                    Err(err) => {
+                        diag
+                            .error("Cargo metadata failed for crate")


Suggested change

.error("Cargo metadata failed for crate")

.error("parsing rustdoc output failed for crate")

oberien · 2021-04-19T15:31:48Z

src/generator/event.rs

+            FeEvent::ResolveInclude(_) | FeEvent::SyntheticInclude(..) => {
                unreachable!("ResolveInclude is handled by Generator")


I think it makes sense to add a different error message for the different already-handled includes.

oberien · 2021-04-19T15:33:42Z

src/generator/event.rs

@@ -40,6 +41,7 @@ pub enum Event<'a> {
    InlineHtml(Cow<'a, str>),
    Latex(Cow<'a, str>),
    IncludeMarkdown(Box<Events<'a>>),
+    IncludeRustdoc(Box<Rustdoc<'a>>),


Doing this as normal include is both crazy and amazing! I both love and hate it so much at the same time! :D

oberien · 2021-04-19T15:37:24Z

src/generator/iter.rs

+
+    pub fn with_rustdoc(rustdoc: Rustdoc<'a>) -> Self {
+        let frontend: Box<FeEvents<'a>> = Box::new(rustdoc.map(|event| {
+            WithRange(event, (0..0).into())


Is it possible to add proper source and span information for diagnostics to be pointing to the original rust source code? I think we should open an issue for that as using the span 0..0 for everything feels weird.

oberien · 2021-04-19T15:40:09Z

src/generator/iter.rs

-    frontend: Fuse<Frontend<'a>>,
+type FeEvents<'a> = dyn Iterator<Item=WithRange<FeEvent<'a>>> + 'a;
+
+pub struct MarkdownIter<'a> {


Is there a reason you're renaming this struct? So far I've followed having the different components in heradoc (frontend, generator, backend), and everyone that can be iterated over has its own Iter struct somewhere. Should it really be called "MarkdownIter"? It doesn't only iterate over markdown, but now also over rustdoc-output.

oberien · 2021-04-19T15:48:57Z

src/main.rs

@@ -108,7 +108,7 @@ fn main() {
        }
        None => FileConfig::default(),
    };
-    let tmpdir = TempDir::new("heradoc").expect("can't create tempdir");
+    let tmpdir = std::mem::ManuallyDrop::new(TempDir::new("heradoc").expect("can't create tempdir"));


Where are we dropping the tempdir?

oberien · 2021-04-20T13:01:43Z

src/backend/latex/preamble.rs

+    fn write_simple_header(name: &str, into: &mut impl Write,  font: &str, level: i32) -> Result<()> {
+        writeln!(into, "{}", "\\makeatletter")?;
+        write!(into, r#"\newcommand\{}"#, name)?;
+        write!(into, "{}{}{}{}", r#"{\@startsection{"#, name, "}{", level)?;


You can have { / } in the result of format strings by using double {{ / }} like so:

println!("{{{}}}", 1337) // "{1337}"

HeroicKatora added 8 commits September 5, 2020 21:07

Split cache_dir from generic temp_dir

fa96acc

Plan an unimplemented rustdoc type

191aa0c

Add bindings for described rustdoc format

20c13dd

Use the rustdoc type definitions

b53f7ce

Sketch iterative event generation from rustdoc

a2dae5e

Hookup rustdoc include frontend to generator

9047ce8

Fix bugs in rustdoc pipeline

5e1ca1b

Implement struct and fields formatting

6fec236

HeroicKatora marked this pull request as draft January 23, 2021 14:37

HeroicKatora added 8 commits January 23, 2021 20:24

Implement constant, static, enum, functions

83dec0e

Add an example crate

4824e18

Adjust implementation of some items

eaee7b4

Fixes infinite recursion when we hit a specific case of an itertools iterator, a few visual glitches (too many or few line breaks), and adds the struct type distinction.

Ensure that rustdoc behaves like a file URL

dddea3b

Check rustdoc exit status, implement unions

6228615

Add trait, to the best of current ability

9b7119f

Some deficiencies mean we don't currently get all information. For example, qualifiers such as extern ABI and unsafe are dropped from methods and the information has_body is true even for provided methods.

Fix small inconsistencies in traits and methods

dc88444

Add impl items for struct, enum, traits

0d13830

HeroicKatora force-pushed the rustdoc branch from 99dc256 to 0d13830 Compare January 24, 2021 15:15

HeroicKatora added 4 commits January 24, 2021 17:08

Add support for typedefs

a20820d

Implement remaining type kinds

8ef0f1c

Trait bounds and generics are still missing. Unifies the way that impls are scheduled such that we might group them soon.

Generic arguments and bounds

7a3a8e5

Implementation of traits with complex bounds

ffa9f8b

HeroicKatora marked this pull request as ready for review January 24, 2021 23:23

HeroicKatora added 6 commits January 26, 2021 00:32

Allow includes to modify the header level

c998649

Also gates header levels to those that the backend actually supports. There is no subsubsubsubsection.

Add a full synthetic way to inject includes

1669ccb

Update types to the newest nightly

124854b

We had to make one slight adjustment in the order of Union and Struct, since they appear to have the same items now. This means that union can match union, with the type as an extra field that is ignored!

Adjust headers of deep sections

aaabc8f

HeroicKatora requested a review from oberien February 13, 2021 01:28

HeroicKatora added 2 commits February 19, 2021 18:58

Update code for new function qualifier style

42113f9

Validate format version number

c672da9

It's important that this is the first step. This avoid any potential error that would come from trying to serialize a field that has an incompatible type, which would lead to the incorrect diagnosis that rust has output broken documentation.

oberien reviewed Apr 19, 2021

View reviewed changes

oberien reviewed Apr 20, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Rustdoc #167

Implement Rustdoc #167

HeroicKatora commented Jan 23, 2021 •

edited

Loading

HeroicKatora commented Feb 13, 2021 •

edited

Loading

oberien left a comment

oberien Feb 1, 2021

oberien Feb 1, 2021

oberien Feb 1, 2021

oberien Feb 1, 2021

oberien Feb 1, 2021

oberien Apr 19, 2021

oberien Apr 19, 2021

oberien Apr 19, 2021

oberien Apr 19, 2021

oberien Apr 19, 2021

oberien Apr 20, 2021 •

edited

Loading

	"Metadata call failed",
	"couldn't get crate metadata from cargo",

	.error("Cargo metadata failed for crate")
	.error("parsing rustdoc output failed for crate")

		FeEvent::ResolveInclude(_) \| FeEvent::SyntheticInclude(..) => {
		unreachable!("ResolveInclude is handled by Generator")

Implement Rustdoc #167

Are you sure you want to change the base?

Implement Rustdoc #167

Conversation

HeroicKatora commented Jan 23, 2021 • edited Loading

Example

Inline documentation

Source locations

Implementation progress

HeroicKatora commented Feb 13, 2021 • edited Loading

oberien left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oberien Apr 20, 2021 • edited Loading

Choose a reason for hiding this comment

HeroicKatora commented Jan 23, 2021 •

edited

Loading

HeroicKatora commented Feb 13, 2021 •

edited

Loading

oberien Apr 20, 2021 •

edited

Loading