cargo doc could support building/testing standalone markdown files

[huonw]

E.g. project specific guides/tutorials, like rust-lang/rust itself.

See also

• Run doc tests on README #383
• doc-comment

[chriskrycho]

Right now, it appears that rust-lang/rust uses pandoc to build its docs. It seems like it would be nice to go ahead and get this moving forward so that dependency could be dropped. Two questions, therefore:

1. What is the status on this? Has anything been done?
2. What would need to be done to make it work?

[steveklabnik]

I'm not aware of anything that's being done.

I don't actually think pandoc is used anymore, all that is stale.

[chriskrycho]

Ah, interesting. I'll go read the makefile; if nothing else I can submit a PR to update that.

[steveklabnik]

I was editing but since you replied, I'll make it a new comment:

I don't actually think pandoc is used anymore, all that is stale. It was being used to generate PDF and EPUB, but we don't do that anymore.

You'd basically need a way to tell cargo to invoke rustdoc on MD files, wherever those are. I would suggest maybe a top-level doc directory that any doc/**/*.md gets rustdoc called on would make sense.

[chriskrycho]

One quick question about that: I assume you're meaning top-level doc relative to src, in line with the conventions in e.g. this repository? So a basic crate setup for Cargo would have something like:

/
  Cargo.toml
  README.md
  src/
    lib.rs
    and_modules/
    doc/
      some_stuff.md
      more_stuff.md

Or am I misreading you?

Also, would it make sense to include the README.md file if one is supplied—as the index, perhaps?

Fair warning: it'll probably take me at least a month to implement, if not more, but I'd like to give it a go. I'll probably be inhabiting #rust an awful lot on the weekends I work on it. 😛

[alexcrichton]

I've thought about this a bit in the past, and the general idea is something like:

1. On cargo doc, Cargo will automatically crawl /doc (in the project root) for *.md files
2. Each markdown file will be passed through rustdoc and rendered, maintaining the same basic structure as the doc folder itself
3. On cargo test Cargo will automatically run rustdoc --test over all markdown files

Feel free to reach out to me with any questions!

[chriskrycho]

@alexcrichton, very good.

The reason I was thinking to use /src/doc rather than /doc for the root was because both this repository and the main Rust repository both already have their docs there. As such, my guess was that a fair bit of the community may have their packages likewise structured (because my own first instinct for structuring a project is to check how large, official projects in a language do it). Obviously those can move, though,* and I'm happy to defer to you guys in any case. I'll plan for it to work basically as you outlined.

* I was also hesitant initially because I was remembering repository size problems problems with large moves in Mercurial, but that's not a pain point for Git, so.

[chriskrycho]

@alexcrichton and @steveklabnik, how should this interact with (including just ignoring it) rustbook?

[alexcrichton]

I don't have super strong opinions about /src/doc vs just /doc, but I think we'd prefer to root everything in /doc as it seems to fit the conventions of other projects in other languages. The reason the compiler does this is that the top level /doc is actually where all the generated documentation goes. I... don't actually know why cargo does it!

I'm not currently aware of any conventions externally in the community about this, I wouldn't expect there to be too much buy in to either scheme just yet though.

I'm also fine just not worrying about rustbook for now, conventions can always be adapted!

[steveklabnik]

Yeah, I feel the same as @alexcrichton, basically. Same with 'dont worry about Rustbook'.

[chriskrycho]

Excellent—just wanted to check that box before proceeding. I'll go ahead with the root docs directory! Hopefully starting in on it a bit this week(end).

[SimonSapin]

Would these generated HTML files (optionally?) include the crate index side bar and the search bar?

[chriskrycho]

While I haven't had a chance to dig in yet courtesy of start-of-the-semester busyness, my tentative plan is to make those available, yep. Presumably with an optional argument to the command, similar to how you can specify a CSS file or other HTML to include now.

[SimonSapin]

👍

[chriskrycho]

Just a quick update: I've spent several hours last week and expect to spend several more this weekend familiarizing myself with the way cargo doc currently works so that when I start implementing, it'll be in line with how things presently work.

[alexcrichton]

@chriskrycho awesome! Feel free to ask any questions here or on IRC if you run into any speed bumps!

[ebkalderon]

Bump. Any progress on this @chriskrycho?

[chriskrycho]

Alas: no. School and work this semester have taken up more time than I’ve expected. It is still very much on my radar, but of course if someone else gets to it first… I won’t argue.

[steveklabnik]

Just checking in: nobody has started work on this in the meantime, yeah? Maybe I should.

[chriskrycho]

No, I had an abortive start back in October or so, but nada since then. If you give it a go, I’d be happy to chip in.

[hoodie]

Just as a consideration, how would we name these markdown files? Tread them the same way as .rs files so that I pub mod documentation for a documentation.rs? There are tones of projects on github that have Readme.md files in subdirectories. What about (when this is reopened) rustdoc automatically includes Readme.md files in module folders as module level documentation?

[hoodie]

@chriskrycho BTW: I would also love to see this take off, so if you want to collaboratively start working on this again? I'm in.

[steveklabnik]

@hoodie in general, the idea would be that you'd write whatever.md and that will generate a whatever.html.

I'm skeptical of making readmes into module-level documentation, but possibly as an index.html.

[hoodie]

I though, only explicitly if the Readme.md is right next to a mod.rs, per se.

[lilith]

I'm a bit confused about what the docs mean regarding standalone markdown files if they're not referring to this feature. Am I missing something obvious?

[Michael-F-Bryan]

That's what I thought as well. I tried putting a markdown file in my src directory (with the % title line) and then running cargo doc, but it looks like cargo completely ignores the file.

Minimal example:

$ cd /tmp
$ cargo new dummy && cd dummy
$ cat > src/extra_docs.md << EOF
 % A Title           

 # Heading 1

 Lorem ipsum...
 EOF
$ cargo doc --open
$ find ./target/doc -name "*extra_docs*"

In this case I'd expect the extra_docs page to be compiled to html and then linked in with the rest of my documentation. You can compile markdown manually with rustdoc, which is probably what the documentation is referring to, but that won't link it in with the rest of your docs or apply any styling and cargo doesn't have any (easy) ways of including extra pages purely for documentation.

This was done using nightly rustc and cargo.

[drusellers]

https://www.reddit.com/r/rust/comments/5crpfy/announcing_cargo_externaldoc/

[steveklabnik]

@nathanaeljones @Michael-F-Bryan those are the docs for rustdoc. Rustdoc can do this. cargo doc does not call rustdoc on markdown files. That's the core issue.

[JelteF]

cargo-external-doc is really nice. But it would be great to have this directly integrated into cargo. That way I can use docs.rs instead of hosting my docs on github.

[Zteve]

@alexcrichton I'm trying to build doc for a project atm; and not having the md files merged into the cargo generated documentation is a real pain. I don't want to use an add-on like cargo-external-doc if an integrated solution is forthcoming, but on the other hand the list of things you say you need before you can do that seems very daunting.

Is it really necessary to do all those things before you can simply reference the md files from the generated doc?

[alexcrichton]

@Zteve I'd personally prefer a fully fleshed out implementation, yes. Half-baked implementations just get bugs opened against them and I usually end up personally having to fix them, which I'd ideally prefer to head off at the beginning.

[Zteve]

@alexcrichton Fair enough; sentiments I can relate to. Go to it, then :-D

[jamen]

I came across this and #1016 after thinking of workspace documentation. It'd be nice to have doc/index.md when documenting all of the crates, or providing information the subject the project is about.

[benesch]

If you want a hacky shell/jq script to do that in the meantime, here's what we're using at @MaterializeInc:

https://github.com/MaterializeInc/materialize/blob/master/bin/doc
https://github.com/MaterializeInc/materialize/blob/master/misc/doc/crates.jq

[Carter12s]

Hey I'm interested in trying to tackle this! Caveat, first time attempting to contribute to cargo and this old of an issue is pretty intimidating. Reading the contributor's guide right now (what a great doc), and this issue is missing a lot of labels? Could likely use an area, feature accepted (if it is?), mentor maybe?

I'm planning to start reading up on this today, and will hopefully be able to post a design (or at least a list of questions) soon.

I have steve's work from 5 years ago as a bit of a start: #2256