[WIP] Curated Doc Refactoring

[thehydroimpulse]

This isn't ready for prime time yet, feedback is super important in areas like this, so that's what I'm looking for.

This is in relation to the Splitting the Docs thread I posted to the mailing list.

In summary, this will replace the current curated documentation with a more appropriate system. This introduces nanoc as the static site generator. There's no particularly huge reasons why nanoc was chosen, it works pretty well, and Github uses it for it's developer documentation.

There's two parts to this system:

• Nanoc Site
• Validator

Ensuring that the code blocks are legitimate Rust code is super important. The validator takes a directory as an argument ./validator ./dir and walks the files and validates them. It's currently intended for Markdown files, but it can be extended for other uses. That's all it does. It's still somewhat rough and not perfect (could use some parallelism to speed it up).

You can also view a live version.

The ultimate goal is to provide a better user experience for new comers. Currently, we have a few problems:

• Tutorial is a giant page where everything was shoved into
• No direction at all! A user can easily get overwhelmed at massive pages like that.
• Secondary guides are almost invisible. I certainly didn't notice the links at the bottom for a while.
• Adding more information requires more work than is needed.

Thus, turning to a more extensible platform would allow us to control the user through each step of the process. I'm turning to the Julia docs as an example.

Dependencies

Unfortunately (perhaps delegation is a good thing), new dependencies emerge.

• Ruby is required (I specified 2.1.1, but if 1.9 is more appropriate, then I can change it)
• Bundler (to ease the pain of managing the gems)
• Pandoc (I believe this is already currently being used by Rust?)

---

Note: This new system is not tied to Rust in any way. Meaning, doing make check or running the standard commands does not initiate the new docs. They're isolated for the time being (and because I haven't made my strides into the wonderlands of Rust's makefiles).

---

Todo

• Organize the content into guides/categories (introduction, syntax, etc...).
• Polish the validator (perhaps allow it to check multiple files at a time).
• Tie the new system into the rest of Rust.
• Polish the design

Overall, I'm pretty content with the simplistic design, similar to how the current docs are. It's not perfect yet, but the typography feels solid and doesn't require external fonts.

[huonw]

Unfortunately (perhaps delegation is a good thing), new dependencies emerge.

• Ruby is required (I specified 2.1.1, but if 1.9 is more appropriate, then I can change it)
• Bundler (to ease the pain of managing the gems)

This seems quite unfortunate, depending on another language infrastructure. (AIUI, Ruby is somewhat less stable than e.g. Python, too?)

It seems a Python generator would be more appropriate if we are going to use external tools. Another option is extending rustdoc.

• Pandoc (I believe this is already currently being used by Rust?)

It is only used for the optional step of compiling documentation to PDF; the HTML documentation (including tutorials etc.) is build just using rustdoc. i.e. pandoc is not currently a hard dependency for building Rust-with-docs.

[thehydroimpulse]

This seems quite unfortunate, depending on another language infrastructure. (AIUI, Ruby is somewhat less stable than e.g. Python, too?)

I agree about it being unfortunate. I'm not a python person, so I naturally chose to go with a proven Ruby project.

It is only used for the optional step of compiling documentation to PDF; the HTML documentation (including tutorials etc.) is build just using rustdoc. i.e. pandoc is not currently a hard dependency for building Rust-with-docs.

Gotcha. I tried working with a Ruby gem (C + Ruby) for Markdown, but it didn't work correctly with Rust (it didn't detect Rust as a language). Curated docs would only really be built by contributors or those wanting to contribute where as rustdoc would be used by library authors and the like.

Adding more dependencies is definitely a serious concern.

[adrientetar]

Rustdoc can process standalone files, e.g. tutorial and so forth are processed by it currently.

Nothing intrinsically wrong with Ruby.
What about Jekyll? Native integration with GitHub means it could be pushed to the gh-pages branch.

[thehydroimpulse]

@adrientetar I thought about Jekyll, but before it's 2.0 release, it was purely a blogging platform. I'm waiting until Github Pages supports 2.0 (Last time I checked, they were working on it).

[adrientetar]

rust-lang.org uses it, it’s not a blog.

[alexcrichton]

This looks pretty awesome, I'm excited to see where this goes!

I am also concerned by the new dependencies brought in as part of this change. I don't want to require pandoc to build rust documentation, and what does the site generator buy us that we don't already have? We can add options to rustdoc to inject html at the start/end of files (which appears to be what the generator is doing), and we could style the existing markdown similarly.

[adrientetar]

We can add options to rustdoc to inject html at the start/end of files (which appears to be what the generator is doing),

It does it already, @huonw implemented a pandoc (almost) drop-in replacement as part of his standalone Markdown processing PR.

[huonw]

Nothing intrinsically wrong with Ruby.

Yes, there's nothing particularly wrong with Ruby itself, but there are problems with adding new dependencies.

[thehydroimpulse]

rust-lang.org uses it, it’s not a blog.

@adrientetar Simple static sites works fine with Jekyll < 2.0, but it wasn't meant for extensive documentation or other focuses.

I'm a little hesitant about adding more functionality to rustdoc. It seems too much like rebuilding the wheel with no real gains (other than squashing some dependencies).

One solution would be to move to a Python system (like @huonw suggested) and use Sphinx for the curated docs, which I believe @chris-morgan has been advocating for it. There would be a few downsides:

• Not markdown. Meaning it would add some inconsistencies in all the docs (api docs use markdown, but curated docs don't.)
• Another stack to learn
• Sphinx is a lot more complex (which provides some added functionality; like search, etc...)

Sphinx, however, is used by quite a lot of notable projects, including Julia's docs. It also offers the ability to generate PDF, HTML and ePub bundles without much pain.

It also does syntax highlighting to Pygments, which supports Rust, so that's all good. The only thing that needs to change is the validation (which currently works on Markdown code blocks), but Sphinx apparently has code validation built-in. I'm not sure if that allows it to pipe each code block to a specific process or what not, but I'll look into it.

Thoughts @alexcrichton @huonw or anyone else?

[alexcrichton]

We will be rebuilding the wheel in some cases, but the wheel that we're rebuilding is normally quite simple. For example, markdown => html was a small wheel rebuilt via the glue we have to hoedown, but it was well worth our time because now we're able to parse, extract, colorize, and test code examples in rustdoc itself.

I view this as pulling in more than just a dependency, but also adding a whole new system that has to be maintained across all our bots, all contributors checkouts, etc. I've found that everything is much much simpler if we're willing to write a bit more code to implement simple things like markdown => html generation (and others).

I would prefer to explore how to leverage what rustdoc already gives (injection at the header and footer of every page) before exploring any solution written in either python or ruby. We shouldn't have a need for a full-blown static site generator, we're just serving up some simple documentation. We're also not moving to Sphinx at this time, we're sticking with Markdown.

[thehydroimpulse]

@alexcrichton Gotcha. I agree then. I'll start integrating this stuff into rustdoc.

[adrientetar]

I'm a little hesitant about adding more functionality to rustdoc.

From this PR it seems like you are adding index.html to every file, testing code blocks and syntax highlighting code.
Maybe I'm missing something but it seems to me that rustdoc does all that.

[thehydroimpulse]

Closing this PR for now until I'm finished.