switch off of jekyll

[benesch]

It's become increasingly clear to me that we're at the limits of what Jekyll can handle for a documentation website. These are the big problems in my mind:

• Cross-references are based on page names and section headers. Changing a page name or a section header will break every internal link that points at that page or section. We luckily use HTMLProofer to detect this breakage, but every HTMLProofer run takes about ~10m, so hunting down broken links requires a long feedback cycle.

• No support for book-like hierarchy, with chapters, sections, and subsections. I think @knz in particular would give anything for this layout (Proposal for a new sidebar organisation for sql docs. #1021).

• No PDF output (Generate PDF docs #1484).

• No plain text output (Investigate options to output plain text #1717)

• Inconvenient interleaving of content and style. It's very confusing but also dangerous for maintainability to have HTML and CSS inline with the documentation text. This also contributes to the two problems immediately above.

• Markdown is poorly-specified. Switching Markdown processors caused all sorts of unnecessary breakage because Markdown doesn't have a formal grammar to e.g. clarify when to indent a paragraph beneath a list or how many |s you need to start a table.

• Slow compiles. The current site takes 8s to generate, and every new version will add about 4s to the build time. jekyll serve --incremental knocks that down to ~.2s, but only works in simple cases where you change e.g. foo.md and want to see foo.html. If you change a base layout, included file, or sidebar data, Jekyll will have to do a full rebuild, taking another 8s, if it even realizes that.

• Unable to provide code examples with syntax highlighting inside a table. No multiple paragraphs.

• Unable to display math equations (will be necessary once we start documenting resource costs)

Here's my proposal: use Sphinx. It neatly solves all my grievances. In order:

• Supports proper cross-references that are independent of the page and section titles.

• Understands hierarchy.

• Can generate PDFs.

• Supports inline math equations, inline block diagrams

• Supports arbitrary markup inside table cells

• Supports the definition of arbitrary new markup syntax that can be mapped to HTML/CSS using an external transformation.

• Also supports a well-specified markup language called reStructured Text. (Sphinx actually supports multiple markup languages including markdown, AFAIK)

• Compiles reasonably fast. The Python docs tree, which is much, much larger than ours (though not for long, I hope!), takes 2m to generate—but only the first time. After that, Sphinx is able to do an incremental build in ~7s, which includes properly regenerating dependent pages, like indices, and not just the one page that changed.

Changing our documentation generator would obviously have a massive impact on @jseldess, @sploiselle, and @Amruta-Ranade, and I don't mean to step on any toes! Just wanted to get a discussion started somewhere more permanent than Slack. I know Sphinx has been mentioned several times in passing, but I wanted to formalize the arguments for it.

The biggest downsides I see of a Sphinx transition:

• Actually doing the transition. Oof. This smells like a post-1.2 project at best.

• Disrupting everyone's workflow. rST, in particular, is a bit of a special snowflake. It's great once you know it, but it doesn't have the nearly non-existent learning curve that Markdown has.

FWIW, now that I've written all this up, I'm 👎 on switching to Hugo. Of the grievances above, I think it only fixes the slow compiles.

/cc @bdarnell @mjibson @knz

[dt]

While we're considering big changes, I'd also submit for consideration:
serving the site using some very thin application server, rather than purely static files.
This could be as simple as a [strawman] couple lines of node, map of paths to html files, just opens and serves file [/strawman] but it seems like it might be useful to have more control over our path routing (i.e. ability to serve 301s, 302, potentially run search server-side, etc).

[jseldess]

Thanks for starting this, @benesch! @knz and @bdarnell have both brought up a number of these points before, but it's really helpful to see the benefits of Sphinx enumerated.

Longer-term, I agree that all of the above could cause be concerns. Short-term, I think the most immediate concern is Jekyll's lack of PDF output.

Unfortunately, I do think this is at least post-1.1, given what's on our current plate. But let's please continue the discussion here.

[sploiselle]

+1 on an rST-backed site generator.

[knz]

I have modified the PR description at the top to add some considerations about the current problems.

I do not fully align on the description of the work to be done however. There are two separate sets of problems: those that will be solved by using a different markup language (even possibly using the same templating engine) and those that will be solved using a different templating engine (e.g. sphinx, possibly continuing to use markdown).

From the top of my head:

• cross-references, hierarchy, proper TOCs, these are problems solved by Sphinx, irrespective of the markup language. I guess we could bake a markdown extension to address the linking problem ourselves even without Sphinx (although I am not keen on re-implementing this ourselves if an off-the-shelf solution does this right already)

• support for PDF output, plain text, math formulas, arbitrary markup in table cells, avoiding interleaving markup and HTML/CSS, these are problems caused specifcially by markdown and using a different markup format even without sphinx would move us forward on this.

[knz]

Also, let's keep in mind very high that this is not a discussion about moving away from text-based markup formats and towards very noisy languages like TeX/SGML etc.

There is a large class of usability/productivity problems solved by text-based ("almost-WYSIWYG") markup, and we must not go on a path that would cause these problems to happen to us.

I have written before about different markup languages and what problems they solve, to conclude that rST and Asciidoc in particular are superior to markdown: http://www.structured-commons.org/scep0106.html#choice-of-markup-languages

AFAIK Sphinx also supports asciidoc :)

[bdarnell]

@knz the sphinx/markdown integration extends markdown with support for features including math rendering. On the other hand, sphinx's shorthand cross-reference syntax (plain backquotes) is one of the few places where I like RST better than markdown.

FWIW, I can't find any sphinx/asciidoc integration in a few minute's googling. I'm open to asciidoc-based solutions, but I'm not familiar with it at all.

support for PDF output, [...], these are problems caused specifcially by markdown and using a different markup format even without sphinx would move us forward on this.

I don't agree with this part. We could, for example, render markdown to pdf with pandoc. The problem is that to get good pdf output, we need something with more higher-level structure, not just converting our markdown to asciidoc. I'd put PDF output in the "solved by sphinx, irrespective of the markup language" bucket.

I'm generally a fan of sphinx for our docs (and, I think, of moving from markdown to RST), although I must say that speed is probably not a reason to switch. Incremental builds with sphinx will generally be closer in speed to jekyll than hugo. And there's not a built-in equivalent to jekyll serve --incremental, so you have to kick off builds manually (there may be a third-party solution for this; it's been a while since i've looked for one).

[knz]

@bdarnell 1) sphinx I recall supports "incremental builds" in the sense that you can serve the docs and have it re-build files on-demand when the underlying source changes 2) regarding conversion to PDF, my biggest gripe with the current approach is the interleaving of html/css/javascript with the doc source, which really complicates ("complicates" as in "sometimes makes it impossible") conversion to anything else than web pages.

[knz]

I did some homework to match rST and markdown side-by-side:

• rST vs markdown shootout in HTML.
• same doc, rST sources - note, you couldn't make such a document in markdown, because markdown tables ... well...
• if you want to regenerate the doc yourself, use this archive.