Syntax Highlighting in HTML

[tajmone]

Ciao @thoni56,

In these days I've been doing some tests with syntax highlighting in the HTML5 backend (a bit slowed down by flue and fever).

Highlight Integration

There is an example extension to integrate André Simon's Highlight in Asciidoctor:

• https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/highlight-treeprocessor.rb

I've tested it and it works fine, but the sample extension doesn't yet handle callouts and custom substitutions, which is rather limiting for a documentation project.

Right now we don't use callouts in the Manual, but they would indeed be useful. But the Manual uses custom substitutions in syntax examples, and the Beginners' Guide relies heavily on them to assign custom colors to specific lines. So, I think we should look for a solution that supports all Asciidoctor features.

I don't know Ruby and have no experience with Asciidoctor's API, so I wouldn't know where to start editing the sample extension to make it support callouts.

Probably, at this point, the best solution is to create a new Alan syntax for another syntax highlighter.

Looking Into Other Highlighters

Asciidoctor natively supports various syntax highlighter, but I was thinking of using either Rogue or Highlight.js.

Rouge

• http://rouge.jneen.net/
• https://github.com/jneen/rouge

Rouge has the advantage of producing standalone HTML5 docs, which don't require JavaScript at all. Some HTML based formats don't handle well JavaScript (e.g. CHM Help files) or they might target devices which don't support it at all (eBook readers, for ePub and Mobi/Kindle, etc.).

Besides, Rouge is to Ruby like Pygments is to Python, and we could say that it's the highlighter of choice with Asciidoctor.

Furthermore, it's the highlighter used by GitHub, so creating an Alan definition could ultimately lead to its syntax being highlighted on GitHub too (well, in theory at least, because for a file extensions to be registered on GitHub it must have a significant presence in the portal, but you can always declare it manually in Git attributes targeting GitHub linguist, and if Rouge supports the syntax it should get highlighted).

The downside of using Rouge is that I've never used it before, I don't know Ruby and it's a bit more complex than other highlighters. But this shouldn't be an obstacle, it might just require a bit more effort and time. I've already looked at examples and its documentation, and it seems doable.

Highlight.js

• https://highlightjs.org/
• https://github.com/highlightjs/highlight.js

I've already created Highlight syntaxes, and it's fairly simple to use. It's a bit strict when it comes to syntax elements naming conventions, and if you need to create a syntax defiinition that breaks those schemes it won't be included in the official distribution.

Highlight.js is lighteight and fast, and you can either link to its CDN or create a custom release with only the languages you need. Probably Alan isn't a general enough language to be included in the default distribution (or the CDN), but the website's download page allows you to create custom distributions in the browser.

---

Anyhow, I'll start experimenting with Rouge and Highlight.js, and will try to create an Alan syntax for Rouge (or both, maybe).

I'll post updates on this issue when I have some news or working examples.

[tajmone]

This syntax highlighting integration for HTML backend is turning out a bit of a nightmare...

Discarding Rouge

After having looked into it, Rouge is not going to be help us as it's not currently supported in Asciidoctor (although it's 100% Ruby and was supposed to become the native highlighter).

Considering Pygments

This leaves as with Pygments as an alternative. But there are some catches here:

• Asciidoctor uses Pygments via the pygments.rb gem, which only works with Python 2 (although Pygments works with both Py 2 and Py 3!).

• Using a custom Pygment lexer (ie, Alan) requires some customized initialization before launching Asciidoctor:

  https://asciidoctor.org/docs/user-manual/#using-a-custom-pygments-installation

  I'm not sure how this is going to work on all users machines, as it requires a path to the local installation of Pygments.

  We could hope that the Alan syntax would be included soon in the official Pygments distribution and that, in turn, it gets included in the pygments.rb too (which includes Pygments in it). But that's a lot of "ifs" and "let's hopes".

Vaiable Options

I'll look into Pygments, as this would allow total freedom in terms of callouts and custom stylings over source code blocks.

Quite frankly, Highlight.js seems the easier solution right now because it's easy to integrate into the project and doesn't force anyone to install Python, Pygments, etc. Also, Highlight.js releases are fully standalone, and they should support callouts and all the rests (cross fingers). Maybe we should settle on this as a starting point.

I wish the Highlight extension was completed so it could support callouts, but not knowing Ruby there isn't much I can do about it. Asciidoctor team were already very valuable in adding support for it (it started off from a request of mine), and they even provided a base extension to start with; asking more would be too much.

Any thoughts on this mess?

[tajmone]

I've managed to create the Alan syntax definition for Highlight.js!

I've done some testing against edgecases, but it seems to work fine. Tomorrow I'll integrate it into a test branch so I can test how it copes with callouts, etc. If it works, I'll create a custom theme for it and merge it into master branch.

Later on we could bundle with it a BNF syntax too, and maybe AsciiDoc too (for the repo's documentation, which contains AsciiDoc examples).

Before submitting it to the highlight.js repo I want to test it further.

But I'd say we're at a good point here.

[tajmone]

Ciao @thoni56,

It's been a hard day, but It's mission accomplished.

Highlight.js Integration

I've succesfully created an Alan definition for highlight.js, and integrated it into the project:

• https://github.com/alan-if/alan-docs/tree/master/_assets/hjs

I've also created a PR for it to be included in the official distribution, so hopefully its utility will go beyond this project too.

I've update the typography test documents to check if callouts are handled well in both PDF and HTML5 syntax-highlighter blocks, and they are.

Now the HTML version of the Manual is syntax highlighted (only Alan code right now).

I took advantage of Highlight.js' stylesheet to inject into it also styles for BNF blocks and Game Transcripts. So, possibly, any future CSS tweaks will go there too.

I'm glad this got solved, as now both the HTML and PDF templates are starting to get into shape, and the proofreading stage is almost at the end. Hopefully, by November the Alan Manual will be production ready in both formats.