Help With Callouts: Any Examples?

[tajmone]

@mojavelinux,

I've started using the highlight-treeprocessor.rb extension, and it works really great. I've also managed to tweak it slightly in order to support a different CSS theme for each language being highlighted:

• https://gitlab.com/saalen/highlight/blob/master/extras/langs-examples/_adoc/highlight-treeprocessor_mod.rb
• https://gitlab.com/saalen/highlight/blob/master/extras/langs-examples/_adoc/erb/block_listing.html.erb

What I really miss here is support for callouts. I haven't worked with Ruby before, and I don't know Asciidoctor's internals, so if you could provide me a couple of links to Ruby source files which deal with callouts I could study them and try to have a go at implementing the callouts part myself. But right now, I can't figure out where to start from (did sift through Asciidoctor source files, but couldn't figure out how to relate it to extensions).

Thanks.

[mojavelinux]

If you'd like to submit those changes upstream, feel free to send a PR and I'll make sure it gets merged.

What I really miss here is support for callouts.

Callouts with source highlighters are a little tricky, but not insurmountable. The secret is to extract the callouts, pass the remaining string to the highlighter, then restore the callouts. You can find that logic in these two places:

• extract: https://github.com/asciidoctor/asciidoctor/blob/4dcc19f97390382ba00ed9d294569a94772e2781/lib/asciidoctor/substitutors.rb#L1447-L1472
• restore: https://github.com/asciidoctor/asciidoctor/blob/4dcc19f97390382ba00ed9d294569a94772e2781/lib/asciidoctor/substitutors.rb#L1524-L1556

What I'd really like to do is make source highlighting an extension point so you get all this logic for free and only have to worry about highlighting the pure source. That's what asciidoctor/asciidoctor#2106 is about.

[tajmone]

Thanks a lot @mojavelinux, the links with code ranges you provided are very helpful and I've managed to start parsing out the callouts (taking my time, though, for I don't know Ruby from Adam and I'm juggling my way through).

So far, I've managed to edit the treeprocessor so that all callouts (except escaped ones) are correctly extracted and stored. Now I'll start working on the restore part.

I've used the code you linked as a reference to work on. So far, so good.
I have a couple of question for you though.

Why Comment Delimiters Are Stored?

In the extrac part of substitutors.rb, the results from RegEx callout_rx (ie: Asciidoctor::CalloutExtractRx) from both group $1 and $4 are stored for later on:

            (callout_marks[lineno] ||= []) << [$1, $4]

Where $4 is the callout number, and $1 is the comment delimiter (if any). Why is the comment delimiter being stored? In the final highlighted code it's ment to be omitted for it's just a convenience to preserve the original source from breaking up at compile time, but it won't appear in the final document. Why store it then?

Autonumbered Callouts

I also have a question about auto-numbered callouts. They are mentioned in asciidoctor.rb (762–) comments:

    # Matches a callout reference inside literal text.
    #
    # Examples
    #   <1> (optionally prefixed by //, #, -- or ;; line comment chars)
    #   <1> <2> (multiple callouts on one line)
    #   <!--1--> (for XML-based languages)
    #   <.> (auto-numbered)

But I don't recall ever coming across them in the Asciidoctor manual. Are they an undocumented feature? Are they still experimental or they can already be used in documents?

In my edited version of the Highlight extension, will I need to make sure that they are being supported correctly?

[tajmone]

Hi @mojavelinux,

I wanted to keep you up to date with my efforts toward integrating Highlight in Asciidoctor HTML backend. I've just published inside the Highlight project a folder dedicated to the topic:

• https://gitlab.com/saalen/highlight/tree/master/extras/AsciiDoc

It contains a copy of the original HighlightTreprocessor extension as well as a modded version which enables subsitutions. That's how far I've manage to get with this.

Unfortunately, my knowledge of Ruby is too rudimental. So I've decided that I'll be studying Ruby for the sake of being able to contribute to Asciidoctor.

In the meantime, I hope that these assets and documentation on the Highlight repository might help in this direction.

After what you mentioned in asciidoctor/asciidoctor#2990, about the new API 2.0 changing the way highligters will be registered in the future, I guess that it would be better to invest in the new API instead of developing the HighlightTreprocessor extension. Still, it might be worth integrating support for substitutions.

Also, worth poinitng out, when highlight-css is set to styles it doesn't correctly set the background color. I think it's due to the fact that Highlight has changed slightly the tags in the output HTML since the extension was published. If it's OK with you, I could try and fix that.

Furthermore, I suggest that the native Asciidoctor HTML5 template should add the lang attribute also on the <pre> tag for this would allow customizing the CSS themes on a per-language basis:

<pre class="highlight" lang="alan">
  <code class="language-alan" data-lang="alan">

This extra attribute won't intefere with those who don't use it, but it would allow much more control in terms of styling background color, borders, etc. Look at my example document, which uses a customized Haml template to achieve this and discusses the benefits of having this extra attribute:

• https://htmlpreview.github.io/?https://github.com/tajmone/highlight/blob/master/extras/AsciiDoc/example_mod.html#_custom_css_stylesheets

I really hope that Asciidoctor will soon support Highlight natively, for the number of syntaxes for this highlighter is huge and growing fast and, IMO, it's the easier to work with — standalone binaries, easy to write your own syntaxes, and highly customizable.

[mojavelinux]

Yes, I absolutely recommend that you immediately switch to using the new SyntaxHighlighter API. This is the key use case for why it was added.

it might be worth integrating support for substitutions.

That should already work with the new API.

worth poinitng out, when highlight-css is set to styles it doesn't correctly set the background color.

The new API gives you this control. See https://github.com/asciidoctor/asciidoctor/blob/master/lib/asciidoctor/syntax_highlighter/pygments.rb#L50 to find an example.

I suggest that the native Asciidoctor HTML5 template should add the lang attribute also on the <pre> tag for this would allow customizing the CSS themes on a per-language basis:

this is possible using the new API. See https://github.com/asciidoctor/asciidoctor/blob/master/lib/asciidoctor/syntax_highlighter/pygments.rb#L50 to find an example.

Why is the comment delimiter being stored? In the final highlighted code it's ment to be omitted for it's just a convenience to preserve the original source from breaking up at compile time, but it won't appear in the final document. Why store it then?

The comment marker in front of the callout number is only dropped when icon-based fonts are in use. Otherwise, it is restored (to preserve copy-paste functionality).

I also have a question about auto-numbered callouts. Are they an undocumented feature?

They are new in 1.5.8. They haven't been documented in the user manual yet.

I really hope that Asciidoctor will soon support Highlight natively

No additional syntax highlighters are going to be added to Asciidoctor core. Instead, the SyntaxHighlighter API was added so these integrations can be developed and published as discrete packages. You now have the ability to integrate any syntax highlighter you want to integrate.

[mojavelinux]

Btw, if you use the SyntaxHighlighter API, you don't have to worry about callouts at all. They will just work. (Though, if you look low enough, you actually have some control over them as they get passed to the highlight method. See https://github.com/asciidoctor/asciidoctor/blob/master/lib/asciidoctor/substitutors.rb#L942).