add ANSI writer

[silby]

This requires jgm/doclayout#28 in addition to the stuff I already got merged; cabal.project does not reflect this. The required doclayout commit is added to cabal.project.

I think that the surface area supported by what I have here is a minimum scope to be worth releasing, but I don't really think this is quite good enough as is to merge. I'm filing a draft PR to get some feedback on the choices and compromises made so far and what the requirements for a shipping ANSI writer should be. I'd really love some ideas on how to format headings! Some of what's here reflects arbitrary design choices that make me happy, like putting a four-column margin on the whole output.

It's a bit slow; rendering MANUAL.txt to ANSI takes 2.77 seconds on my machine, compared to 0.66 seconds for rendering it to HTML. I have an inkling this may be due to the quantity and size of definition lists in MANUAL.txt, but who knows. The slowness is fixed.

There's no new tests or docs so far.

---

The ANSI writer (-t ansi) outputs a document formatted with ANSI control sequences for reading on the console.

Most Pandoc elements are supported and printed in a reasonable way, if not always ideally. This version does no detection of terminal capabilities nor does it fall back to different output styles for less-capable terminals.

Some gory details:

• Title blocks are formatted with modest extravagance in –standalone mode.
• Strong, Emph, Underline, and Strikeout spans are all formatted accordingly using SGR codes (which will be silently ignored by terminals that don’t support them).
• Headings have somewhat arbitrary styles applied to them that probably need immediate improvement.
• Blockquotes and all flavors of list look pretty good.
• Code spans are colored magenta-on-white, which on the author’s terminal looks kind of like the pinkish treatment of code spans used by many stylesheets. This probably isn’t a good final decision.
• Code blocks are formatted by Skylighting’s formatANSI using standard writer options and included directly in the output. This has some issues; see code comments.
• Links are printed with OSC 8 to create hyperlinks and colored cyan. The author’s terminal automatically adds a dotted-underline to OSC 8 hyperlinks, but only colors them differently on command-mouseover. Setting an underlined style on links may be more broadly accessible. OSC 8 support is not checked for, so on terminals not supporting it or with support disabled, the link text will be colored but not do anything and the links will not be printed.
• Images are displayed as their alt text. Support for the Kitty and iTerm 2 inline image protocols is planned. Supporting other terminals by using Chafa (https://hpjansson.org/chafa/) to print sixels etc would be cool too but the author would have to do some FFI stuff and it would add a dependency to Pandoc.
• Tables are replaced with a useless placeholder. Table output using box-drawing characters is desired.
• Subscripts and Superscripts are just parenthesized when accurate Unicode representations aren’t available. Because these span types could have all kinds of semantics, there’s not an obvious thing to do with them.
• Simple math is translated to Pandoc inlines using existing functionality. An ambitious person could look into emulating the console-mode math output of a computer algebra system, or rendering each display math element as an image with TeX or Typst and including it, or some other thing.

[jgm]

Nice! Upload a screenshot?

The slowness does concern me. I think the main application of this would be reading files on your terminal, and a 2.7 sec delay seems too much for that. Can you pin down where the slowdown comes from, by experimentation or profiling?

[silby]

Ah, there's definitely something with super-linear time complexity going on with renderANSI, I'll try to fix that!

[silby]

See jgm/doclayout#29 for the performance issue.

[silby]

Here's a couple screenshots of how things look presently:

MANUAL.txt:

ol, ul, code, strikeout, math, blockquotes

[silby]

Staring at my first screenshot there I realized DocLayout's line breaking is broken by the styled text, filed jgm/doclayout#30 to track since I don't think I'm going to plow all the way to a solution right away.

[silby]

Rebased. Please note d664dbe; something about jgm/doclayout#31 has caused a whitespace regression here. Actually I just realized what it probably is; hang 2 (bullet <> space) ultimately is putting an empty doc inside the Prefixed, which is getting eliminated now by doclayout's flatten.

The djot reader parses `-\n' parses as an empty list item just fine but nevertheless I'm not sure if tweaking this test is fine or if we should go put the missing space back where it belongs.

[jgm]

I'm not worried about that space; I think removing it is actually the intended behavior so I'll consider this a bug fix.

[jgm]

Some comments on the design choices:

Title blocks are formatted with modest extravagance in –standalone mode.

I think the horizontal line with the emoji is too opinionated (even though it looks cool).

Code blocks are formatted by Skylighting’s formatANSI using standard writer options and included directly in the output. This has some issues; see code comments.

Is the issue just that skylighting produces a fully rendered string? I don't see why that's a problem, necessarily.

Setting an underlined style on links may be more broadly accessible.

Probably a good idea.

OSC 8 support is not checked for, so on terminals not supporting it or with support disabled, the link text will be colored but not do anything and the links will not be printed.

Is there any way to check for it?

Images are displayed as their alt text. Support for the Kitty and iTerm 2 inline image protocols is planned. Supporting other terminals by using Chafa (https://hpjansson.org/chafa/) to print sixels etc would be cool too but the author would have to do some FFI stuff and it would add a dependency to Pandoc.

I'd like to avoid C dependencies.

Tables are replaced with a useless placeholder. Table output using box-drawing characters is desired.

For placeholder purposes, one option would be to include a table generated by the plain writer. This would give you the content, though without ANSI formatting.

I wonder whether it could work to use the gridTable function from T.P.Writers.Shared? That should in principle get you grid tables with ANSI formatting inside the cells.

[jgm]

Code spans are colored magenta-on-white, which on the author’s terminal looks kind of like the pinkish treatment of code spans used by many stylesheets. This probably isn’t a good final decision.

Why not use skylighting for these too, for consistency?

[silby]

Some comments on the design choices:

Title blocks are formatted with modest extravagance in –standalone mode.

I think the horizontal line with the emoji is too opinionated (even though it looks cool).

Admitted. I'll calm this down.

Code blocks are formatted by Skylighting’s formatANSI using standard writer options and included directly in the output. This has some issues; see code comments.

Is the issue just that skylighting produces a fully rendered string? I don't see why that's a problem, necessarily.

Honestly I only partly remember what I was talking about. The main thing that's subpar is if you select a highlight-style that uses a nondefault background color, you get what I would consider subpar results:

If we could get a Doc back from the highlighting we'd be able to keep the background color bounded to the borders of the code block, cf:

Overall I think the state of affairs with code blocks is fine for starters.

Setting an underlined style on links may be more broadly accessible.

Probably a good idea.

• Set underlined style on links.

OSC 8 support is not checked for, so on terminals not supporting it or with support disabled, the link text will be colored but not do anything and the links will not be printed.

Is there any way to check for it?

Not really, I guess. Someone has a list of the terminal emulators that are known to support it, but there's no terminfo capability for it or a report control sequence that can tell it to us. That would leave us with the environment-variable equivalent of user-agent sniffing, which will be annoying to maintain and perpetually incomplete.

I think the ANSI writer does need to be able to print the links out visibly, as there are certainly widely used terminal emulators that don't and may never support OSC 8 (Terminal.app, xterm, urxvt). Some users with supporting terminals might not want it anyway. I think it has to be a writer option provided by a CLI flag.

Images are displayed as their alt text. Support for the Kitty and iTerm 2 inline image protocols is planned. Supporting other terminals by using Chafa (https://hpjansson.org/chafa/) to print sixels etc would be cool too but the author would have to do some FFI stuff and it would add a dependency to Pandoc.

I'd like to avoid C dependencies.

Valid. I don't think the lofi outputs supported by Chafa are a critical feature for Pandoc. The Kitty and iTerm 2 protocols will probably cover most of the users who would want to see images in the terminal anyway.

Tables are replaced with a useless placeholder. Table output using box-drawing characters is desired.

For placeholder purposes, one option would be to include a table generated by the plain writer. This would give you the content, though without ANSI formatting.

I wonder whether it could work to use the gridTable function from T.P.Writers.Shared? That should in principle get you grid tables with ANSI formatting inside the cells.

• Try using gridTable for a first iteration of table output

[jgm]

The main thing that's subpar is if you select a highlight-style that uses a nondefault background color, you get what I would consider subpar results

Ah yes, that is bad. I'd be open to having skylighting-format-ansi export a function that yields a Doc Text.

[jgm]

Another thought about the 4-space indentation on body text. This echoes the formatting of man pages, and it looks good and familiar. But one drawback is that it makes it hard to copy/paste content from a document you are viewing this way -- since you'll get these initial indents.

[silby]

I think the copy-paste drawback you mention is of limited relevance to the ANSI writer use-cases. The analogy to viewing man pages is apt; I think the ANSI writer should optimize for reading over other considerations. Even if there weren't an indented margin, copy-pasting would still be a bit annoying due to hard breaks in the formatted output. At least in Pandoc if someone wants a chunk of more pasteable text you can rerun with -t plain --wrap=none; I don't think there's a way to get unwrapped/unindented text output from mandoc for this purpose.

[silby]

Code spans are colored magenta-on-white, which on the author’s terminal looks kind of like the pinkish treatment of code spans used by many stylesheets. This probably isn’t a good final decision.

Why not use skylighting for these too, for consistency?

I see relevant examples in other writers that support syntax highlighting. I sort of assume code spans marked up with their language are rare but it makes sense to provide as much support for it here as elsewhere. We'll still need some kind of fallback for generic code spans. The basic issue there of course is that all the text in the terminal is already fixed-width, so we can't use that for contrast.

Magenta-on-white sort of resembles what Slack does with code/fixed-width spans:

The GitHub CLI uses Glamour to render gfm and its code spans default to bright red on dark red, plus an extra space's worth of padding on each end of the span for additional contrast.

I want to land on something here that doesn't require me to get to jgm/doclayout#32 just yet, because I uh, don't want to work on it right now.

Any particular preferences or suggestions?

[jgm]

Copy-paste: I'm not sure. I certainly have wanted to copy code from man pages and READMEs before, and wouldn't have wanted to context shift, exit man, fire up a new program, and find the relevant section again. Even if we're optimizing for reading, copying text IS something one often wants to do when reading a document.

Certainly if I wanted to copy some text from a Word or HTML document or EPUB, I'd rather fire up pandoc -t ansii in the terminal and copy it from that instead of starting Word or a browser or an ebook reader.

Note also that eliminating the 4-space indent would make part of the problem with highlighted code blocks with backgrounds go away. (There would still be the odd color shift at the end of the first line, which I don't really understand....maybe this is something that needs to be fixed in skylighting-format-ansi?)

[silby]

Note also that eliminating the 4-space indent would make part of the problem with highlighted code blocks with backgrounds go away. (There would still be the odd color shift at the end of the first line, which I don't really understand....maybe this is something that needs to be fixed in skylighting-format-ansi?)

I'm not positive I totally understand it either. Removing nest 4 from the body yields a code block that looks like this, in less -R,

but like the prior screenshot, when echoing directly to the terminal.

I think the different behavior of less is due to this documented behavior of less -R:

Color escape sequences are only supported when the color is
changed within one line, not across lines.  In other words, the beginning of each
line is assumed to be normal (non-colored), regardless of any escape sequences in
previous lines.

(wow there I go copy pasting from the man page)

So I interpret this to mean that less imputes a default-color control sequence at each \n in the input, in contrast to what my terminal is doing.

In any case, the best thing to do is use doclayout to put code blocks into an actual rectangle but as noted I think the current state is acceptable. The default pygments highlight style appears to stick to the default background color.

---

I don't want to bikeshed the matter of the margin; I can drop it from this PR with the idea that it makes sense to err on the side of less-opinionated output at least to begin with.

That does make the question of how to style headings a bit more of a headscratcher. What's here so far isn't thoroughly worked out but shifting the headings into the margin like man output was at least an available option. Since we can't make headings larger or use a different font family, it's not obvious how to create hierarchy.

Lynx's defaults are not at all exuberant:

The style used by the Charm.sh folks in https://github.com/charmbracelet/gum uses box-drawing characters for this in a way that seems fairly sensible, on top of a color contrast. I could take or leave the color part. (They have a 2-column left margin too, heh.)

We could try modestly more spacious decorations with box drawing characters as well, cf.

┌╴
╵ Heading 1 ╷
           ╶┘

If we we could even take advantage of markdown's familiarity and put the appropriate number of # before each heading.

Converting H1s to all-caps would match how man pages are written, but Pandoc doesn't really do any such thing by default elsewhere and makes no difference for many scripts.

Do you have any inclinations on this? I think getting headings right is the big remaining design decision I want to make.

[jgm]

One question is whether it's necessary to distinguish different heading levels visually. Personally, I think it's okay not to. Using indentation would give you a way to distinguish a couple of levels of heading, but not six (unless you indent the body text really far, or do what lynx seems to do, which is just ugly -- with the heading indented more than the contained body text).

Putting the heading in boldface and a different color from the body text definitely sets it apart as a heading, even without indentation.

I don't like the box-drawing character options much.

[silby]

Added a commit with what I think is a reasonably good approach to headings, which is marking them with §. Only h1-h3 get this treatment; h4-h6 are indented and italicized but I think in any reasonable document hierarchy making them more distinctive with more § is too much ink.

[silby]

rebased and force-pushed.

[jgm]

I'm not a big fan of the new idea for sections. Having section headings indented to the right relative to body text just seems wrong. And I'm not sure I like the section symbols.

Would it be so bad just to make all headings boldface, flush-left, with space above and below? You wouldn't then be able to distinguish heading levels, but how big a drawback is that? If you have a document with many levels and need to keep track of them, you could always render with --number-sections (assuming your writer implements that), and it would make the levels very clear with numbering.

Another option would be, e.g.,

level 1 - bold, underlined, all caps
level 2 - bold, underlined
level 3 - bold
level 4 - underlined
level 5 - italics

or something like that. (Though personally I think this might be ugly.)

[silby]

Numbered sections isn't implemented so far, I can spend some time on it but I think it could go in the backlog for after merge.

I think the minimum I want to ship with is H1s and H2s that are distinguishable from each other and from H3-H6. At this point that'd probably be bold+caps for H1, bold for H2, italic for H3-H6. This is slightly worse for scripts without capitals but I guess most such scripts also commonly don't have italics, so there's some degradation anyway. The remaining option which I have mostly discounted til now is to add a color; since I've used it sparingly elsewhere I think a contrasting color shared by all headings might be helpful after all.

Since you are pretty firmly on the side of keeping it simple I'll do another iteration with something like the above.

[jgm]

That sounds reasonable. I think the idea of having a color for the headings is a good one. (Might as well be the same for all levels, because color isn't a good way to distinguish heading levels.) The scheme you suggest is fine with me. Anyway, we can get a version of this out and see what people say...the heading scheme can always be tweaked in the future.

[jgm]

Implementing numbering shouldn't be too complicated.

The basic approach is to do something like

let blocks' = makeSections (writerNumberSections opts) Nothing blocks

where blocks are the body blocks of the document and opts are the writer options.
If writerNumberSections opts is True, this will add a number attributes on the Header elements, and then you can just have your renderer for headings print these when present, perhaps in a special color to distinguish them from the heading text. Some extra Divs will be added in blocks', but your renderer can just ignore these (as I assume it does already).

[silby]

@jgm Hi, I'm back! Thinking too hard about table rendering plus the temperature in my room during the summer deterred me for a few months but I have finally banged out a reasonable first draft of table rendering for the ANSI writer. I haven't put it through an aggressive stress test but it renders simple tables pretty well. The most obvious thing that doesn't work is rowspans. I don't want to try to work on rowspans any more now than I did in the spring.

With this I believe that all Pandoc elements are rendered by the ANSI writer to a reasonable degree of quality. If you have any further thoughts on the default aesthetics of tables, or anything else I worked on previously, let me know.

Apart from aesthetics, what else needs to be done to merge? Should I add docs + tests to this PR?

[jgm]

I don't know what happened here; I didn't intend to merge this quite yet!

In any case, it's OK I think; it was quite close to being ready. I would have rebased it, though.

[silby]

Haha oops, I was a bit surprised. Good news though, I did rebase it and it looks like it ff-merged. If you make an issue for me with test and docs tasks I’ll try to tackle them.

…

On Sep 3, 2024, at 1:08 PM, John MacFarlane ***@***.***> wrote: I don't know what happened here; I didn't intend to merge this quite yet! In any case, it's OK I think; it was quite close to being ready. I would have rebased it, though. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jgm]

One of the things I'm not sure about is the addition of the Table #: before uncaptioned tables. Perhaps we should only add these labels to captioned tables?

Another question: will the ANSI output look good with various color schemes in the terminal? E.g. if I have white text on a black background?

I think I can take care of basic test and docs.

[silby]

My doclayout ANSI code always resets to the "default" foreground and background colors, rather than apply specific colors from the 8-color palette. I think it's fair to make the assumption that the non-black/white colors (red, green, yellow, blue, magenta, cyan in regular and bold/bright variants) will be legible against the user's background color. When that's not the case users should adjust their color schemes. Here's a random extreme example from one of the default Terminal.app themes, none of which have theme-specific ANSI colors:

I think we'd have to be more careful if the default appearance used the indexed or true color palettes.

[jgm]

I added a basic test: test/ansi-test.txt is the (markdown) input, and test/ansi-test.ansi is the expected output. Feel free to add to the test case if you like.

[silby]

The table thing I did by analogy to figures. It looks like leaving off Table n when there's no caption would match what you get by default with (e.g.) LaTeX output after rendering, though the longtable environments are still numbered whether the caption is printed or not. I'm guessing there's not tons of uniformity here and leaving the table numbers out whether there's a caption or not would be no biggie.

[jgm]

Maybe we should just omit the label "Table N"? It's only really useful when you also have a mechanism for referring to tables, as in LaTeX. And it adds some unwelcome clutter esp. for tables without captions.

[silby]

Yeah I agree

…

On Sep 3, 2024, at 1:51 PM, John MacFarlane ***@***.***> wrote: Maybe we should just omit the label "Table N"? It's only really useful when you also have a mechanism for referring to tables, as in LaTeX. And it adds some unwelcome clutter esp. for tables without captions. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jgm]

I'll make that change.

[bpj]

@silby you mentioned way up there that superscript and subscript are just paranthesised. I wonder if it wouldn't be better to render them in a more Pandoc Markdown or TeX style as ^sup^ ~sub~, ^[sup] -_[sub] or ^{sup} _{sub}? It wouldn't look snazzy but it would preserve the semantics which to me seems more important. A possible alternative might be square brackets + distinctive colors, much like you say links would degrade to be distinguished by color (though FWIW I would prefer for external URLS if any to be shown). FWIW I wrote a filter which I use on top of the plain writer which "wraps" various styled elements in various Unicode brackets like ⟨emph⟩, ⟪strong⟫, ⁅strike⁆, ⦋underline⦌, ⸢sup⸣, ⸤sub⸥ which I sometimes use to view things in the terminal. It's kind of primitive but is a useful fallback when not just using Markdown. (Yes I really hate when information is lost just because renderers are dumb! :-)

[silby]

@bpj those are some good ideas; it'll be easier for me to keep track of them when I come back to this if you make some fresh issues!