Styled markdown, with a few tweaks #51928

tecosaur · 2023-10-29T13:56:54Z

With StyledStrings and JuliaSyntaxHighlighting we can make the rendering of Markdown.MD:

prettier
customisable
more composable

Since this is mostly about rendering, here are some screenshots:

Current

New default

New customised

Depends on #51807, #51810, and #51869

tecosaur · 2024-02-01T18:49:02Z

What do you know, all the dependent PRs are now merged 🥳. We can now think about reviewing and merging this.

tecosaur · 2024-02-02T06:54:16Z

I'll push a version with the fixed pkgimage.mk and tweaks based on API changes that happened in #51807 once #53155 is merged.

pkgimage.mk

stdlib/Markdown/src/render/terminal/formatting.jl

stdlib/Markdown/src/render/terminal/render.jl

stdlib/Markdown/src/render/terminal/formatting.jl

tecosaur · 2024-02-07T20:17:32Z

@vtjnash any further thoughts on the current version?

vtjnash · 2024-02-08T21:01:41Z

stdlib/Markdown/src/render/terminal/render.jl

+    elseif md.language == "julia-repl" || Base.startswith(md.language, "jldoctest")
+        hl = AnnotatedString(md.code)
+        for (; match) in eachmatch(r"(?:^|\n)julia>", hl)
+            StyledStrings.face!(match, :repl_prompt_julia)


This seems a bit suspect to imply that applying a face! to the result of eachmatch will alter the string hl

When taking a substring of an AnnotatedString, the underlying string is not mutable, but the annotations are. This is rather convenient when you want to search for patterns to style (no other convenient patterns are currently possible, actually).

vtjnash · 2024-02-08T21:02:06Z

stdlib/Markdown/src/render/terminal/render.jl

+        hl = AnnotatedString(md.code)
+        for (; match) in eachmatch(r"(?:^|\n)julia>", hl)
+            StyledStrings.face!(match, :repl_prompt_julia)
+            afterprompt = match.offset + match.ncodeunits + 1


There does not appear to be a field ncodeunits (it is a function)

The match is of type SubString{AnnotatedString}, so it indeed has a field ncodeunits. May as well use the function though.

vtjnash · 2024-02-08T21:08:12Z

stdlib/Markdown/src/render/terminal/render.jl

+        for (; match) in eachmatch(r"(?:^|\n)julia>", hl)
+            StyledStrings.face!(match, :repl_prompt_julia)
+            afterprompt = match.offset + match.ncodeunits + 1
+            _, exprend = Meta.parse(md.code, afterprompt, raise = false)


It feels a bit tricky to have a full REPL here, but anyways, I think this is supposed to use Base.parse_input_line aka Meta.parseall here and repeatedly add lines until the expression is complete. The result may not be quite the same as eachmatch would return, but may be closer to what eachline returns, followed by if ismatch followed by Meta.parseall consuming more lines until it reaches a concluding statement (a blank line or a complete expression or a parse error)

This is modelled after the current behaviour of OhMyREPL which seems to work. I think Kristoffer and I had a brief conversation on this on Slack where I asked "why Meta.parse not Meta.parseatom?" but I forget the details and it's been swallowed by the Slack-hole at this point 🥲.

I'll also note that it's actually somewhat difficult to tell the behavioural difference between parse, parseatom, and parseall as the latter two have no docstrings and just call Core._parse, which itself seems to have no (useful) comments.

vtjnash · 2024-02-08T21:13:07Z

stdlib/Markdown/src/render/terminal/render.jl

+function term(io::IO, md::MarkdownElement, columns)
+    a = IOContext(AnnotatedIOBuffer(), io)
+    term(a, md, columns)
+    print(io, read(seekstart(a.io), AnnotatedString))


Isn't there an optimized version of this, for copying between IO objects without the intermediate copy?

Suggested change

print(io, read(seekstart(a.io), AnnotatedString))

write(io, seekstart(a.io))

Or is the problem that io might be wrapped up somewhat, so that dispatch currently won't reliably know if it is annotation-capable and end up discarding those?

io may be stdout, and we currently don't have a specialised print/write method for AnnotatedIOBuffer to .

I'd suggest that we perhaps add a "TODO change to direct write after implementing specialised AnnotatedIOBuffer printing"-type mental note or comment so we can come back to this if/when we do so.

vtjnash · 2024-02-08T21:16:12Z

stdlib/Markdown/src/render/terminal/render.jl

-            i < lastindex(L) && println(io)
+    code = if md.language ∈ ("", "julia")
+        highlight(md.code)
+    elseif md.language == "julia-repl" || Base.startswith(md.language, "jldoctest")


Actually, could you drop this specific particular part of the PR, merge everything else, and then add this in a later PR? It is a bit of a bonus feature, and I don't want it to hold up everything else

(perhaps just add it to the list along with "julia" to get some primitive highlighting, or just leave it bare for now)

Sure, I'll bundle it with julia for now.

For future reference, this is the code being removed:

elseif md.language == "julia-repl" || Base.startswith(md.language, "jldoctest") hl = AnnotatedString(md.code) for (; match) in eachmatch(r"(?:^|\n)julia>", hl) StyledStrings.face!(match, :repl_prompt_julia) afterprompt = match.offset + ncodeunits(match) + 1 _, exprend = Meta.parse(md.code, afterprompt, raise = false) highlight!(hl[afterprompt:prevind(md.code, exprend)]) if (nextspace = findnext(' ', md.code, exprend)) |> !isnothing nextword = hl[exprend:prevind(hl, nextspace)] if nextword == "ERROR:" StyledStrings.face!(nextword, :error) end end end hl

vtjnash · 2024-02-10T19:36:14Z

stdlib/Markdown/src/render/terminal/render.jl

-            print(io, ' '^margin, L[i])
-            i < lastindex(L) && println(io)
-        end
+    code = if md.language ∈ ("", "julia", "julia-repl", "jldoctest")


Suggested change

code = if md.language ∈ ("", "julia", "julia-repl", "jldoctest")

code = if md.language ∈ ("julia", "julia-repl", "jldoctest")

just arbitrary <raw> blocks marked with ``` shouldn't get julia-formatted, right?

I guess it depends how you interpret them. I tend to see them as "whatever the default language is" which on GitHub is text and in Julia docstrings is julia.

Ultimately, it's a heuristic choice, and I may well have gone the wrong way here.

I think raw is the conservative choice here to stick with

Sure. It would be nice to syntax highlight function signatures though, which usually appear like so:

""" annotate!(char::AnnotatedChar, label::Symbol => value) Annotate `char` with the pair `label => value`. """

True. Probably should be a separate PR though, since it is potentially a more significant change that downstream consumers would not be able to distinguish between annotated and un-annotated content

Yes, the initial code element of the docstring would be highlighted as Julia code, if that's what you mean.

@KristofferC might have some helpful comments here, since OhMyREPL does Julia syntax highlighting of markdown code blocks.

It seems like we would need a way to distinguish arbitrary Markdown that is being formatted from specifically Julia docstrings that are being rendered

The only way to conclusively tell, is to use language annotations on code blocks. Short of that, it's a guess/default. I think "guess Julia because we're in Julia" and requiring raw/text to be explicit makes sense, but as you demonstrated earlier that's not the only reasonable position.

If defaulting to rendering Julia syntax gives us highlighting in the docstrings and otherwise not, then I think we should default to Julia.

fingolfin · 2024-02-23T08:10:54Z

@vtjnash you added a "merge me" label but did not approve the PR... Should this be read as an "implicit approval"?

fingolfin · 2024-02-23T08:11:30Z

There seem to be genuine errors in the CI job logs

tecosaur · 2024-04-27T11:50:24Z

From the comments earlier, I take it that this should be fine to merge as soon as CI is sorted. I'll try to give this a look shortly.

tecosaur · 2024-04-28T03:11:49Z

Hmmm, almost all of the test failures are Pkg related. @KristofferC might you have any thoughts here?

tecosaur · 2024-04-28T03:13:48Z

The other test failure is from REPL:

REPL                                             (16) |         failed at 2024-04-10T19:54:00.266
Test Failed at /cache/build/tester-amdci5-9/julialang/julia-master/julia-e217e1ea80/share/julia/stdlib/v1.12/REPL/test/precompilation.jl:37
  Expression: n_precompiles <= expected_precompiles
   Evaluated: 1 <= 0

I'm not sure what's going on here.

tecosaur · 2024-04-30T11:20:26Z

The Pkg CI errors will be resolved once JuliaLang/Pkg.jl#3884 is merged, and Pkg bumped.

It's convenient for dispatch.

Using StyledStrings for styled printing has a number of benefits, including but not limited to: - Italics "just working" on terminals that announce support - Functioning links, for the first time - Greater compossibility of rendered markdown content - Customisability of the printing style Then with JuliaSyntaxHighlighting, we get support for syntax-highlighted Julia code too.

The spacing between list items might as well represent whether the list is a tight or loose list.

It seems to make sense not to treat everything other than "rst" as Julia. We may as well follow the same heuristics as the terminal rendering for consistency.

Since we're already using StyledStrings for rendering Julia in the terminal, we can also handle "styled"-labelled code blocks fancily, thanks to the `styled` function provided by StyledStrings. In all non-terminal contexts, the styling metadata is simply discarded, but could be used in the future (for instance StyledStrings currently supports HTML output too).

In the course of the markdown PR, an issue with the use of deepcopy in StyledStrings was revealed. This has now been fixed, and to obtain the fix StyledStrings is bumped.

tecosaur · 2024-05-02T10:25:35Z

We might be in the clear now! 🎉 🤞

stdlib/Markdown/src/render/terminal/render.jl

tecosaur · 2024-05-07T02:03:58Z

If there are no objections, I'll merge this shortly.

mortenpi · 2024-05-07T02:06:05Z

Maybe just a quick check to do if you have the branch compiled: do MarkdownAST and Documenter test suites pass with this?

tecosaur · 2024-05-07T02:11:57Z

Thanks for that suggestion Morten, I do indeed and the results are:

MarkdownAST: all tests pass
Documenter: fails to load due to libgit2 error

mortenpi · 2024-05-07T02:14:53Z

Documenter: fails to load due to libgit2 error

Interesting, but that seem completely unrelated, and so I think we can ignore that. Documenter's reliance on the Markdown stdlib should mostly be covered by the MarkdownAST tests anyway. Thanks for checking!

tecosaur · 2024-05-09T16:34:05Z

Since it's been a while, Triage thinks we should just let this appear in 1.12.

Fixes #54399 by re-introducing the code seperated out from the styled Markdown PR at Jameson's request (#51928 (comment)). The code itself is modelled after [equivalent code in OhMyREPL](https://github.com/KristofferC/OhMyREPL.jl/blob/b0071f5ee785a81ca1e69a561586ff270b4dc2bb/src/MarkdownHighlighter.jl#L15-L31). The new `markdown_julia_prompt` face allows people to make the "prompt" shown in Markdown code visually distinct, to [avoid confusing it with the REPL prompt at a glance](KristofferC/OhMyREPL.jl#100). By way of example, I make it italic by augmenting my `faces.toml` with ```toml [markdown] julia_prompt = { italic = true } ```

tecosaur force-pushed the styled-markdown branch from 0bfb3b2 to 453a91b Compare October 29, 2023 13:58

tecosaur added domain:display and printing Aesthetics and correctness of printed representations of objects. stdlib:markdown kind:don't squash Don't squash merge labels Oct 29, 2023

tecosaur mentioned this pull request Oct 29, 2023

Meta-issue for improving the REPL experience #50817

Open

15 tasks

tecosaur mentioned this pull request Jan 31, 2024

Introduce AnnotatedIOBuffer #51807

Merged

vtjnash reviewed Feb 5, 2024

View reviewed changes

pkgimage.mk Outdated Show resolved Hide resolved

vtjnash reviewed Feb 5, 2024

View reviewed changes

stdlib/Markdown/src/render/terminal/formatting.jl Outdated Show resolved Hide resolved

vtjnash reviewed Feb 5, 2024

View reviewed changes

stdlib/Markdown/src/render/terminal/render.jl Outdated Show resolved Hide resolved

vtjnash reviewed Feb 5, 2024

View reviewed changes

stdlib/Markdown/src/render/terminal/render.jl Outdated Show resolved Hide resolved

vtjnash reviewed Feb 5, 2024

View reviewed changes

stdlib/Markdown/src/render/terminal/render.jl Outdated Show resolved Hide resolved

vtjnash reviewed Feb 5, 2024

View reviewed changes

stdlib/Markdown/src/render/terminal/formatting.jl Outdated Show resolved Hide resolved

stdlib/Markdown/src/render/terminal/formatting.jl Outdated Show resolved Hide resolved

tecosaur force-pushed the styled-markdown branch 2 times, most recently from c2d7340 to 8639b74 Compare February 7, 2024 20:12

vtjnash reviewed Feb 8, 2024

View reviewed changes

tecosaur force-pushed the styled-markdown branch from 8639b74 to 4d1ffb4 Compare February 10, 2024 18:51

vtjnash reviewed Feb 10, 2024

View reviewed changes

vtjnash added the status:merge me PR is reviewed. Merge when all tests are passing label Feb 10, 2024

tecosaur force-pushed the styled-markdown branch from bd47dc8 to 8151af1 Compare February 11, 2024 18:56

tecosaur added the backport 1.11 Change should be backported to release-1.11 label Feb 16, 2024

vtjnash removed the status:merge me PR is reviewed. Merge when all tests are passing label Feb 23, 2024

tecosaur force-pushed the styled-markdown branch 3 times, most recently from 008be87 to 3927f8e Compare April 30, 2024 02:29

tecosaur mentioned this pull request Apr 30, 2024

Relax overly-specific manifest tests JuliaLang/Pkg.jl#3884

Merged

tecosaur force-pushed the styled-markdown branch from 3927f8e to c12142f Compare April 30, 2024 15:27

tecosaur added 6 commits May 2, 2024 18:24

Introduce MarkdownElement abstract type

c04d40d

It's convenient for dispatch.

Variable spacing markdown list rendering

d998d7c

The spacing between list items might as well represent whether the list is a tight or loose list.

Synchronise Markdown rendering to RST with term

80dbf1e

It seems to make sense not to treat everything other than "rst" as Julia. We may as well follow the same heuristics as the terminal rendering for consistency.

Bump StyledStrings

ada49c3

In the course of the markdown PR, an issue with the use of deepcopy in StyledStrings was revealed. This has now been fixed, and to obtain the fix StyledStrings is bumped.

tecosaur force-pushed the styled-markdown branch from c12142f to ada49c3 Compare May 2, 2024 10:24

kimikage reviewed May 5, 2024

View reviewed changes

stdlib/Markdown/src/render/terminal/render.jl Show resolved Hide resolved

tecosaur merged commit 1614e11 into JuliaLang:master May 7, 2024
7 checks passed

tecosaur mentioned this pull request May 8, 2024

REPL stdout output should not be highlighted in code examples #54399

Closed

tecosaur deleted the styled-markdown branch May 9, 2024 10:04

tecosaur removed the backport 1.11 Change should be backported to release-1.11 label May 9, 2024

tecosaur mentioned this pull request May 9, 2024

Highlight julia-repl code in Markdown specially #54423

Merged

tecosaur added the kind:feature Indicates new feature / enhancement requests label May 11, 2024

	print(io, read(seekstart(a.io), AnnotatedString))
	write(io, seekstart(a.io))

	code = if md.language ∈ ("", "julia", "julia-repl", "jldoctest")
	code = if md.language ∈ ("julia", "julia-repl", "jldoctest")

Styled markdown, with a few tweaks #51928

Styled markdown, with a few tweaks #51928

Conversation

tecosaur commented Oct 29, 2023

tecosaur commented Feb 1, 2024

tecosaur commented Feb 2, 2024

tecosaur commented Feb 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tecosaur Feb 10, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fingolfin commented Feb 23, 2024

fingolfin commented Feb 23, 2024

tecosaur commented Apr 27, 2024

tecosaur commented Apr 28, 2024 • edited

tecosaur commented Apr 28, 2024

tecosaur commented Apr 30, 2024 • edited

tecosaur commented May 2, 2024

tecosaur commented May 7, 2024

mortenpi commented May 7, 2024

tecosaur commented May 7, 2024

mortenpi commented May 7, 2024 • edited

tecosaur commented May 9, 2024

tecosaur Feb 10, 2024 •

edited

tecosaur commented Apr 28, 2024 •

edited

tecosaur commented Apr 30, 2024 •

edited

mortenpi commented May 7, 2024 •

edited