Adding support for code diff in Code Blocks

[jeromew]

Hello,

I have been looking into adding diff support to code blocks. I will describe here-under what are my findings and the challenges that I see. I think there is a way to reasonably do it but would like to discuss how it could be integrated API wise and UI wise.

current implementation of code blocks

Currently when a code block is created and edited, the text is tokenized with Prismjs which outputs tokens. These tokens are flattened into a sequence of 3 Lexical Nodes: LineBreakNode | TabNode | CodeHighlightNode
These are in turn reconciled into a linear html sequence of <span> and <br>

For example

let i = 0;
i++;

could be transformed into

<span class="PlaygroundEditorTheme__tokenAttr" data-lexical-text="true">let</span>
<span data-lexical-text="true"> i </span>
<span class="PlaygroundEditorTheme__tokenOperator" data-lexical-text="true">=</span>
<span data-lexical-text="true"> </span>
<span class="PlaygroundEditorTheme__tokenProperty" data-lexical-text="true">0</span>
<span class="PlaygroundEditorTheme__tokenPunctuation" data-lexical-text="true">;</span>
<br>
<span data-lexical-text="true">i</span>
<span class="PlaygroundEditorTheme__tokenOperator" data-lexical-text="true">++</span>
<span class="PlaygroundEditorTheme__tokenPunctuation" data-lexical-text="true">;
<br>

The important thing here is that the Lexical token sequence is not hiearchical but linear.
Changing this structure could be possible but would I think require a bit of work so changing the structure from 'linear' to 'tree' would need more thinking as to the necessity of it and to what features it could enable.

the visual structure of a diff

I will consider only a line-by-line diff as it is what we are used to : some lines are green (inserted), some lines are red (deleted)
for this you need to be able to select lines.

After a bit of research, I realize that the linear structure of the HTML is not a perfect fit for this, because I could not find a selector that would be able to select ranges among siblings (like all span siblings between 2 br, of all span siblings starting from span.inserted to br).

There is a CSS draft in discussion css-cascade level 6 that will probably add a @scope-siblings rule that will enable to scope the selection of siblings from span.inserted to br, but it will not be available soon (and firefox still has some road to cover before full adhesion to the notion of @scope from what I see on forums - https://caniuse.com/css-cascade-scope)
I discovered this @scope-siblings discussion from this proposal - w3c/csswg-drafts#8815 where the OP was trying to solve a similar problem to the one we have here.

So it would seem that there is no current solution other than change the html markup generated for code blocks.

BUT there is an interesting rule that I found along the way that could be an acceptable tradeoff : as long as the code lines do not wrap in the editor (which is I think a reasonable expectation in most cases), the rule can select and background-color a whole line only via the first span.inserted of the line.
That rule is

.inserted {
  border-image: conic-gradient(#CFC 0 0) fill 0//0 100vw;
}

.deleted {
  border-image: conic-gradient(#FCC 0 0) fill 0//0 100vw;
}

This would make it easy to have a layout like

that would I think be a good start for an editable diff display with colored inserted/deleted lines + language highlighting

what prismjs has in store

Currently, prismjs already has a way of tokenizing for both diff & language. This is mainly done via a plugin https://prismjs.com/plugins/diff-highlight/

what it does is that

1. it tokenizes the code text with a diff grammar that outputs a sequence of inserted/unchanged/deleted blocks (to simplify, in a diff, the first character of each line is either +/-/space)
2. it then tokenizes each block with the language grammar

For this it uses a sort of "virtual language", "diff-xxx" for diffing language "xxx" so their API is pretty simple.

I am not sure there is a simple way of giving Lexical access to the Prismjs plugins as from what I see in 1.30.0 the plugin hook mechanism happens at the highlight() level and Lexical uses the lower level tokenize() API which do not call the hooks.
But I could port the implementation of diff-highlight into the Lexical code so that it does the same work.

what I would like to discuss is :

• is this feature interesting for the project ?
• would the interim border-image: solution be accepted ? (until the @scope-siblings directive becomes available in browsers)
• how would you see it work API wise ? is the "diff-xxxx" solution used in native prismjs acceptable ? it would be backward compatible when diff is not used, would work when invoked via markdown markup, but would need a bit more work probably on the playground to find a good UX along the language selector.

I will work on a PR for this, but would like to measure its acceptability in this discussion

[jeromew]

closed via #7613