Fix bug in selection handling in dom renderer

[PerBothner]

If you use the dom renderer and make a selection, the visible selection extends to the end of the line. This fixes that.

This also fixes a logic weirdo: A || (!A && B) should be simplified to A || B.

[jerch]

Suggestion - for the sake of readability lets separate concerns here:

  // selection state must not change
  && isInSelection === oldIsInSelection
  // outside of selections BG must match
  && (!isInSelection && cell.bg === oldBg)
  // default FG must match
  // exception: in selections selectionForeground may override FG
  && (cell.fg === oldFg || (isInSelection && colors.selectionForeground))

To me thats much easier to grasp than deep nested conditions, hopefully I covered all cases 😸.

The FG rule is still nested, at least it pulls the more likely case to the left, which should result in faster processing.

[PerBothner]

That re-write seems to work fine my testing. And I'm inclined to agree it is cleaner. I was aiming for a minimal fix, but I'm happy also doing minor logic simplification.

[Tyriar]

We have multiple comments for the same lines here, we should either move these up, or change the top comments to // Check if cells can be merged and then inline the other comments

[Tyriar]

Tests are failing pointing out less optimal results, can we handle this case too?

  1) DomRendererRowFactory
       createRow
         selectionForeground
           should force whitespace cells to be rendered above the background:

      AssertionError: expected '<span style="background-color:#ff0000…' to equal '<span style="background-color:#ff0000…'
      + expected - actual

      -<span style="background-color:#ff0000;" class="xterm-decoration-top"> </span><span style="background-color:#ff0000;" class="xterm-decoration-top">a</span>
      +<span style="background-color:#ff0000;" class="xterm-decoration-top"> a</span>
      
      at Context.<anonymous> (src/browser/renderer/dom/DomRendererRowFactory.test.ts:319:16)
      at processImmediate (node:internal/timers:476:21)

[jerch]

It breaks out of the merge here because FG is not set on the empty cell. I think this should fix it:

&& ((chars !== ' ' && cell.fg === oldFg) || (isInSelection && colors.selectionForeground))

This adds another merge chance - if there is no content in a cell, it can be merged regardless of FG.
Edit: Oh wait, not sure if FG is free of other SGR attributes, that would also show up...
Edit2: Nope that doesnt work, as INVERSE is also stored in FG. So this needs a slightly different patch. (Ideally we'd have on fg only fg-related attributes, well INVERSE deals with fg & bg stuff)

[jerch]

Looking at it again, I think the simplifications are all slightly off because of the way we store SGR flags on bg and fg. In fact we'd need color inspection for the selection tests here. The difference in certain flags should still break out of the merge (otherwise we lose some SGR flag decorations during selection).
Thats the reason why I went with fg == oldFg && bg === oldBg in the first place as an opportunistic approach, that, if in doubt, falls back to single span rendering.

[PerBothner]

Maybe for now just go with my original minimal fix?

[PerBothner]

FWIW In my buffer-cell-cursor branch I added to IAttributeData new accessor functions getFg and getBg which return combined getFgColor+getFgColorMode and getBgColor+getBgColorMode as a 26-bit int. I think that is a cleaner way of dealing with colors, with a new getStyleFlags function for the SGR flags and more (up to 28 bits). Maybe we can revisit further optimizations/cleanups later.

[jerch]

The initial commit does look safer, fixes problems and passes the tests so that's cool with me.

Im pretty sure that INVERSE gets swallowed atm by selections (edit: and when a selecton fg is defined), its just that we dont have a test for it (not tested though, just guessing). But that might be fine? No clue whats expected to happen with INVERSE in that case...

[Tyriar]

Inverse when there's no selectionForeground won't get covered 🤯

[jerch]

I guess we need to consider FgFlags.INVERSE for the background check and BgFlags.DIM for the foreground check? Not sure if extended underlines are handled. We need some failing tests to fix against.

Yeah I think we should cleanup the flag association a bit and put any font/content altering flag into fg.

[Tyriar]

#4682

[jerch]

Can do the nasty flag rework (that affects quite some code places), but it has to wait, as I am still on Per's big grapheme PR (only halfway through so far 😺).

So for now I am fine if we add this PR as Per suggested initially.

[PerBothner]

Not sure the best way to undo the last commit. Can I remove the last commit from the PR? Should I use a git revert - which will make a needless-complicted history - or can you clean that up on merge?

[jerch]

@PerBothner
If you want a clean repo log afterwards, imho revert + a force push should do.

[Tyriar]

Since we've decided what to do you can go either way. I don't normally care about many commits going in PRs as you can always revert by merge commit in the off chance we need to.

[PerBothner]

@jerch "imho revert + a force push should do".

Did you mean git reset + git push --force? I did that, and it seemed to work. However, the tests reported failures, which I'm looking into.

[jerch]

Well its either:

git revert <commit to revert>
git push --force

or (HEAD~1 assumes to undo last commit):

git reset --soft HEAD~1
git push --force

[PerBothner]

[I did a plain git reset 957e3e0f08606678be8e9346113b0aaefba4e433 followed by git push --force. After I cleaned up the local directoyr (git diff|git apply -R -) the tests passed.

[PerBothner]

I think it's ready for merge now.