TextFormatter port

[Gillibald]

No description provided.

[danipen]

I gave it a try. Good progress!

I found some issues so far, that you probably noticed:

• The intra-line spacing is smaller now. Can we mimic the old metrics?
  
• Empty lines are overlapped:
  
• The syntax highlight is broken.
• The caret only works fine for the first line
• Editing doesn't work fine. It offsets the caret while typing (I only tried editing the first line).
• The inter-line controls are inserted in a wrong position:
  

[danipen]

Hey @Gillibald, I performed another test run. Some more findings:

• Using _textEditor.Options.ShowTabs = true; doesn't work fine:
  

---

• Using _textEditor.Options.ShowSpaces = true; throws an exception in ShapedBuffer

---

• A weird one: typing ANY char at the end of the first line, starts typing the beginning of the line:
  

---

• Add buttons in the middle of the text is still broken.
  

---

• There are some weird chars between control chars:
  

---

• Navigating a big file throws an IndexOutOfRange exception in TextLineImpl:
  

---

• We're leaking thousands of ForegroundTextTransformations. Not sure if this is new or if that was already there:
  

---

• Long lines support is broken again. Editing/selection/going home-end in very long line is super slow. You can find a test file here. We'd need to re-apply this PR to the new codebase.

---

• WordWrap option hangs AvaloniaEdit

---

• Unicode text doesn't work well yet.
  

[danipen]

• After pasting an emoji, such as ❤, the text font on the right is changed:
  

• Caret moving is weird using control characters:

1. The caret doesn't respect the "space before the character box.
2. The caret doesn't move right when it's at the end of the character box (you need to hit Right or Left keys twice).
   NOTE that issue also happens when using folding (you can visualize folding setting up XML in the syntax language combo).
   

[danipen]

• IndexOutOfRangeException opening a binary file as text with ShowBoxForControlCharacters option disabled.
  Yes, I know it's an extreme case, but I think it's worth protecting this kind of exceptions.
  

How to repro:

1. Download this file
2. Disable ShowBoxForControlCharacters in MainWindow.xaml.cs:

_textEditor.Options.ShowBoxForControlCharacters = false;

4. Open the bin file using the following code in MainWindow.xaml.cs:

_textEditor.Document = new TextDocument(
    System.IO.File.ReadAllText(@"c:\path\to\the\file.bin"));

[Gillibald]

Embedded controls are currently not properly hit tested against regular hits and trailing hits. This is something I want to write some tests for this week.

[Gillibald]

I wonder how TabGlypRun ever worked. The run itself should reuse measurements of the replaced sequence otherwise the general text layout changes when ShowTabs is enabled.

[danipen]

Although the implementation was a mess, it was fixed in this PR and it worked fine 😂

bounding-box.mp4

[Gillibald]

I had an idea about long lines. What if we enforce line wrapping if we encounter a certain line length? We might get vertical virtualization that way and it is easier to edit in theory.

[HendrikMennen]

I think the default should be to just cut the lines off after a specific length and show a few dots, like VSCode default behaviour:

Of course it would be nice to have both options.

[danipen]

Well. I'm pretty sure forcing the user to wrap long lines will end in a bunch of support requests for us 😂.

We already did that in the past for the macOS editor, and some users handling minified javascript hit that issue and complained a lot (they don't care about long lines but they don't want to polish the document, wrapping long lines).

Using the VSCode approach could be an option. But IMO, the best editor for handling long lines is Sublime Text. It completely virtualizes lines horizontally. Not sure how, but it does. It just displays the whole line, and you can navigate/edit/select text without any delay, supporting double-byte chars and emojis. So not sure how they do that, but it's doable.

BTW the current version of AvaloniaEdit is able to do that. For long lines, it just assumes the same len for every glyph (when using a monospaced font), and it splits line runs in runs of MAX 3000 chars. Can't we do the same with the new implementation?

[Gillibald]

Without shaping the text, we will never know how many glyphs are rendered in the final result so lazy loading portions of the long line is required aka horizontal virtualization. It is possible to do shaping on mutliple threads in the background but not sure how much we would improve performance that way.

[Gillibald]

Avalonia's TextFormatter is still not 100% matching WPF's version so we need to slightly adjust text processing at certain points. In general this port is mostly working and I am happy that RTL support is also kinda working so we are making huge progress here.

[Gillibald]

@mattj23 Could you test my latest changes? TextWrapping should now work with variable size lines.

[mattj23]

@Gillibald, for some reason I can't get the nightly build you used, I'm getting this error:

AvaloniaEdit.csproj: [NU1102] Unable to find package Avalonia with version (>= 0.10.999-cibuild0020141-beta)
  - Found 197 version(s) in Avalonia Nightly [ Nearest version: 0.10.999-cibuild0020138-beta ]
  - Found 63 version(s) in nuget.org [ Nearest version: 0.10.13 ]

I'll give it a try on a different machine tomorrow and let you know!

[Gillibald]

You need to use this feed: https://nuget.avaloniaui.net/repository/avalonia-all/index.json

[mattj23]

@Gillibald, I tested the latest commit and as best I can tell you've fixed those issues I saw with the blank line height and the variable text height. Looks great.

[danipen]

@Gillibald I think I still found an issue with inline controls:

• Inline controls are still not working as expected. Check this scenario:

• Place the caret in a certain position. Click the "Add button". The button is added in the expected position.

• Place the caret in another position. Click the "Add button". The second button is NOT added in the expected position.
  
  As you can see after that the behavior is erratic.

• Inline controls are moved when editing the file:
  

[danipen]

• Failing tests:
• Visual_Line_Should_Create_Two_Text_Lines_When_Wrapping
• Visual_Line_Should_Create_One_Text_Lines_When_Not_Wrapping

[danipen]

• selection decorator is not accurate (I selected the text using Shift+Left but it also happens when selecting text with the mouse.
  

• in general there are some issues with selection:
  

[danipen]

• InvalidOperationException undoing editions:
• Make some edits in the middle of the document.
• Make some edits at the end of the document.
• Undo the editions with Ctrl+Z
  

[Gillibald]

I have fixed some hit testing and rendering issues here: https://github.com/AvaloniaUI/Avalonia/pull/8094/files

Will test my changes against AvaloniaEdit and will fix the remaining issues

[workgroupengineering]

Hi @Gillibald, do you have any idea when this PR might be merge?

[Gillibald]

This is only blocked by some compiler issues in the test project. Once that is sorted out we can merge.