Improve long lines support #172

danipen · 2021-12-21T16:20:19Z

This PR allows AvaloniaEdit to handle long lines with acceptable performance.

Before this PR, AvaloniaEdit only supported lines no longer than 10.000 characters. If a line in the document was longer than 10.000 characters, an exception was raised.

Previous issues handling long lines

Before this PR, removing that restriction, AvaloniaEdit is able to display longer lines, but some issues appear:

The TextMate tokenizer was slow for super long lines.
Editing, navigating, selecting text, etc ... was quite slow.
There is a accumulated error between the position a glyph is drawn and the position the caret is.
In general performance was quite bad with really long lines (>250.000) characters.

Changes made to support long lines

I made several changes to better support very long lines:

Fixed the accumulated error between the glyph position and the caret position: Avalonia's FormattedText measuring accumulates error when measuring glyphs approx above index 3000. Splitting the VisualLine in chunks of 3000 characters fixes the issue.
Disable the VisualLineElementGenerators for long lines: Some generators, such as the one that detects email addresses or hyperlinks that let the user click to open the link, performed badly with long lines because they use regular expressions.
Calculate glyph widths under demand and use cheap measurement for long lines:
1. Do not calculate all the glyph widths at the beginning. Calculating them under demand allows us to improve performance thanks to the virtualization of the ScrollViewer.
2. Measure each glyph for long lines kills performance. So, as an improvement, when using a monospaced font, if we detect a long chunk (>3000 characters), just assign the same width for all glyphs.
Update to TextmateSharp 1.0.14: This version only tokenizes the first 10.000 characters of each line, which is enough for most cases.

Demo video handling long lines

I created a file with a really huge line (~5 million column). After this PR changes, AvaloniaEdit is able to deal with it.

avaloniaedit-huge-line.mov

Avalonia's FormattedText measuring starts to accumulate error when measuring glyphs aprox at index 3000. Splitting the VisualLine in chunks of 3000 characters fixes that. Additionally, editing, selection and navigation performance is better if we split the line in smaller chunks.

…g lines Two changes here: 1) Do not calculate all the glyph widths at the beginning. Calculating them under demand allows us to improve performance thanks to the virtualization of the ScrollViewer. 2) Measure each glyph for long lines kills performance. So, as an improvement, when using a monospaced font, if we detect a long chunk (>3000 characters), just assign the same width for all glyphs.

Gillibald · 2021-12-21T16:46:40Z

src/AvaloniaEdit/Text/TextLineRun.cs

+
+            double MeasureGlyphAt(int index)
+            {
+                return new FormattedText


I think we should rewrite this to use the current platform's method of measuring a character and avoid constructing a FormattedText instance for every character.

For Skia https://docs.microsoft.com/en-us/dotnet/api/skiasharp.skpaint.getglyphwidths?view=skiasharp-2.80.2#SkiaSharp_SKPaint_GetGlyphWidths_System_ReadOnlySpan_System_Char__

@Gillibald, thank you for your comments. I would just to clarify... What do you exactly mean?

Avalonia already implements something to get all the glyph widths?

We should call SKPaint.GetGlyphWidths directly from AvaloniaEdit?

We should implement some API in Avalonia to get the glyph widths from SKPaint.GetGlyphWidths?

Avalonia already supports getting a glyph's width via GlyphTypeface so we might just be able to use that API.

I have added this code lately and it should work for all platforms.

https://github.com/AvaloniaUI/Avalonia/blob/master/src/Skia/Avalonia.Skia/FormattedTextImpl.cs#L715

…I/AvaloniaEdit into fix-long-lines-support

Gillibald

LGTM

HendrikMennen · 2022-01-20T12:02:46Z

I think this commit is the reason I constantly get this exception:

System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at AvaloniaEdit.Text.TextLineRun.GlyphWidths.GetAt(Int32 index) in C:\Users\HendrikMennen\source\repos\VHDP\AvaloniaEdit\src\AvaloniaEdit\Text\TextLineRun.cs:line 364
   at AvaloniaEdit.Text.TextLineRun.GetCharacterFromDistance(Double distance) in C:\Users\HendrikMennen\source\repos\VHDP\AvaloniaEdit\src\AvaloniaEdit\Text\TextLineRun.cs:line 326
   at AvaloniaEdit.Text.TextLineImpl.GetCharacterFromDistance(Double distance) in C:\Users\HendrikMennen\source\repos\VHDP\AvaloniaEdit\src\AvaloniaEdit\Text\TextLineImpl.cs:line 187
   at AvaloniaEdit.Rendering.VisualLine.GetVisualColumnFloor(Point point, Boolean allowVirtualSpace, Boolean& isAtEndOfLine) in C:\Users\HendrikMennen\source\repos\VHDP\AvaloniaEdit\src\AvaloniaEdit\Rendering\VisualLine.cs:line 591
   at AvaloniaEdit.Rendering.VisualLine.GetVisualColumnFloor(Point point, Boolean allowVirtualSpace) in C:\Users\HendrikMennen\source\repos\VHDP\AvaloniaEdit\src\AvaloniaEdit\Rendering\VisualLine.cs:line 568
   at AvaloniaEdit.Rendering.VisualLine.GetVisualColumnFloor(Point point) in C:\Users\HendrikMennen\source\repos\VHDP\AvaloniaEdit\src\AvaloniaEdit\Rendering\VisualLine.cs:line 559
   at AvaloniaEdit.Rendering.TextView.GetVisualLineElementFromPosition(Point visualPosition) in C:\Users\HendrikMennen\source\repos\VHDP\AvaloniaEdit\src\AvaloniaEdit\Rendering\TextView.cs:line 1710
   at AvaloniaEdit.Rendering.TextView.OnPointerMoved(PointerEventArgs e) in C:\Users\HendrikMennen\source\repos\VHDP\AvaloniaEdit\src\AvaloniaEdit\Rendering\TextView.cs:line 1637
   at System.Reactive.Subjects.Subject`1.OnNext(T value) in /_/Rx.NET/Source/src/System.Reactive/Subjects/Subject.cs:line 145
   at Avalonia.Interactivity.EventRoute.RaiseEventImpl(RoutedEventArgs e) in /_/src/Avalonia.Interactivity/EventRoute.cs:line 148
   at Avalonia.Interactivity.EventRoute.RaiseEvent(IInteractive source, RoutedEventArgs e) in /_/src/Avalonia.Interactivity/EventRoute.cs:line 79
   at Avalonia.Interactivity.Interactive.RaiseEvent(RoutedEventArgs e) in /_/src/Avalonia.Interactivity/Interactive.cs:line 123
   at Avalonia.Input.MouseDevice.MouseMove(IMouseDevice device, UInt64 timestamp, IInputRoot root, Point p, PointerPointProperties properties, KeyModifiers inputModifiers) in /_/src/Avalonia.Input/MouseDevice.cs:line 268
   at Avalonia.Input.MouseDevice.ProcessRawEvent(RawPointerEventArgs e) in /_/src/Avalonia.Input/MouseDevice.cs:line 142
   at Avalonia.Input.InputManager.ProcessInput(RawInputEventArgs e) in /_/src/Avalonia.Input/InputManager.cs:line 37
   at Avalonia.Win32.WindowImpl.AppWndProc(IntPtr hWnd, UInt32 msg, IntPtr wParam, IntPtr lParam) in /_/src/Windows/Avalonia.Win32/WindowImpl.AppWndProc.cs:line 487
   at Avalonia.Win32.WindowImpl.WndProc(IntPtr hWnd, UInt32 msg, IntPtr wParam, IntPtr lParam) in /_/src/Windows/Avalonia.Win32/WindowImpl.WndProc.cs:line 33
   at Avalonia.Win32.Interop.UnmanagedMethods.DispatchMessage(MSG& lpmsg)
   at Avalonia.Win32.Win32Platform.RunLoop(CancellationToken cancellationToken) in /_/src/Windows/Avalonia.Win32/Win32Platform.cs:line 205
   at Avalonia.Threading.Dispatcher.MainLoop(CancellationToken cancellationToken) in /_/src/Avalonia.Base/Threading/Dispatcher.cs:line 61
   at Avalonia.Controls.ApplicationLifetimes.ClassicDesktopStyleApplicationLifetime.Start(String[] args) in /_/src/Avalonia.Controls/ApplicationLifetimes/ClassicDesktopStyleApplicationLifetime.cs:line 132
   at Avalonia.ClassicDesktopStyleApplicationLifetimeExtensions.StartWithClassicDesktopLifetime[T](T builder, String[] args, ShutdownMode shutdownMode) in /_/src/Avalonia.Controls/ApplicationLifetimes/ClassicDesktopStyleApplicationLifetime.cs:line 187

This happens quite often when I use folding and then click on the document.
Also using invalid characters (BEL, ESC or emojis?) seem to break this and sometime cause this crash.
These characters cause crashes:

Also using folding: Probably because of the [...] box?

danipen · 2022-01-20T12:14:34Z

@HendrikMennen thanks for reporting.

Yes, you're right. The failing code was added on this PR. It seems under some scenarios there is an issue measuring the glyph widths.

Please, file a new issue, including a consistent repro case, and I'll try to fix it.

danipen added 7 commits December 21, 2021 12:20

Do not process long lines in LinkElementGenerator

8c56b3f

Disable all generators for long lines

ccc8f66

Remove the restriction of 10000 characters per line

a388286

Added a status bar to visualize current line and column

c653bda

Upgrade to TextMateSharp 1.0.14

d885c6d

danipen requested review from danwalmsley and Gillibald December 21, 2021 16:20

Merge branch 'master' into fix-long-lines-support

d24a8e0

Gillibald reviewed Dec 21, 2021

View reviewed changes

danipen added 2 commits December 22, 2021 08:47

Get glyph's width via GlyphTypeface

6548f57

Merge branch 'fix-long-lines-support' of https://github.com/AvaloniaU…

a70b049

…I/AvaloniaEdit into fix-long-lines-support

danipen requested a review from Gillibald December 22, 2021 07:48

Gillibald approved these changes Dec 22, 2021

View reviewed changes

danipen added 2 commits December 22, 2021 10:01

Merge branch 'master' into fix-long-lines-support

d7bd131

Merge branch 'master' into fix-long-lines-support

3aab302

danipen merged commit 314f6ce into master Dec 22, 2021

danipen mentioned this pull request Jan 22, 2022

Index out of range measuring glyphs #189

Closed

Takoooooo deleted the fix-long-lines-support branch February 14, 2022 16:06

danipen mentioned this pull request Mar 10, 2022

TextFormatter port #216

Merged

mattj23 mentioned this pull request Apr 2, 2022

Generator support for long lines #223

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve long lines support #172

Improve long lines support #172

danipen commented Dec 21, 2021 •

edited

Gillibald Dec 21, 2021

danipen Dec 21, 2021 •

edited

Gillibald Dec 21, 2021

Gillibald left a comment

HendrikMennen commented Jan 20, 2022 •

edited

danipen commented Jan 20, 2022 •

edited

Improve long lines support #172

Improve long lines support #172

Conversation

danipen commented Dec 21, 2021 • edited

Previous issues handling long lines

Changes made to support long lines

Demo video handling long lines

Gillibald Dec 21, 2021

Choose a reason for hiding this comment

danipen Dec 21, 2021 • edited

Choose a reason for hiding this comment

Gillibald Dec 21, 2021

Choose a reason for hiding this comment

Gillibald left a comment

Choose a reason for hiding this comment

HendrikMennen commented Jan 20, 2022 • edited

danipen commented Jan 20, 2022 • edited

danipen commented Dec 21, 2021 •

edited

danipen Dec 21, 2021 •

edited

HendrikMennen commented Jan 20, 2022 •

edited

danipen commented Jan 20, 2022 •

edited