rework positioning/rendering and enable softwrap/virtual text

[pascalkuthe]

Supersedes #5008

This PR is a large rework of the core text positioning and rendering code to
remove the assumption that on-screen columns/lines correspond to text columns/
lines. This is required to implement both softwrap and virtual text.

Both of these features are included in this PR, so the original goals outlined
in #5008 can be achieved with this PR.

Implementation

This PR introduces generic DocFormatter that positions graphemes from various
sources. This formatter is used both for rendering and for movements/scrolling.
The design is heavily inspired by led, although I streamlined/adjusted some
Details as helix needs are different (especially with the introduction of
virtual text).

The basics of the softwrap implementation from #5008 mostly remains (so indent
carry over, and word splitting are supported) but rendering and positioning are
cleanly separated now and much of the architecture explained in #5008 is not
present anymore as I had to evolve the architecture for the positioning code.

The DocFormatter is still a single grapheme iterator (so more efficient)
that transverses the entire document. However, it does not deal with syntax
highlighting or rendering at all. It's an iterator that is only concerned witch placing
graphemes from the document and virtual text at the correct position.
Note that the DocFormatter does not deal with the contents of line virtual text
as that is easily handeled trough the new LineDecoration rendering mechansim.

The DocFormatter is constructed from an anchor char index which it backtracks
to the last known line breaks. These "known" line breaks are called blocks and
were also described by @cessen in #136. Right now only normal line breaks are
considered as blocks, but the code is written so long lines can be chunked for
improved performance for files with very long text in the future.
I did not implement this logic in this PR as it's only an optimization and can be
implemented later as suggested by @cessen.

I added the functions visual_offset_from_anchor/visual_offset_from_block
and char_idx_at_visual_offset to the position module in helix core
that utilize the DocFormatter. These replace visual_coords_at_pos and
pos_at_visual_cords respectively. Their names are different to more accurately
reflect their functions. The old functions have been replaced everywhere except
for the align_selections command. I am not sure if virtual text and softwrap
should affect selection alignment and refactoring this function was more
involved, so I left it untouched for now.

I have also updated the vertical movement command to be softwrap aware which
feels much more intuitive than the behavior in other editor like nvim. If there
is demand for vertical movements that ignore softwrap a separate command could
be added that simply sets softwrap = false in the TextFormat before passing
it to move vertically. Note that many commands (like goto_line_end) are not softwrap
aware yet. I am not sure if that is desirable and therefore left it to future work.

The rendering code has been somewhat streamlined and moved to document.rs to
allow reuse in other views as it had to be rewritten from to utilize
the new DocFormatter anyway.

Future Work

• Make various decorations softwrap aware
  • Display diagnostic markers on the visual line they occur
  • Display cursor row only on the selected visual line instead of the entire document line
    challenging as the char range of a visual line is only known once rendered, but
    the style needs to be applied before the line is rendered
• Potentially adjust some commands to operate on visual lines instead of document lines
• Chunk very long lines into blocks (around 4096 chars) where a wrap is forced to improve performance
• enable sofwrap by default in the future once it had time to mature?
• add a language specifc softwrap setting so softwrap can be enabled by default for some langauges like markdown

[pascalkuthe]

This PR is a large change to many core components of helix and therefore needs testing before merging.
I found this branch to work well now in my testing but I would appreciate more feedback.

You can enable softwrap either using :set softwrap.enable true or by setting soft-wrap.enable = true in the [editor] section of your configuration.

While virtual text is implemented in this PR, it does not include any functionality that actually makes use of it so it's harder to test altough rebasing #3791 would be neat.

[kirawi]

Is this "fully" implemented and usable for editing?

[pascalkuthe]

Is this "fully" implemented and usable for editing?

yeah unless I missed something. There is nothing intentionally left out

[lazytanuki]

Thanks a lot for working on this, gonna daily drive it a bit but already looks awesome !
I just have a question about what should be the defaults for relative vertical jumps. As of now, with relative line numbers enabled, jumping to a relative line does not line up with the displayed relative line numbers because we jump virtual lines instead of document lines.

To overcome this, I see the following solutions for the defaults:

• we keep the default vertical motions on virtual lines (feels more intuitive as you say), keep the current display of relative lines, and make it so that a relative jump moves across (however I see how it can break the current implementation logic given the way repeated motions are implemented...)
• we keep the default vertical motions on virtual lines and find a way to display relative virtual line numbers in the gutter
• we restore the defaults to vertical motions on document lines and try to find a new, intuitive default for virtual line motions.

I personally prefer the first point, mainly because this was NeoVim's behavior once you had remapped vertical motions to use virtual lines, but I'm curious about your thoughts on this.

Thanks !

[CBenoit]

I’m going to daily drive this as well!

You can enable softwrap either using :set softwrap.enable true or by setting soft-wrap.enable = true in the [editor] section of your configuration.

Is there a reason for the name of the option to not be the same in both contexts? (both softwrap or both soft-wrap)

[pascalkuthe]

I’m going to daily drive this as well!

You can enable softwrap either using :set softwrap.enable true or by setting soft-wrap.enable = true in the [editor] section of your configuration.

Is there a reason for the name of the option to not be the same in both contexts? (both softwrap or both soft-wrap)

It's always soft-wrap that was just a typo in my comment

[pascalkuthe]

Thanks a lot for working on this, gonna daily drive it a bit but already looks awesome ! I just have a question about what should be the defaults for relative vertical jumps. As of now, with relative line numbers enabled, jumping to a relative line does not line up with the displayed relative line numbers because we jump virtual lines instead of document lines.

To overcome this, I see the following solutions for the defaults:

* we keep the default vertical motions on virtual lines (feels more intuitive as you say), keep the current display of relative lines, and make it so that a relative jump moves across (however I see how it can break the current implementation logic given the way repeated motions are implemented...)

* we keep the default vertical motions on virtual lines and find a way to display relative virtual line numbers in the gutter

* we restore the defaults to vertical motions on document lines and try to find a new, intuitive default for virtual line motions.

I personally prefer the first point, mainly because this was NeoVim's behavior once you had remapped vertical motions to use virtual lines, but I'm curious about your thoughts on this.

Thanks !

I don't think we should handle vertical movement differently when a count is specified. That seems inconsistent. I will be adding a command for vertical movement that ignores softwrap later. Users that behaviour could change the j/k bindings or bind to J/K

[CBenoit]

So far so good, it’s quite pleasant to work with multiple panes using soft-wrapped lines!

One small bug I found when looking up references to the authenticate function:

The highlighted line is a bit shifted out.

[kirawi]

In the above image, I feel like it would be more readable if whole words were split, rather than being split mid-word. It's a bit distracting.

[archseer]

That's going to complicate things since you'd need to backtrack and determine where a reasonable break would be. It wouldn't work in cases where you're looking at minified JSON either.

[archseer]

From what I can tell the github code rendering doesn't support this either but I didn't check other editors

[archseer]

Is the indentation on wrapped lines deliberate though? I think most editors will wrap to the beginning of the line

[kirawi]

VSCode implements this, and I think it uses a config option to determine maximum word length before wrapping (80 columns). Led also implemented it in that way (though of course it wasn't a major editor). VSCode's wrapping implementation is in https://github.com/microsoft/vscode/blob/82db01c21a0d0d08b156b725a37c39cc3714c238/src/vs/editor/common/viewModel/viewModelLines.ts#L60

I checked and it seems GitHub does do it:

[gabydd]

Is the indentation on wrapped lines deliberate though? I think most editors will wrap to the beginning of the line

It is deliberate and there is also a config option for it wrap-indent which defaults to 2

[archseer]

Led also implemented it in that way

I'm not opposed to it, but it could also be done as a follow-up, depends on what @pascalkuthe thinks

[David-Else]

I would very much like the option to split on words rather than in the middle of them, it is set using 'linebreak' in NeoVim.

The following is taken from the NeoVim documentation:

'wrap' boolean (default on)
This option changes how text is displayed. It doesn't change the text
in the buffer, see 'textwidth' for that.
When on, lines longer than the width of the window will wrap and
displaying continues on the next line. When off lines will not wrap
and only part of long lines will be displayed. When the cursor is
moved to a part that is not shown, the screen will scroll
horizontally.
The line will be broken in the middle of a word if necessary. See
'linebreak' to get the break at a word boundary.

'linebreak' 'lbr' boolean (default off)
If on, Vim will wrap long lines at a character in 'breakat' rather
than at the last character that fits on the screen. Unlike
'wrapmargin' and 'textwidth', this does not insert s in the file,
it only affects the way the file is displayed, not its contents.
If 'breakindent' is set, line is visually indented. Then, the value
of 'showbreak' is used to put in front of wrapped lines. This option
is not used when the 'wrap' option is off.
Note that characters after an are mostly not displayed
with the right amount of white space.

'breakat' 'brk' string (default " ^I!@*-+;:,./?")
This option lets you choose which characters might cause a line
break if 'linebreak' is on. Only works for ASCII characters.