Letter reordering creates false positives for diagnostics

[jc00ke]

I don't know if this is an Expert issue or a neovim (nightly) issue. I never saw this with ElixirLS or Nextls, and have not seen it with other LSPs.

vim.lsp: Active Clients ~
- expert (id: 1)
  - Version: 0.0.1
  - Root directory: ~/projects/website
  - Command: { "expert" }
  - Settings: {}
  - Attached buffers: 3, 5, 7

with neovim v0.12.0-dev-1078+g088ba835ed.

Please let me know if there's any other helpful information I can add, or if it's an upstream issue.

[mhanberg]

Can you elaborate on the issue a bit more? I see the typo in the module reported by the diagnostic but I'm not entirely sure what the steps you took to achieve that screenshot.

[jc00ke]

cd
mix new foo
cd foo
nvim lib/foo.ex
# def bar(name) do
# return
# tab (I think)
# String.upcase(name
# close paren
# diagnostic error appears with misspelled variable name

Just to make sure...

[throttle2k]

Hi, I'm experiencing the same issue on stable Neovim (0.11.4-1).

The problem occurs when typing quickly: although the words are entered correctly, the diagnostics sometimes display characters in the wrong order and mark them as errors.

If I type more slowly, the issue does not occur.
This problem happens only with expert, and does not occur with elixirls.

[andyleclair]

Hey all, I'm also having this, neovim 0.11.3. Seems to happen when I type quickly. Is there a debounce timeout? Increasing that slightly might just make it ok

[vstegen]

I've tried to set update_in_insert to false for the nvim diagnostics and it does not help, so it seems that it's something internal to expert

[badosu]

Just a note that this issue existed in lexical for a while, so maybe the fix might be similar to the one that was applied to lexical at the time?

[heywhy]

Just a note that this issue existed in lexical for a while, so maybe the fix might be similar to the one that was applied to lexical at the time?

Can you point to the fix applied to Lexical?

[heywhy]

@mhanberg @doorgan So, I made a change to the cloned expert repo. I added a couple of log statements that show the content changes received from the client (Neovim) and the state of the document after the changes have been applied. Here is the log file expert.log. Below is a recording showing how to reproduce the issue.

If you look at the log file, you will find out that Neovim seems to use the same range for all content changes when you select a completion item, but Expert didn't handle the situation correctly. One of the ways I tried to fix it was by batching consecutive changes that share the same range into a single change map. But I worry the solution is not sophisticated, so I'm still testing with various interaction patterns to make sure no edge case is missed.

It's tricky to tell if the problem is from Neovim or Expert.

Excerpt:

# Current state of document held in memory
defmodule Proxy do
  String.a
end

# Notification received when navigating in the completion popup
%XPGenLSP.Structures.DidChangeTextDocumentParams{
  content_changes: [
    %{text: "", range: XpRange[<<2, 10>>...<<2, 11>>], range_length: 1},
    %{text: "b", range: XpRange[<<2, 10>>...<<2, 10>>], range_length: 0},
    %{text: "a", range: XpRange[<<2, 11>>...<<2, 11>>], range_length: 0},
    %{text: "g", range: XpRange[<<2, 11>>...<<2, 11>>], range_length: 0},
    %{text: "_", range: XpRange[<<2, 11>>...<<2, 11>>], range_length: 0},
    %{text: "d", range: XpRange[<<2, 11>>...<<2, 11>>], range_length: 0},
    %{text: "i", range: XpRange[<<2, 11>>...<<2, 11>>], range_length: 0},
    %{text: "s", range: XpRange[<<2, 11>>...<<2, 11>>], range_length: 0},
    %{text: "t", range: XpRange[<<2, 11>>...<<2, 11>>], range_length: 0},
    %{text: "a", range: XpRange[<<2, 11>>...<<2, 11>>], range_length: 0},
    %{text: "n", range: XpRange[<<2, 11>>...<<2, 11>>], range_length: 0},
    %{text: "c", range: XpRange[<<2, 11>>...<<2, 11>>], range_length: 0},
    %{text: "e", range: XpRange[<<2, 11>>...<<2, 11>>], range_length: 0}
  ],
  text_document: %XPGenLSP.Structures.VersionedTextDocumentIdentifier{uri: "file:///Users/home/Dev/me/proxy/lib/proxy.ex", version: 27}
}

# After Expert Applied the above content changes, the document state in memory is now the below.
defmodule Proxy do
  String.becnatsid_ga
end

[mrluc]

Re: the recent report of "typing very fast" #119 causing letter order issues -- I can report that I definitely get it more consistently with things I've pasted, which could be related since that's like typing very fast.

[heywhy]

So, I was able to trace the whole flow, and I got to find out that Neovim has this concept of incremental updates, and that's the reason for different edits having the same range values, which Expert does not handle correctly. I've been able to identify a few new patterns, and I already have a fix that handles a lot of them, but I'm currently unwell, so I can't tackle it right now (hopefully, I should be better in the next couple of days).

To mitigate temporarily, you should turn off incremental updates for Expert using the below config:

vim.lsp.config('expert', {
  flags = {
    allow_incremental_sync = false,
    -- When I set the below option to nil and have incremental sync on, I got substantial improvements with the fixes
    -- I had in place until a weird edge case popped up
    -- debounce_text_changes = nil
  }
})

The downside of turning off incremental updates is that the whole file content is sent for every keystroke.

And I also tried looking in the Zed repo (per this #83 (comment)) to see if they share the same concept, but I can't find anything substantial yet.

[badosu]

To mitigate temporarily, you should turn off incremental updates for Expert using the below config:

vim.lsp.config('expert', {
flags = {
allow_incremental_sync = false,
-- When I set the below option to nil and have incremental sync on, I got substantial improvements with the fixes
-- I had in place until a weird edge case popped up
-- debounce_text_changes = nil
}
})

Thank you so much! I can confirm this workaround works, I was almost giving up on expert due to having to reload the document everytime it failed ingesting text changes correctly.

[heywhy]

Guess what?! I think I've resolved the following issues:

1. Letter reordering problem
2. Formatting wrangling the inner representation of the document
3. I don't need to turn off incremental sync in Neovim anymore

I would love it if anyone could build this #123 and test (with incremental sync turned back on, in case you turned yours off). I would love it if people using other editors also give it a try; hopefully, it solves the problem across the board.