LSP: vim.lsp.buf.formatting_sync() messes up marks

[MagicDuck]

• nvim --version:

NVIM v0.5.0-dev+1157-g0ab88c2ea
Build type: Release
LuaJIT 2.1.0-beta3
Compilation: /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -O2 -DNDEBUG -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=always -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LEVEL=3 -I/Users/runner/work/neovim/neovim/build/config -I/Users/runner/work/neovim/neovim/src -I/Users/runner/work/neovim/neovim/.deps/usr/include -I/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include -I/Library/Frameworks/Mono.framework/Headers -I/Users/runner/work/neovim/neovim/build/src/nvim/auto -I/Users/runner/work/neovim/neovim/build/include
Compiled by runner@Mac-1616045056542.local

Features: +acl +iconv +tui

• language server name/version: N/A
• Operating system/version: macos

Steps to reproduce using nvim -u minimal_init.lua

1. Set up some language server that does formatting. For instance diagnosticls:
   (You will need prettier installed in the example below, but I don't think the particular formatter matters)

lspconfig.diagnosticls.setup {
  root_dir = function(filepath)
    return vim.fn.expand("~")
  end,
  filetypes = {
    "javascript",
  },
  init_options = {
    formatFiletypes = {
      javascript = {"prettier"},
    },
    formatters = {
      prettier = {
        command = "prettier",
        args = {"--stdin", "--stdin-filepath", "%filepath"},
        isStdout = true,
        rootPatterns = {".prettierrc", ".prettierrc.json"}
      }
    }
  },
  on_init = function(client)
    client.resolved_capabilities.document_formatting = true
  end,
  on_attach = function()
      vim.api.nvim_command [[autocmd BufWritePre <buffer> lua vim.lsp.buf.formatting_sync({}, 5000) ]]
  end,
}

2. Open a js file with a bunch of code in it
3. add mark A on line 3 for example
4. move to line 7, add a bunch of whitespace lines, then save to trigger formatter

Actual behaviour

Mark is removed (or moved to the top of the file)

Observation:
If :wshada! is executed before saving and :rshada! after, the marks are restored to proper position

Expected behaviour

Marks stay in place.

A hacky workaround:

Wasn't sure how expensive :wshada/rshada are so wrote the following instead to use on BufWritePre

LspFormatBuffer = function(bufnr) 
  local marks = {}
  local letters = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
  for i = 1, #letters do
    local letter = letters:sub(i, i)
    local markLocation = vim.api.nvim_buf_get_mark(bufnr, letter)
    marks[letter] = markLocation;
  end

  vim.lsp.buf.formatting_sync({}, 5000) 

  for i = 1, #letters do
    local letter = letters:sub(i, i)
    local markLocation = marks[letter];
    local linenr = markLocation[1]
    if (linenr ~= 0) then
      vim.api.nvim_command("" .. linenr .. "mark " .. letter)
    end
  end
end

[mfussenegger]

I think the root cause of this is that formatting_sync applies the changes using nvim_buf_set_lines, and nvim_buf_set_lines doesn't preserve extmarks when only part of a line is modified.

There's a PR pending (#13703) which would change it to use nvim_buf_set_text which preserves on the extmarks on non-modified parts of changed lines.

[chentoast]

I think the author might be talking about regular vim marks instead of extmarks; nvim_buf_set_text and nvim_buf_set_lines will both invalidate any marks that lie within the changed range.

[MagicDuck]

Yes, that's correct, talking about regular marks! Wasn't sure what extmarks were 😄
Unfortunately, with most formatters they operate on the whole file instead of a range, so all marks are lost. I think it's ok to at least keep the same number as before as typically re-formats don't produce world-shaking movements of code.

[folke]

I noticed that as well, but never thought I was due to LSP :) Will see if I can make a PR for this, since it's pretty annoying

[JoaquimEsteves]

Any chance the maintainers could revisit this?

I noticed that @folke was kind enough to provide a pr that has been closed.

In the comments to the PR a contributer mentioned that this is due to the fact that after formatting some marks point to garbage; and as such the user should "assume" that after formatting a file all marks will be removed.

As a user I don't agree, and would appreciate some config option that would incorporate this pr.

Something like:

nvim_lsp.my_lang_server.setup({ keep_marks_even_if_garbage = True })

In practise, formatting the text would in all likelihood keep most of my marks exactly the same.

Let's take this example:

a| 0|         some_obj = {
 | 1|             "integer": {
b| 2|                 "numpy": np.int32,
 | 3|                 "pandas_str": "int32",
c| 4|                 "sql": sql.types.Integer,      "python": int,
 | 5|             },

On the first column we have the marks, second we have the line number.

Assume that mark C points to the 'p' character in "python" on line 4.

After running my formatting program I'd end up with:

a| 0|         some_obj = {
 | 1|             "integer": {
b| 2|                 "numpy": np.int32,
 | 3|                 "pandas_str": "int32",
c| 4|                 "sql": sql.types.Integer,
 | 5|                 "python": int,
 | 6|             },

Marks 'a' and 'b' remain exactly the same, and using 'c I'd still end up on the correct line - typing `c would point me to garbage, but ideally that would just result in an error message.

Edit: Another point in favour of adopting the PR is that marks aren't deleted when calling an external formatter on a buffer. If I invoke !formatter % then vim will do it's best to make it so the marks remain in place.

[justinmk]

I noticed that @folke was kind enough to provide a pr that has been closed.

In favor of reopening that, if @folke feels motivated.