LSP: using changedtick as the document version breaks workspace/executeCommand usage

[icholy]

Problem

#28943 changed the lsp code to use vim.b[bufnr].changedtick as the LSP document version.

Now there’s no way to send the correct document version in a workspace/executeCommand after saving a modified buffer. This is because changedtick is incremented when "Resetting modified when writing the buffer" https://neovim.io/doc/user/eval.html#b:changedtick but the server is not notified of this new version.

Steps to reproduce

1. User edits buffer
   • Client sends textDocument/didChange (version 1)
2. User saves buffer
   • changedtick is incremented to version 2
   • Client sends textDocument/didSave (no version sent)
3. User runs custom lsp command (ex: :EslintFixAll)
   • Client sends workspace/executeCommand (version 2)
   • LSP ignores command because it hasn’t seen version 2 yet.

Expected behavior

Some way to get the last sent buffer version for workspace/executeCommand usage.

Neovim version (nvim -v)

NVIM v0.11.0-dev-158+g05435a915

Vim (not Nvim) behaves the same?

N/A

Operating system/version

Ubuntu 22.04.4 LTS

Terminal name/version

alacritty 0.12.2

$TERM environment variable

alacritty

Installation

bob

[icholy]

cc @mfussenegger

[icholy]

Repro:

init.lua

vim.api.nvim_create_autocmd("FileType", {
    pattern = "go",
    callback = function()
        vim.lsp.start({ cmd = { "gopls" } })
    end
})

main.go

// delete me
package main

func main() {
	fmt.Println()
}

Steps:

1. nvim --clean -u init.lua main.go
2. Delete the first line dd
3. Save the buffer :w
4. Run code action :lua vim.lsp.buf.code_action({ apply = true })

[mfussenegger]

Thanks for the repro.
I'm not entirely sure if #29170 is a sound approach, but it would fix the issue as far as I can tell.

[icholy]

The change seems reasonable to me (I tested locally and can't break it). But I suspect we're going to have similar issues with the semantic tokens & inlay hints. If that does end up being the case and more edge-case handling is required, I think it might make sense to revert #28943

[mfussenegger]

But I suspect we're going to have similar issues with the semantic tokens & inlay hints.

I was wondering about that too, but can't think of a scenario where it could be a problem. If the buffer is unchanged (modified=false), there shouldn't be any updates for semantic tokens, inlay hints, code lens and so on anyway.

[icholy]

I'm having a difficult time figuring out how a plugin should call workspace/executeCommand with the right version. I'm using the :EslintFixAll command as the example here.

The original (current) implementation has the problem where the version is being rejected on the server side because it's too new.

local bufnr = util.validate_bufnr(opts.bufnr or 0)
request(0, 'workspace/executeCommand', {
	command = 'eslint.applyAllFixes',
	arguments = {
		{
			uri = vim.uri_from_bufnr(bufnr),
			version = lsp.util.buf_versions[bufnr],
		},
	},
})

I tried rewriting it to use vim.lsp.buf.code_action. This doesn't work because code_action automatically populates the request context with diagnostics from the current line. (This results in the eslint.applyAllFixes only fixing the issues on the current line).

lsp.buf.code_action({
	apply = true,
	filter = function(a)
		return vim.tbl_get(a, 'command', 'command') == 'eslint.applyAllFixes'
	end,
})

I would need to use the unexported diagnostic_vim_to_lsp from vim.lsp.diagnostic to send all the diagnostics in the current buffer to achieve the original behaviour.

lsp.buf.code_action({
	apply = true,
	context = {
		diagnostics = diagnostic_vim_to_lsp(vim.diagnostic.get(0)),
	},
	filter = function(a)
		return vim.tbl_get(a, 'command', 'command') == 'eslint.applyAllFixes'
	end,
})

This feels like a lot of hoops to jump through to do something relatively simple.

[mfussenegger]

I'm not sure how to best deal with that but I'm reluctant to revert the change. lsp.util is not the right place for the version.

One option that would work is to apply the same modified check as in the linked fix (version = vim.b[bufnr].changedtick + (not vim.bo[bufnr].modified and 1 or 0))

Another option could be to simplify the code_action API by:

• Take vim.diagnostics instead of lsp diagnostics. Given that a user has no means to provide the lsp diagnostics without accessing internals the parameter doesn't make sense anymore
• Support a string as filter in addition to the function, that matches on the command.

In general it's a bit unusual that the server even requires a version parameter with the request. Can't it be set to nil with in other places means the client/server should ignore the version?

[icholy]

I'm not sure how to best deal with that but I'm reluctant to revert the change. lsp.util is not the right place for the version.

It could live in changetracking and only expose a changetracking.buf_version(bufnr) function.

• The internal state gets updated by changetracking.send_changes (using vim.b[bufnr].changedtick).
• Returns 0 if there's no existing value.
• No need to ever reset back to 0, just keep incrementing it.

edit it can't live in changetracking due to circular dependencies.

One option that would work is to apply the same modified check as in the linked fix (version = vim.b[bufnr].changedtick + (not vim.bo[bufnr].modified and 1 or 0))

This breaks if you open a file with existing issues.

In general it's a bit unusual that the server even requires a version parameter with the request.

Yes, this is true. I've tried to find other examples to strengthen my case and have came up empty handed.

Can't it be set to nil with in other places means the client/server should ignore the version?

This was the first thing I tried. No luck.

[icholy]

I've been thinking about this a little more and I'm starting to question the use of changedtick all-together. The behaviour with modified creates an impedance mismatch between changedtick and lsp document versions. I propose a lsp/_versions.lua file with the following:

local M = {}

---@type table<integer, integer>
local buf_versions = {}

---@param bufnr integer
---@return integer
function M.get(bufnr)
	-- we default to 1 instead of 0 because some LSPs return 0
	-- when they don't support versioning. Starting at 1 disambiguates
	-- that case.
	return buf_versions[bufnr] or 1
end

---@param bufnr integer
function M.changed(bufnr)
	buf_versions[bufnr] = M.get(bufnr) + 1
end

return M

[jdrouhard]

I've noticed recently as well (even after the above follow up fix for the modify case) that semantic tokens aren't updating as reliably as they were. When I first implemented that feature, I remember looking into changedtick vs the way we were tracking versions and came to the conclusion at the time that we couldn't use changedtick and what we were doing was necessary, but I don't remember exactly how I came to that conclusion.

I'll try to dig into that again at some point soon to see if I can remember why changedtick wasn't sufficient.

[icholy]

I've also noticed some strangeness with semantic tokens, but I haven't been able to come up with a repro.

[jdrouhard]

Ok I haven't 100% confirmed this, but I think I know why I'm occasionally seeing some weird stuff with semantic tokens.

So, the crux of the issue is that b:changedtick can change without an on_lines() callback being fired. changedtick is incremented for all sorts of things that don't fire that callback (but on_changedtick() is fired for all of them). But because we only send change notifications (and re-send new semantic token requests) from on_lines, we can easily get out of sync on the version we think a buffer is on vs whatever was last sent to the server.

Specifically for semantic tokens this means that when a request comes back and the changedtick number changed without firing on_lines, we drop the response without ever sending a new request.

Because we actually only care about changes to a buffer that occur with an on_lines callback, that is why we needed a separate version--one that only increments when we actually get that callback (so we send change notifications, request new tokens, etc).

[mfussenegger]

Reverted it for now #29217