[RFC] Built-in LSP Support by tjdevries · Pull Request #6856 · neovim/neovim

tjdevries · 2017-06-07T03:27:29Z

So, here's the very beginnings of LSP support in neovim.

It can currently, start a server, say that it has opened the file and request references from the server. It loads the references using setloclist.

Here's my vision (or at least a rough draft of it). It's too late for me right now to clean more of it up and I'm too excited not to finally at least put something as a WIP PR :)

Feedback welcome and appreciated.

Goals:

Registering callbacks

Should be something along the lines of

function lsp#request(request_name, arguments, use_default_callback, optional_callback)

as well as a lua function that has the same signature.

Where:

request_name
- The name of the request that you want, i.e. 'textDocument/references'
- This is specified in microsofts protocol.
arguments
- If you want to hardcode in a value, pass in the values that you would like, the rest will be filled in with the default values
- These default values would be grabbed programmatically, i.e. for Position, we would use line('.') and col('.')
- This allows the functions to be tested or used outside of the LSP context
  - For example if you want to highlight all the references of something, but it's not the item under the cursor, you could pass in the location of that item to the function
use_default_callback
- A boolean value to determine whether or not to use the default callback value
optional_callback
- We will provide a default callback that will handle anything within the TUI
- This will also use built-in neovim items (such as popupmnu, loclist, quickfix, etc.) so that if those are externalized by a GUI, they don't even have to do anything special.
- Behavior could be to simply override the callback or to add it to a list of callbacks to call
  - Might want to use a list so that multiple hosts/clients of neovim could grab onto these events.
  - Probably don't want neovim to process the events though if the host/client override.
  - I'd just start with overriding to start with, since it is simple.

I imagine optional callbacks essentially allowing an even more extensive API, one that doesn't require Neovim explicit knowledge. So, to embed neovim in another editor, you could try and do lots of the information transfer with LSP callbacks.

From nvim
:call lsp#request('textDocument/references', [], v:false, custom_callback)

From remote
nvim.call_function('lsp#request', ['textDocument/references', [], false, custom_callback])

Creating default callbacks

function lsp#handle_response(request_name, response)
-> function lsp#response#textDocument.references(response)
-> function lsp#response#textDocument.hover(response)

Where:

response:
- The response as specified by the protocol.
- These should be made available so that people can use these as well to do more complicated tasks easily.
- For example, could use the hover callback to create a simple hover pane that other hosts/clients/GUIs would understand.

bfredl · 2017-06-07T04:41:08Z

Nice! I hope to improve jobs/channels in lua #6844 (comment) to the extent this can just use them and don't need to use libuv directly.

tjdevries · 2017-06-07T12:29:27Z

@bfredl that sounds great! That would certainly ease some of the configuration for the client. I will try and watch out for that.

ZyX-I · 2017-06-07T17:48:31Z

runtime/autoload/lsp/request.vim

+function! s:get_client() abort
+   if g:nvim_lsp_client == -1
+      lua << EOF
+local client = require('runtime.lua.lsp.client')


This dedented block looks ugly, do not use <<EOF. Everything which does not fit into one line (lua require('lsp.client').get_client(), or, better, just return luaeval("require('lsp.client').get_client()") (move :if to lua to not split the logic)) should be in a required function.

Also should be require('lsp.client'), I do not know why #6789 still is not merged.

Okay, I have removed many of them and rebased, so I can use the updates from #6789 . I will clean up the code as I go along. Still brainstorming a lot of ideas.

ZyX-I · 2017-06-07T17:48:41Z

runtime/autoload/lsp/request.vim

+
+function! lsp#request#textdocument_references()
+   lua << EOF
+print('TODO')


ZyX-I · 2017-06-07T17:50:51Z

runtime/lua/json.lua

@@ -0,0 +1,383 @@
+--
+-- json.lua


Can you just use vim.api.nvim_call_function('json_…'? There already is a JSON parser in project, do not need to add a second one. If some features are missing they can be added.

I didn't want to use the API to do it, since some of the json parsing might be done in a different thread I think. Unless it can work in another thread?

You are using luv to fork out different Neovim threads? I do not think this is a good idea, most of the code does not expect this and making sure that different shared resources (memory, file descriptors except for a small subset, etc) are not accessed through spawned threads is close to impossible. Generally: until there are Neovim own functions for creating threads or fork()ing don’t do this in core plugins, it risks producing nasty heizenbugs.

As to the question, though not exactly API, but JSON decoder could be easily edited to work with lua natively: most of code is parsing there, just need to abstract away creation of values (I would personally go with creating jsondecode.c.h macros-parametrized file for this purpose, should also be possible to do abstraction via struct with function references). Encoder is a separate issue, the “easy” way is to not touch it at all, but create duplicate lua -> VimL conversion functions which are exactly like existing ones, but do not put allocated lists and dictionaries into M&S GC double-linked list. In both cases you will spend more time creating the tests then adjusting the code.

Luv won't spawn threads, but it can run callbacks at times which are not considered vimL-safe, though as long as GC won't be invoked in such a time (potentially, rooting might be incomplete), json_decode shouldn't be a problem.

@bfredl Container allocation functions are not reentrant because they alter a linked list in a number of steps. So it is not safe to call callbacks at random times even without GC run (how, BTW? I do not know how uv is going to perform async read at unsafe time without either using SIGALARM (should probably mess up with our timers), or a separate thread, or a separate process), unless you edit C code of the parser to work with lua containers.

Why not use existing jobs in any case, this is going to be safe for sure? Just have a lambda which immediately calls lua.

Not random times, just times the event loop is invoked. As previously this could be done in a way that queues all vimL until later, such spots could potentially be vimL GC unsafe. Can container allocation invoke GC or does that only happen at states?

The plan is indeed to improve jobs so they can be used here.

ZyX-I · 2017-06-07T17:55:08Z

runtime/plugin/lsp.vim

+
+" TODO: Automatically open file if not opened already
+function! LSP_References() abort
+lua << EOF


lua lsp.client_references(), indented:

Do not use ugly <<EOF blocks.

All globals must be namespaced. This is not the only plugin using lua. I personally would prefer a single global per plugin or no globals at all (use lua require("…").func()).

Just like on #6789, I would say that it is vastly preferable if we put some of these finer points in the help docs, where people are likely to find them.

ZyX-I · 2017-06-07T17:55:15Z

runtime/plugin/lsp.vim

+endfunction
+
+function! LSP_DidOpen() abort
+lua << EOF


ZyX-I · 2017-06-07T17:55:37Z

runtime/plugin/lsp.vim

+" TODO(tjdevries): Make sure this works correctly
+" TODO(tjdevries): Figure out how to call a passed callback
+function! LSP_Request(request, params, callback) abort
+  return luaeval('client_request(_A.request, _A.args)', {request: request, args: params})


Same (globals).

ZyX-I · 2017-06-07T17:57:06Z

runtime/plugin/lsp.vim

@@ -0,0 +1,28 @@
+" TODO(tjdevries): Move this to autoload?
+
+execute('luafile ' . expand('<sfile>:h') . '/../lua/lsp/plugin.lua')


Should be something like lua lsp = require('lsp.plugin').setup(). (Needs #6789 or you adding a path to package.path.)

ZyX-I · 2017-06-07T17:59:10Z

test/functional/lsp/callbacks/textDocument.lua

@@ -0,0 +1,38 @@
+local helpers = require('test.functional.helpers')(after_each)


I guess this should be test/functional/plugin/lsp/…: other runtime files are tested in plugin unless these are runtime files used from core in a special fashion (like providers which have corresponding C code). Applies to all test files here.

ZyX-I · 2017-06-07T18:02:01Z

runtime/lua/util.lua

+vim = vim or {}
+
+util.echom = function(message)
+  vim.api.nvim_command('echom "' .. tostring(message) .. '"')


Just remove this function and use print(). And you have escaping issues here in any case.

ZyX-I · 2017-06-07T18:03:20Z

runtime/lua/lsp/client.lua

+
+function client:_on_error(level, err_message)
+  if type(level) ~= 'number' then
+    util.echom('we seem to have a not number' .. util.tostring(level))


Message should start with a capital letter.

wsdjeg · 2017-06-08T13:09:31Z

runtime/plugin/lsp.vim

+
+execute('luafile ' . expand('<sfile>:h') . '/../lua/lsp/plugin.lua')
+
+function! LSP_Start() abort


@tjdevries hi, I think maybe some people like autoload function more. for example

function! lsp#start() abort

and I have read most of vim's runtime file, the file in runtime/plugin/ do not definde func with capital letter.

I have changed them to autoload. Thanks :)

aktau · 2017-06-20T12:23:34Z

runtime/lua/lsp/client.lua

@@ -0,0 +1,304 @@
+local uv = require('luv')


Is luv already included by default?

For me, inside of nvim:

:lua print(package.searchpath('luv', package.cpath)) /home/aktau/github/neovim/neovim/.deps/usr/lib/lua/5.1/luv.so

:lua print(package.cpath) ./?.so;/usr/local/lib/lua/5.1/?.so;/home/aktau/github/neovim/neovim/.deps/usr/lib/lua/5.1/?.so;/usr/local/lib/ lua/5.1/loadall.so

Or:

$ nvim -u NONE +'lua io.stderr:write(package.cpath)' +'lua os.exit()' ./?.so;/usr/local/lib/lua/5.1/?.so;/home/aktau/github/neovim/neovim/.deps/usr/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so%

It appears my build folder is in my package.cpath, this is the default package path from the LuaJIT compiled into nvim, but it can be overridden if the user sets their own path. This should be made much more robust. It is also not guaranteed that a user has the neovim .deps folder on their machine. We should attempt to not rely on optional libraries (especially those that need to be compiled, like luv).

I'm hoping to not have to use luv soon. Once we get the jobs update (which I assume will happen before I finish this :) ) then I will use nvim's included job functionality, rather than a hacked together interpretation.

tjdevries · 2017-06-27T22:50:36Z

@ZyX-I Is there a preferred way to get rid of the luacheck errors about referencing "global vim" without declaring it?

Should I just set local vim = vim or {}? Or change my luacheck config?

justinmk · 2017-06-28T08:44:01Z

Instead of callbacks why not publish RPC notifications?

bfredl · 2017-06-28T09:03:14Z

Then we need to be able to subscribe to them from vimL and Lua. Which we probably want anyway.

ZyX-I · 2017-06-28T09:09:27Z

@tjdevries Tell luacheck that there is global vim. CMakeLists.txt already does just that for vim.lua.

tjdevries · 2017-06-28T13:14:02Z

@justinmk I think RPC notifications would work as well. Originally I was thinking of callbacks because I thought GUIs/plugins/etc. might want to add their own callback or remove the default callback to extend behavior. But I think RPC would work well for that too.

I never got around to implementing an async rpcrequest though. So whenever I send an LSP request, it would block, correct? Or are you saying after a request message is sent back, do rpcnotify(0, 'lsp/textDocument/references', ...)?

justinmk · 2017-06-29T08:18:15Z

If the callbacks aren't needed then my suggestion is moot. My suggestion is only relevant under that assumption.

euclio · 2017-08-16T18:10:53Z

What's the status of this?

tjdevries · 2017-08-16T19:05:29Z

I'm waiting on a few PRs to get merged to ease development of this.

I'm also working on a few different projects right now, so I haven't had a lot of time to work on this. I would recommend using one of the other plugins that provides the functionality for now.

I'm still planning on finishing this though :)

balta2ar · 2017-08-16T19:15:23Z

I'm waiting on a few PRs to get merged to ease development of this

For those who are curious, could you name them, please?

tjdevries · 2017-08-16T20:45:33Z

There's some work being planned for job_control to be added into Lua, being able to cancel stuck loops in lua, and perhaps more lua integration with the vim.api module.

Primarily the job_control items, since that could effect the structure of the support within neovim.

jvican · 2017-09-28T11:37:04Z

Could we set up a BountySource for this to motivate @tjdevries get this through the finish line? I would be happy to donate some money to have native LSP support in neovim.

blueyed · 2017-09-28T12:09:50Z

How does it compare to https://github.com/autozimu/LanguageClient-neovim?
(listed on http://langserver.org/#implementations-client also)

tjdevries · 2017-09-29T15:51:33Z

Hi everyone 😄

Sorry for the long pause in dev, been working on some personal projects and practicing lua and lua within nvim.

@blueyed It currently compares to the language client as not as many features and its broken 😄. The idea however is that if this ships with neovim, people will be able to contribute more effectively, since it will be an "official" implementation. Also, it won't require any dependencies (so you won't have to install python and set up neovim-python) as the Lua will all run with whatever neovim ships with. I have lots of other ideas, but I should really finish the basic implementation first :)

I imagine there will still be other plugins around that integrate the core features built here to interact with other plugins. I can imagine sources for deoplete that use the builtin lsp#request(...) to get completion options. Maybe we will also implement some async api calls to interface with lsp as well. As I said, lots of ideas.

@jvican Not sure how that would work. If you're interested in using LSP in the mean-time, I would really recommend the implementation @blueyed linked. That way you could use most of the up-side of LSP and not have to spend money :)

KillTheMule · 2017-12-18T14:00:43Z

I'm currently using https://github.com/autozimu/LanguageClient-neovim with the RLS. If you're at a point where you'd want someone to test this, let me know :)

tjdevries · 2017-12-19T01:54:30Z

@KillTheMule I'll let you know :D Thanks for the offer.

andreypopp · 2019-01-23T06:18:29Z

Maybe neovim could only expose Lua API for LSP? Seeing how much effort is being put in user space to support LSP it would be unfair to discard it (and especially given that such effort has value both for vim and neovim users). That way neovim would provide backing API on which underlying user space implementations could rely when run on neovim.

justinmk · 2019-01-23T09:00:07Z

vim-lsp also uses autoload/lsp.vim

Maybe neovim could only expose Lua API for LSP?

Somewhere in the comment cloud above, we discussed minimizing and, ideally, eliminating the VimL wrapper for this, so lsp.vim definitely isn't a hard requirement. It can be renamed for now, and revisited later.

mcepl · 2019-01-23T10:38:53Z

Hi. The branch uses autoload/lsp.vim as for Vim script API.
But it conflicts with vim-lsp.

I understand, you were first, but if this gets built-in to Neovim, then I am afraid it should get a precedence, shouldn't it?

bfredl · 2019-01-23T10:43:35Z

@mcepl but as said above this will be trivially solved by renaming the file.

Alternatively, as I might already have said in the hidden 170 comments, we should change the init.vim definitions to be in the form let g:lsp_some_good_var_name = { server definitons }, so no autoload calls will be needed from vimL.

mcepl · 2019-01-23T12:43:29Z

@bfredl certainly, and it is not a big deal. Actually, eliminating autoload seems to me like a good idea.

mcepl · 2019-02-15T18:30:52Z

It is extremely easy to add support for renaming of symbols (thank you!):

function! LSPRename()
    let s:newName = input('Enter new name: ', expand('<cword>'))
    call lsp#request_async('textDocument/rename',
        \ {'newName': s:newName})
endfunction

au FileType lua,sh,c,python
    \ noremap <silent> <buffer> <F2> :call LSPRename()<CR>

but there is a bug somewhere. Whenever I run this command, I get one more EOL in the end of the file.

KillTheMule · 2019-03-02T19:43:49Z

@tjdevries I've made nvim into a fake languageserver and used it for a test here. I'm planing to extend this a bit, seems to be a good way to test this.

Generally, though, what's you're plan, do you think you'll find time to work on this? I offer my help, up to the point of bringing it over the finish line.

tjdevries · 2019-03-07T01:40:23Z

@KillTheMule I have my last day of work at my current company on Friday and then have a week off after that. If I don't make any progress during that week, it'd be great if you could take this PR to the finish line for me. I'd be happy to hop on gitter more often if you need to bounce ideas around or try and figure out what I was doing during parts of the PR.

Thanks for following up and offering your help.

KillTheMule · 2019-03-07T12:47:51Z

That's good to hear. Let me throw out some questions that occured to me during writing the integration test. Feel free to discard any of my thoughts, but I think I should at least mention them :) Talking on gitter would be fine, too :)

Right now, you're sending Notifications as Requests. The spec has them different, in that at Notification does not have an id field, and needs no response. Are you planning to make a difference here? That would also make my life easier for the integration test :D From reports, things seem to be working, so servers don't seem to mind, but I guess going with the spec would be a good thing.
Sync requests don't return the response, but use callbacks. I found that pretty unintuitive, and I think it brings quite a bit of procedure to clients using this code. Wouldn't it make sense to simply return the response? Async requests would of course still need a callback...
Sync requests are async requests with a (busy?) waiting loop. You're waiting 100ms in each step... isn't that much too much?
From my pov, this PR provides two things: An lsp-lib that can be used for clients, and a simple client. Would it make sense to separate those more? I found it a bit surprising that potential other clients have to unregister the default handlers, for example. Also, I see a lot of code that already handles things in a specific way, which befits a client, but not a lib. Not sure how important that is, but I'd separate those things :)
I did not yet dive into that, but generally I found that user autocommands are inadequate because they can't carry arguments, i.e. you can't distinguish from which buffer they originated. Might that pose a problem when someone has several clients running? Or do you have a way to transport that information?
Not a question, but please consider doing this.

mcepl · 2019-06-09T19:14:11Z

@tjdevries @KillTheMule ping? This LSP client is accused of being a cause of #10167.

KillTheMule · 2019-06-10T14:53:31Z

Hey, my latest info was that tj was working on a larger refactoring, but I haven't heard from him since. I'm pressed for time these days, so I did not push this.

h-michael · 2019-06-11T01:38:32Z

@KillTheMule @tjdevries Can I take over this PR?

bfredl · 2019-06-11T09:24:32Z

@h-michael Sure, open a new PR with your changes.

KillTheMule · 2019-06-11T09:30:01Z

@h-michael Sure, open a new PR with your changes.

Definitely. I'll keep track and try to help out, let me know if there's something particular you'd like me to do. In particular, I could extend my "nvim as a fake language server" approach for testing :)

h-michael · 2019-06-14T00:34:24Z

I open the PR #10222. And I would like to discuss where to set the fish line. :)

mcepl · 2019-09-16T08:07:53Z

Shouldn’t this PR be closed? The story is no longer here.

justinmk · 2019-09-22T00:54:00Z

Resolution/Summary

Continued at #10222 .

Locked to keep the summary visible. You can always chat or open a ticket if you have new information/topics to discuss.

tjdevries mentioned this pull request Jun 7, 2017

test with clangd tjdevries/nvim-langserver-shim#17

Closed

Shougo mentioned this pull request Jun 7, 2017

Language Server Implementation (LSP) #5522

Closed

ZyX-I suggested changes Jun 7, 2017

View reviewed changes

wsdjeg reviewed Jun 8, 2017

View reviewed changes

wsdjeg mentioned this pull request Jun 16, 2017

Use Language Server Protocol for autocompletion wsdjeg/SpaceVim#512

Closed

marvim added the WIP label Jun 18, 2017

aktau reviewed Jun 20, 2017

View reviewed changes

tjdevries force-pushed the lsp branch from a7089f7 to 7ebadae Compare June 27, 2017 22:47

justinmk added the lsp label Aug 22, 2017

tjdevries force-pushed the lsp branch from f6c183e to 05ae150 Compare September 29, 2017 15:43

bfredl mentioned this pull request Oct 13, 2017

[RFC] News neovim/neovim.github.io#153

Merged

tjdevries force-pushed the lsp branch 2 times, most recently from 88a04a0 to a950255 Compare December 28, 2017 19:20

justinmk mentioned this pull request Feb 26, 2019

cscope freeze UI when search non-existence symbol on large cscope database #9651

Closed

This comment has been minimized.

Sign in to view

aktau requested a review from ZyX-I March 13, 2019 13:00

m-pilia mentioned this pull request May 5, 2019

Add API for custom LSP requests dense-analysis/ale#2474

Closed

Shougo mentioned this pull request May 22, 2019

signatureHelp Fortran functions Shougo/echodoc.vim#69

Closed

bfredl mentioned this pull request Jun 8, 2019

Artifacts in terminal #10167

Closed

h-michael mentioned this pull request Jun 14, 2019

[RFC] Built-in LSP Support #10222

Closed

20 tasks

justinmk closed this Sep 22, 2019

neovim locked as resolved and limited conversation to collaborators Sep 22, 2019

dundargoc removed the request for review from ZyX-I October 4, 2022 11:58

		@@ -0,0 +1,28 @@
		" TODO(tjdevries): Move this to autoload?

		execute('luafile ' . expand('<sfile>:h') . '/../lua/lsp/plugin.lua')

		@@ -0,0 +1,38 @@
		local helpers = require('test.functional.helpers')(after_each)


		execute('luafile ' . expand('<sfile>:h') . '/../lua/lsp/plugin.lua')

		function! LSP_Start() abort

Uh oh!

Conversation

tjdevries commented Jun 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goals:

Registering callbacks

Creating default callbacks

Uh oh!

bfredl commented Jun 7, 2017

Uh oh!

tjdevries commented Jun 7, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wsdjeg Jun 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tjdevries commented Jun 27, 2017

Uh oh!

justinmk commented Jun 28, 2017

Uh oh!

bfredl commented Jun 28, 2017

Uh oh!

ZyX-I commented Jun 28, 2017

Uh oh!

tjdevries commented Jun 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justinmk commented Jun 29, 2017

Uh oh!

euclio commented Aug 16, 2017

Uh oh!

tjdevries commented Aug 16, 2017

Uh oh!

balta2ar commented Aug 16, 2017

Uh oh!

tjdevries commented Aug 16, 2017

Uh oh!

jvican commented Sep 28, 2017

Uh oh!

blueyed commented Sep 28, 2017

tjdevries commented Jun 7, 2017 •

edited

Loading

wsdjeg Jun 8, 2017 •

edited

Loading

tjdevries commented Jun 28, 2017 •

edited

Loading

justinmk commented Jan 23, 2019 •

edited

Loading

mcepl commented Jun 9, 2019 •

edited

Loading

h-michael commented Jun 11, 2019 •

edited

Loading