NCM2 completion support?

[HiPhish]

Hi there,

This is a great plugin, but I would like to make completion work with ncm2 rather than Deoplete. I know how to write a source (done that already for Vlime), but I need a bit of help understanding how completion is implemented in Comrade.

Looking at the Deoplete source and comrade#RequestCompletion it appears that you are using a blocking RPC request to fetch completion results. You then use a combination of Deoplete's is_async context entry and the is_finished entry of the results object.

Can you please elaborate how they fit together? NCM2 uses callbacks instead, the idea is that when completion is requested you send off an RPC notification and specify a callback which will be called when the other side notifies Neovim that it is done. See here for an example where I pass a lambda as the callback.

I think once I understand this part I can write the NCM2 source with ease.

[beeender]

Hi!

When deoplete wants to do completion, it sends a rpc request to comrade intelliJ side through comrade#RequestCompletion, see

Comrade/autoload/comrade.vim

Line 25 in 33c0d2f

function! comrade#RequestCompletion(buf, param)

IntelliJ side handles the request here https://github.com/beeender/ComradeNeovim/blob/6006eaf8b26f1baec12cd84e190d7dc96cda4ed2/src/main/kotlin/org/beeender/comradeneovim/completion/CompletionManager.kt#L42

The response definition can be seen at https://github.com/beeender/ComradeNeovim/blob/6006eaf8b26f1baec12cd84e190d7dc96cda4ed2/src/main/kotlin/org/beeender/comradeneovim/completion/CompletionManager.kt#L12
Basically it is a map like:

{
"is_finished": bool
"candidates": list)
}

the way that deoplete handles async can be seen at

Comrade/rplugin/python3/deoplete/sources/comrade_complete.py

Line 27 in f1ec541

def gather_candidates(self, context):

The whole process is something like:

1
[Neovim]   send comrade_complete
2
[IntelliJ] receive comrade_complete
           create a result response object
           set response.is_finished = false
           start completion in background
           (there is a callback keeps adding completion result)
           return the response to Neovim. At this time, the response probably contains an empty completion list.
3
[Neovim]  deoplete get the response, show the completion list.
          check if response.is_finished == true
          not finished, send comrade_complete request again in a certain amount of time.
4
[IntelliJ] receive comrade_complete
           check if it is a new request
           not new request, send the result which is created in step 2 to Neovim
           (The result object is being filled in a background thread started in step 2)

repeat step 3 and 4

5
[IntelliJ] Finished the completion, set the response.is_finished = true
6
[Neovim]  ...
          send comrade_complete request

[IntelliJ] check if it is a new request
           send the current result response (is_finished has been set to true)
[Neovim]   add result to the completion list
           check if response.is_finished == true
           DONE

The intelliJ part code is a bit bind with the deoplete work flow especially this kind of delta report. I am not sure if NCM2 supports something similar. But it won't be too difficult to support both of them.

[HiPhish]

Wow, thank you for the detailed response, especially the last part. If I understand this part correctly

2
[IntelliJ] receive comrade_complete
           create a result response object
           set response.is_finished = false
           start completion in background
           (there is a callback keeps adding completion result)
           return the response to Neovim. At this time, the response probably contains an empty completion list.
3
[Neovim]  deoplete get the response, show the completion list.
          check if response.is_finished == true
          not finished, send comrade_complete request again in a certain amount of time.

Comrade immediately returns a dictionary of incomplete results, then keeps collecting more results on a separate thread. Every time Deoplete requests more results (send comrade_complete request again in a certain amount of time), Comrade returns a dictionary with the results it has accumulated so far. Eventually it sends a final response which contains all results and then Deoplete knows it can stop bothering Comrade. Is this correct?

[beeender]

Yes! correct!

[HiPhish]

I think I'm getting somewhere, but it's not quite there yet. The manual scheduling of repeated requests is what is giving me trouble.

" This function sends a blocking request to IntelliJ; IntelliJ returns almost
" immediately, but the result is incomplete, it keeps building up further
" results in the background.
function! s:send_request(ctx, buf_id, ret)
	" body
	let l:results = comrade#RequestCompletion(a:buf_id, a:ret)
	let a:ret['new_request'] = v:false

	while !l:results['is_finished']
		let l:results = comrade#RequestCompletion(a:buf_id, a:ret)
		if !empty(l:results.candidates)
			call ncm2#complete(a:ctx, a:ctx.startccol, l:results.candidates)
		endif
		call wait(10, { -> v:false })
	endwhile
endfunction

It does generate completion results, but not beyond the first two characters. And the results are not the same as in IntelliJ, for example pr will not suggest private, but a number of class names. Is this normal or is the problem on my end?

The intelliJ part code is a bit bind with the deoplete work flow especially this kind of delta report. I am not sure if NCM2 supports something similar. But it won't be too difficult to support both of them.

I am no expert when it comes to NCM2, but in general there are two functions: the first one is a callback defined by the user which gets called when completions need to be generated, the other one is defined by NCM2 and gets called by the completion source. The way I did it in Vlime is a follows:

• In my callback I send a notification to Vlime to get completion results. Since it is a notification my function immediately returns and nothing happens.
• Vlime computes the result and sends a notification back to Neovim when it's done
• The notification triggers another callback (also defined by me) which actually feeds the results into NCM2 (calls ncm2#complete), which displays the results.

This leaves me only with having to connect the callbacks and massage the data into the proper shape. All the scheduling Is done already for me.

[HiPhish]

It's me again, I tried Comrade with Deoplete to see how it fares, and I am not getting the same completion results as I am getting in IntelliJ. It will only complete symbols which are inside the current class (like this or members of it), but not keywords (like private) or suggest classes from other packages. Is that normal, or have I not set up things correctly?

I also found that my LSP plugin (LanguageClient-neovim) does not work at all if Comrade is installed and IntelliJ is running. No completion suggestions or anything, even though it does load.

[beeender]

It's me again, I tried Comrade with Deoplete to see how it fares, and I am not getting the same completion results as I am getting in IntelliJ. It will only complete symbols which are inside the current class (like this or members of it), but not keywords (like private) or suggest classes from other packages. Is that normal, or have I not set up things correctly?

Right, the completion results are not exact same with IntelliJ. The completion system of IntelliJ is quite complex, and it is not fully designed to be used like this.
Some results has been filtered out since the deoplete cannot handle it, see https://github.com/beeender/ComradeNeovim/blob/master/src/main/kotlin/org/beeender/comradeneovim/completion/DeopleteCandidate.kt#L14
Some results are not given to our completion handler for some reasons I am not aware.

I also found that my LSP plugin (LanguageClient-neovim) does not work at all if Comrade is installed and IntelliJ is running.

I found the same problem some days ago. Maybe it is caused by the nature that this plugin has to do the buffer sync with IntelliJ. It takes over the buf write here. So the LSP cannot sync buffer with LSP server anymore? (just a guess). Not sure it is something can be fixed. Keeping the buffer sync with 3 clients sounds difficult (LSP, neovim, IntelliJ).

Also, from neovim 0.5.0, the LSP client will be a built-in feature. So maybe intellij-lsp-server is the direction to go?

[HiPhish]

Right, the completion results are not exact same with IntelliJ. The completion system of IntelliJ is quite complex, and it is not fully designed to be used like this.

Oh, that really sucks. It means I won't be able to use Comrade for my work where I need to depend on much better completion. I'll still try to finish my work now that I have started it (I just found out a few days ago that Vim 8 introduced timers and that Neovim has merged that feature as well, should make things much easier). I can send you a PR, or I can make it into a standalone plugin if you don't want code you don't maintain yourself in your repo.

Also, from neovim 0.5.0, the LSP client will be a built-in feature. So maybe intellij-lsp-server is the direction to go?

I have seen that and I have taken a peek at the API in the currently nightly builds. It looks really cool that it is so low-level, that way I should be able to really make it my own and hook up anything I want. I really hate how IDEs are built in such a way that they force you into writing your project in The One True Way. I much prefer Vim as a "desintegrated development environment" where you have a number of specialized tools that you wire up the way you want it.

[beeender]

Oh, that really sucks. It means I won't be able to use Comrade for my work where I need to depend on much better completion. I'll still try to finish my work now that I have started it (I just found out a few days ago that Vim 8 introduced timers and that Neovim has merged that feature as well, should make things much easier).

Do you have any special completion case that comrade doesn't work but the IntelliJ does? Maybe I can take a look at the IntelliJ plugin side to see if there is any options can be tuned to support that.

I can send you a PR, or I can make it into a standalone plugin if you don't want code you don't maintain yourself in your repo.
Yeah, PR is good, support both deoplete and NM2 sounds nice!

[HiPhish]

Do you have any special completion case that comrade doesn't work but the IntelliJ does?

For example if you take the following main class:

package com.company;

public class Main {

    public static void main(String[] args) {
    }
}

And start typing Str in IntelliJ I get suggestions for classes not in any of the imported packages, such as Stream. In Neovim I only get what I can have with the current imports.

Yeah, PR is good, support both deoplete and NM2 sounds nice!

OK

[joshua7v]

The intelliJ part code is a bit bind with the deoplete work flow especially this kind of delta report. I am not sure if NCM2 supports something similar. But it won't be too difficult to support both of them.

what does "support both of them" mean?
let ncm2 support current comrade/deoplete completion flow
or let comrade support ncm2 completion

I'm interested because Im making an extension for coc
which seems does not support deoplete-like completion work flow

	while !l:results['is_finished']
		let l:results = comrade#RequestCompletion(a:buf_id, a:ret)
		if !empty(l:results.candidates)
			call ncm2#complete(a:ctx, a:ctx.startccol, l:results.candidates)
		endif
		call wait(10, { -> v:false })
	endwhile

currently i'm using some logic kind like this (by HiPhish),
checking 'is_finished' after some duration over and over again, which is no good

about coc
I'm also new to coc extension development,
seems when completion triggered, it could call an async custom function to prepare completion items,
and the issue here is I cannot know the exact time when comrade is ready.

[beeender]

@joshua7v Maybe you can try to check the JetBrain side log to see if it receives the request from coc plugin.

[joshua7v]

@beeender thanks,

well, with the logic mentioned above, completion actually works,
but my concern is about this

The intelliJ part code is a bit bind with the deoplete work flow especially this kind of delta report.

ncm2 / coc.nvim may not support this delta report thing, could comrade provide another approach like collect all - then notify

[HiPhish]

@joshua7v I have a fork of Comrade if you want to know how I did it. It's a pretty ugly hack though: I use a timer to repeatedly poll Comrade for completion results and display whatever is available. Once Comrade tells me that it is done I deleted the timer. I haven't bothered making a pull request because the result is pretty wonky and I don't use Comrade anymore.

@beeender If you are OK with half-assed support for NCM2 I can make a pull request. Otherwise this issue can be closed as far as I'm concerned.