The completion table does not perform prefix filtering

[minad]

Thank you for the bug report

• I am using the latest version of lsp-mode related packages.
• I checked FAQ and Troubleshooting sections
• You may also try reproduce the issue using clean environment

Bug description

The lsp completion table is non conforming to the specification. In particular for the all-completions action (t) the returned candidates must be filtered with the given prefix string.

The problem is here:

lsp-mode/lsp-completion.el

Line 504 in fc30baf

((equal action t) (funcall candidates))))

Original issue:

minad/consult#346

Steps to reproduce

See minad/consult#346

Expected behavior

The completion table should be comforming to the specification.

Which Language Server did you use?

Any

OS

Linux

Anything else?

The issue here is that lsp wants to avoid double filtering. My proposal is to not use basic but a custom completion style which does not make use of prefix filtering for the lsp-capf category. Given such a completion style allows to use a fully compliant completion table. In most scenarios the end result will be equivalent. However not in all scenarios, e.g., not in the case when consult-completion-in-region is used, where the completion is transferred to the minibuffer. I can help out with a patch.

Cc @kiennq

[minad]

Eglot has an eglot-stay-out-of mechanism. Would it make sense to provide something like this also for lsp? In this particular case the user may want to opt out from the completion-category-defaults modification.

lsp-mode/lsp-completion.el

Line 734 in fc30baf

(setq-local completion-category-defaults

However it may be easier to just overwrite the completion-category-defaults after activating the lsp-mode. So I think I would not add a separate stay out mechanism.

[yyoncho]

The issue here is that lsp wants to avoid double filtering

The issue is that the emacs completion tables are broken abstraction and do not work with lsp protocol. Emacs completion framework has several assumptions that do not hold in lsp case.

1. The completion prefix is calculated upfront - in lsp-mode case has to be calculated after server has returned.
2. The completion prefix is one - in lsp-mode case the completion prefix is per item and it is calculated based on the content of the buffer.
3. Filtering should not happen on labels but on filterText.

There are probably much more than that.

To solve this issue we might introduce a variable which will force lsp-mode to behave more like the traditional backends and generally recommend against enabling that feature because it will give you an incorrect set of completion items.

[minad]

Yes, I am aware of this impedance mismatch. My proposal is to use a different
workaround, which is more compliant with how completion tables usually behave.
I propose to use this completion table:

(lambda (str pred action)
  (if (eq action 'metadata)
      '(metadata (category . lsp-capf)
                 (cycle-sort-function . identity)
                 (display-sort-function . identity))
    (complete-with-action action (funcall candidates) str pred)))

In order to prevent double filtering a lsp-passthrough completion style can be
introduced, which does not perform filtering but only applies some best-effort
highlighting. Using this completion style for the lsp-capf category ensures
that no candidates are lost due to double filtering. In comparison, Eglot uses
the completion style flex which can potentially lead to the loss of candidates
(unlikely?) but at least ensures that some highlighting is applied.

This change will then give the user more freedom, since the double filtering
will only worked around in a single place, i.e., the lsp-passthrough style. In
cases where the user actually prefers to use double filtering, e.g. in
consult-completion-in-region, which transfers completion to the minibuffer,
the user can then configure a different completion style like basic.

Unfortunately there will still be issues with this approach, in particular in
combination with consult-completion-in-region. The lsp server may only return
a small subset of items, which fit at point. By transferring the completion to
the minibuffer, the further user input in the minibuffer is not taken into
account by the lsp server. The input will only be used for the secondary
filtering. Ultimately I think consult-completion-in-region is not a good fit
for lsp due to the transfer to the minibuffer. Instead completion in region in
the actual buffer should be used (Company or my Corfu package).

[minad]

Above I outlined my proposal again. I also want to discuss your points a bit more specifically:

The completion prefix is calculated upfront - in lsp-mode case has to be calculated after server has returned.

As I see it, the lsp server returns a list of items which fit at point. The server may use the current prefix or something else. The prefix filtering I am talking about is a pure Emacs thing and will always lead to double filtering.

The completion prefix is one - in lsp-mode case the completion prefix is per item and it is calculated based on the content of the buffer.

I am not sure I understand what you mean by that.

Filtering should not happen on labels but on filterText.

The lsp-passthrough completion style could filter based on filterText, which should then be attached as text property to the candidates. Then the name lsp-passthrough is of course not correct anymore, since it is not a passthrough style anymore. But I think it is okay to not make use of filterText and to not perform any double filtering in lsp-passthrough, which should yield the same result as you have now.

To solve this issue we might introduce a variable which will force lsp-mode to behave more like the traditional backends and generally recommend against enabling that feature because it will give you an incorrect set of completion items.

I would avoid this. My proposal involves adding a custom lsp-passthrough style. If this style is used no double filtering is performed and the user gets the correct set of completions. However the user will still have the possibility to opt out of lsp-passthrough by overwriting the style. This would be equivalent to setting such a variable forcing "traditional behavior".

[kiennq]

The completion prefix is one - in lsp-mode case the completion prefix is per item and it is calculated based on the content of the buffer.

I am not sure I understand what you mean by that.

Language server can have a different understanding of prefix compare to Emacs, thus each candidate item returned by the language server can have different prefixes and replace a different range in Emacs's buffer.
Also, there're language servers that don't do filtering but just return everything too, we will need to do filtering in lsp-mode side for that.
What we're doing is we're trying to guess the prefix (not really prefix but more like predicate string) from the buffer based on the returned item, and use that prefix to decide if the item is matched to displayed as results of capf.

As you see, that's why we cannot use the predicate in the completion table function, since that predicate is not so correct (we just try to return the symbol at point, which can be a very different concept between Emacs and LSP).

Note that the (setq-local completion-category-defaults (add-to-list 'completion-category-defaults '(lsp-capf (styles basic))) is to stop company-mode from further filtering the candidate with the wrong prefix via completion-all-completions.
This might be a bug of Emacs but last time I found only completion style basic with its associated functions doesn't touch the (wrong) prefix and just returns whatever the completion table function returns.

Unfortunately there will still be issues with this approach, in particular in
combination with consult-completion-in-region. The lsp server may only return
a small subset of items, which fit at point. By transferring the completion to
the minibuffer, the further user input in the minibuffer is not taken into
account by the lsp server. The input will only be used for the secondary
filtering. Ultimately I think consult-completion-in-region is not a good fit
for lsp due to the transfer to the minibuffer. Instead completion in region in
the actual buffer should be used (Company or my Corfu package)

I agree with this, consult-completion-in-region is not a good fit for lsp, since it's doing change in the minibuffer, and the input is not reflected to the language server. That will make some smart server like clangd, go ... give incomplete results.

Also, since we can only perform secondary filtering at this point, why bothering calling lsp-capf's completion table function again, that would cost more time (request to server again if there's no cache or server with only incomplete results) and jrpc i/o processing with no gain.
Can consult-completion-in-region evaluate the completion table function once when it's launch to get the realized completion list, then filter that normally? (I think Helm is doing this)

Alternatively, perhaps you can simulate the inserting from the minibuffer to the current window buffer so the filter will not be secondary anymore. That would make the current lsp-mode's capf works as well.

[minad]

As you see, that's why we cannot use the predicate in the completion table function, since that predicate is not so correct (we just try to return the symbol at point, which can be a very different concept between Emacs and LSP).

You can use the predicate here. But it does not matter anyways since the predicate will be nil except if the user overwrites it via some advices.

This might be a bug of Emacs but last time I found only completion style basic with its associated functions doesn't touch the (wrong) prefix and just returns whatever the completion table function returns.

This is not correct. The completion styles basic, emacs21 and emacs22 use the prefix for filtering. Only the combination of basic with your completion table, which ignores the prefix, leads to the prefix being ignored. Other completion styles like substring, partial-completion, initials, flex and orderless cannot make use of the prefix and will always pass the empty string to the completion table. You can see it like this - by calling the completion table with a prefix string, basic takes advantages of the filtering inside the table as an optimization.

See also the comment in minibuffer.el: https://github.com/emacs-mirror/emacs/blob/1dba0ca278f8185912e8d39b2af05fc6739b65f8/lisp/minibuffer.el#L3455-L3456

Also, since we can only perform secondary filtering at this point, why bothering calling lsp-capf's completion table function again, that would cost more time (request to server again if there's no cache or server with only incomplete results) and jrpc i/o processing with no gain.

This is how completion tables are supposed to work. The completion table must be called again with each new input. You may want to add some caching in the lsp completion table (There is also completion-table-with-cache). However in practice it only matters for consult-completion-in-region and similar completion-in-region functions which transfer the completion to the minibuffer. If the completion is really performed inside the buffer this is not an issue.

Can consult-completion-in-region evaluate the completion table function once when it's launch to get the realized completion list, then filter that normally? (I think Helm is doing this)

Of course it could, but this is incorrect. It won't work for file table completion then for example. The correct approach is to always recall the completion table. The dynamic completion table should then take care of caching itself such that this is not more costly than calling it once. My Vertico package implements this correctly, as does Icomplete.

Alternatively, perhaps you can simulate the inserting from the minibuffer to the current window buffer so the filter will not be secondary anymore. That would make the current lsp-mode's capf works as well.

Yes, I've considered this. But this is a very lsp specific workaround. I think in that case it would be better to provide a separate consult-lsp-complete function, which accesses the lsp api directly. It could even make use of the async capabilities of consult (see consult-ripgrep for example). Async completions are not possible with capfs. There is the consult-lsp package, such a consult-lsp-complete functionality could go there.

[kiennq]

This is not correct. The completion styles basic, emacs21 and emacs22 use the prefix for filtering. Only the combination of basic with your completion table, which ignores the prefix, leads to the prefix being ignored. Other completion styles like substring, partial-completion, initials, flex and orderless cannot make use of the prefix and will always pass the empty string to the completion table. You can see it like this - by calling the completion table with a prefix string, basic takes advantages of the filtering inside the table as an optimization.

See also the comment in minibuffer.el: https://github.com/emacs-mirror/emacs/blob/1dba0ca278f8185912e8d39b2af05fc6739b65f8/lisp/minibuffer.el#L3455-L3456

You're right.
The problem is that for other completion styles that don't make use of prefix, it alters the result returned from the completion table function. That undesirable since the prefix provided at that time can be different from what the language server expects.
You can see an example in company-capf that use completion-all-completions and supplies the prefix there.
So basically, the basic completion style here means pass-through for lsp-mode.

This is how completion tables are supposed to work. The completion table must be called again with each new input. You may want to add some caching in the lsp completion table (There is also completion-table-with-cache). However in practice it only matters for consult-completion-in-region and similar completion-in-region functions which transfer the completion to the minibuffer. If the completion is really performed inside the buffer this is not an issue.

You miss my response, I specifically stated that if we can only do the secondary filtering there.
We do have caching in the completion table, however, that largely depends on the language server. Some language servers, in order to minimize the RPC payload (and its processing) only return part of completion items (which can be different even with the same input) (1) and expects that we should query again when there's new input in the edit buffer.
Not only that, since we're doing the filtering with supplied predicate from consult-completion-in-region over the returned results from the language server (which has different context), user might see something like candidate items which suppose to be matched suddenly disappeared after entering the next input. The reason is as I state in (1).

Yes, I am aware of this impedance mismatch. My proposal is to use a different
workaround, which is more compliant with how completion tables usually behave.
I propose to use this completion table:

(lambda (str pred action)
  (if (eq action 'metadata)
      '(metadata (category . lsp-capf)
                 (cycle-sort-function . identity)
                 (display-sort-function . identity))
    (complete-with-action action (funcall candidates) str pred)))

Yes, this can work. I will take a look at this

[minad]

The problem is that for other completion styles that don't make use of prefix, it alters the result returned from the completion table function. That undesirable since the prefix provided at that time can be different from what the language server expects.

What do you mean exactly? The list returned by the completion table can be mutated. For example the default completion UI mutates the list by sorting. But the candidate strings itself are not mutated, the highlighting is applied to copies of the strings.

You can see an example in company-capf that use completion-all-completions and supplies the prefix there.
So basically, the basic completion style here means pass-through for lsp-mode.

Yes. The solution I am proposing is equivalent, only that the completion table is compliant. basic won't be passthrough as it is now. If you use an explicit lsp-passthrough completion style the user will have the choice.

You miss my response, I specifically stated that if we can only do the secondary filtering there.
We do have caching in the completion table, however, that largely depends on the language server. Some language servers, in order to minimize the RPC payload (and its processing) only return part of completion items (which can be different even with the same input) (1) and expects that we should query again when there's new input in the edit buffer.
Not only that, since we're doing the filtering with supplied predicate from consult-completion-in-region over the returned results from the language server (which has different context), user might see something like candidate items which suppose to be matched suddenly disappeared after entering the next input. The reason is as I state in (1).

I am sorry, but I don't understand exactly what you are trying to tell here. It is good that you are doing caching and I agree that it is difficult since the server may not expect that you are doing caching. I am also aware that only a part of the candidates may be returned, which will is the first problem with minibuffer-driven completion as in consult-completion-in-region. Furthermore the minibuffer input is not in any way conveyed to the server which is the second problem.

There is one thing I would like to understand better - you say that the lsp server may return different results even with the same input. I don't think that's the case. If consult-completion-in-region calls the table multiple times at the same point in the original buffer without modifying the buffer, the results should always be the same?

[kiennq]

If consult-completion-in-region calls the table multiple times at the same point in the original buffer without modifying the buffer, the results should always be the same?

It's not guaranteed and entirely depends on the language server implementation.

[kiennq]

@minad Can you check if #2975 works for you?
You may need to modify/bind completion-category-overrides though

[minad]

It's not guaranteed and entirely depends on the language server implementation.

Yes, of course the language server could show whatever non-deterministic behavior it likes. But this is what I call a bug then.

[minad]

#2975 looks mostly like what I had in mind, but not quite. I added a few comments.

[kiennq]

Yes, of course the language server could show whatever non-deterministic behavior it likes. But this is what I call a bug then.

Not really, the language server can indexing the code slowly and only returns part of completion items at first. After that at a later time, it can return more accurate items that may not include the previous ones.

[minad]

Not really, the language server can indexing the code slowly and only returns part of completion items at first. After that at a later time, it can return more accurate items that may not include the previous ones.

Okay, fair enough. It is a bit of an ugly behavior. Are there language servers which do that?

[yyoncho]

In comparison, Eglot uses
the completion style flex which can potentially lead to the loss of candidates
(unlikely?) but at least ensures that some highlighting is applied.

We used to use flex in the past and it does not work well - it will either return more results than needed or it will return zero results(the same is true for other completion categories). Everyone applying a different(not the one implemented in lsp-mode) filtering strategy should be warned that the completion result might not be the real one.

[minad]

@yyoncho

We used to use flex in the past and it does not work well - it will either return more results than needed or it will return zero results(the same is true for other completion categories).

You are right that it may return fewer results than not performing double filtering. However it cannot be that it shows more results than with the current implementation which does not perform double filtering. Overall I agree with you that flex is not an ideal solution. I am also not proposing it here.

Everyone applying a different(not the one implemented in lsp-mode) filtering strategy should be warned that the completion result might not be the real one.

Agree, with a custom filtering strategy you may see fewer candidates than with the lsp-passthrough strategy I am proposing or the current strategy (basic plus completion table, which ignores the prefix).

[yyoncho]

Agree, with a custom filtering strategy you may see fewer candidates than with the lsp-passthrough strategy I am proposing or the current strategy (basic plus completion table, which ignores the prefix).

Maybe I misunderstood your suggestion. I thought that inserting the text in the buffer is problematic for you. If it is not, as long as the buffer is up to date and the completion table filtering is performed by lsp-mode it will work fine.

[minad]

Maybe I misunderstood your suggestion. I thought that inserting the text in the buffer is problematic for you. If it is not, as long as the buffer is up to date and the completion table filtering is performed by lsp-mode it will work fine.

Yes, this is problematic for me. I won't implement this in consult-completion-in-region since it would be an lsp-specific workaround. All I am concerned with this issue here is that lsp-mode provides a completion table which is compliant with the specification. I rely on the specification in my completion packages (Corfu, Consult and Vertico).

[yyoncho]

I got confused:

All I am concerned with this issue here is that lsp-mode provides a completion table which is compliant with the specification. I rely on the specification in my completion packages (Corfu, Consult and Vertico).

I am not sure what your proposal for complying with Emacs completion specification and follow lsp protocol specification is. Without having the text inserted we cannot handle partial results and we cannot perform filtering.

[minad]

The PR by @kiennq goes in the right direction and only has a few minor issues. The point is that the completion table should follow the Emacs completion specification.

(lambda (str pred action) ;; compliant completion table
  (if (eq action 'metadata)
      '(metadata (category . lsp-capf)
                 (cycle-sort-function . identity)
                 (display-sort-function . identity))
    (complete-with-action action (funcall candidates) str pred)))

In comparison the current completion table:

lsp-mode/lsp-completion.el

Lines 488 to 504 in 4f23138

	(lambda (probe _pred action)
	(cond
	;; metadata
	((equal action 'metadata)
	`(metadata (category . lsp-capf)
	(display-sort-function . identity)))
	;; boundaries
	((equal (car-safe action) 'boundaries) nil)
	;; try-completion
	((null action)
	(when-let ((cands (funcall candidates)))
	(if (cl-rest cands) probe (cl-first cands))))
	;; test-completion: not return exact match so that the selection will
	;; always be shown
	((equal action 'lambda) nil)
	;; retrieve candidates
	((equal action t) (funcall candidates))))

By providing a custom completion style (called lsp-passthrough in this issue) you can achieve the same result as you have now and it will work with the compliant completion table, but it will also work in more scenarios (e.g. consult-completion-in-region). Note that this is a best-effort attempt - it won't be a perfect solution since this is not possible due to the impedance mismatch between lsp completion and Emacs completion as you pointed out in your first comment here.

[minad]

For comparison - Eglot filters the candidate by checking for the probe prefix string: https://github.com/joaotavora/eglot/blob/5cc8df63d86a6c43134dd6e4e3ae26cfae14e66a/eglot.el#L2324-L2331 This is the correct thing to do. In comparison to my snippet above it is even a bit more compliant with the lsp specification since it respects filterText.

[yyoncho]

Note that this is a best-effort attempt - it won't be a perfect solution since this is not possible due to the impedance mismatch between lsp completion and Emacs completion as you pointed out in your first comment here.

Ok, what I was just saying is that we should make that visible to avoid misleading the users. In certain cases, I find broken completion extremely irritating.

[minad]

Ok, what I was just saying is that we should make that visible to avoid misleading the users. In certain cases, I find broken completion extremely irritating.

I agree. I am just trying to get the best out of it with what we have. With my proposal we should get a solution which is as good as it gets regarding compliance with lsp and Emacs completion specification (except for the highlighting, but this is currently also imperfect).

• in buffer completion with company - no double filtering
• in buffer completion with corfu - no double filtering
• minibuffer completion with consult-completion-in-region plus vertico or selectrum - double filtering if another completion style than lsp-passthrough is used. Lsp server does not see the updated filtering string, but the user still has the ability to filter the candidate returned by the lsp server in the minibuffer. best effort only 🤷

It makes sense to point out that minibuffer completion like consult-completion-in-region is not a good solution for lsp-mode/eglot. I still see some value if each component tries to follow the APIs as closely as possible.

[kiennq]

For comparison - Eglot filters the candidate by checking for the probe prefix string: https://github.com/joaotavora/eglot/blob/5cc8df63d86a6c43134dd6e4e3ae26cfae14e66a/eglot.el#L2324-L2331 This is the correct thing to do. In comparison to my snippet above it is even a bit more compliant with the lsp specification since it respects filterText.

Eglot is not respecting prefix defined by language server and is using the prefix from Emacs only, that's why in some cases, you will see that it put much less item after filtering compare to lsp-mode.
Note that we do the prefix matching inside the candiates function already (using LSP's defined prefix), the (complete-with-action action (funcall candidates) str pred) is just for second filtering to conform with Emacs capf.
And I've realized that the complete-with-action is actually not really doing filtering at all, since what it's doing is calling all-completions which is not using any styles defined in completion-styles.

Good thing is that with the custom completion style defined, we separated the case between calling by the completion table via completion-all-completions (1) and invoke the completion table directly (2).
@minad How is consult-completion-in-region is getting data from capf? Can I assume it falls into the case (2) above?

except for the highlighting, but this is currently also imperfect

The highlighting is working for company-mode via :company-match.

[minad]

@kiennq

Eglot is not respecting prefix defined by language server and is using the prefix from Emacs only, that's why in some cases, you will see that it put much less item after filtering compare to lsp-mode.

Correct.

Note that we do the prefix matching inside the candiates function already (using LSP's defined prefix), the (complete-with-action action (funcall candidates) str pred) is just for second filtering to conform with Emacs capf.

Correct. The prefix filter is the second-level Emacs filtering to conform with the Emacs specification. This is what I am asking for here.

And I've realized that the complete-with-action is actually not really doing filtering at all, since what it's doing is calling all-completions which is not using any styles defined in completion-styles.

No, complete-with-action calls all-completions with the probe as prefix argument. all-completions returns only the candidates with the probe prefix, so in fact it performs filtering.

Good thing is that with the custom completion style defined, we separated the case between calling by the completion table via completion-all-completions (1) and invoke the completion table directly (2). How is consult-completion-in-region is getting data from capf? Can I assume it falls into the case (2) above?

All my completion packages (and also Emacs default completion and Icomplete) invoke the completion table via completion-all-completions since this is how you make sure that the candidates are filtered by the completion style. Note that the completion frontends should work with many different completion tables (not only lsp), such that there is no way around calling completion-all-completions. In order to avoid filtering I am proposing the introduction of the lsp-passthrough style specifically for lsp-mode which explicitly does not perform filtering and only adds highlighting.

The highlighting is working for company-mode via :company-match.

Of course it does. But this is a best effort approach. The highlighting is applied by the basic completion style, not by the language server. The language server may filter the candidates differently than prefix completion, such that the prefix highlighting may be imperfect. Afaik the lsp protocol does not specify a candidate highlighting for the returned completion candidates.

[kiennq]

Of course it does. But this is a best effort approach. The highlighting is applied by the basic completion style, not by the language server. The language server may filter the candidates differently than prefix completion, such that the prefix highlighting may be imperfect. Afaik the lsp protocol does not specify a candidate highlighting for the returned completion candidates.

The highlighting is determined via the :company-match property (which is a function) returned from capf.
This is generally better than try to do highlighting all the returned item since this can be done lazily, only when the item is in view.

No, complete-with-action calls all-completions with the probe as prefix argument. all-completions returns only the candidates with the probe prefix, so in fact it performs filtering.

Yes, and it does not respect the user's setting completion-styles, which is why I consider it not really doing filtering in user;s favor

[minad]

The highlighting is determined via the :company-match property (which is a function) returned from capf.

Ah I see, you are right. So this makes use of the highlighting data as returned by lsp? The downside is that this is a company specific extension, which is incompatible with other completion UIs. It would be better to apply the highlighting on the level of the completion style and removing :company-match. Then one would get a solution working with every UI (including company).

Yes, and it does not respect the user's setting completion-styles, which is why I consider it not really doing filtering in user;s favor

No, it respects the completion styles or rather you are confusing the different levels here. The backend/completion-table implementation sits one level below the completion styles. all-completions belongs to this backend level. The completion style can ask the completion table for all the candidates (passing the argument t). If the completion style wants the backend to perform prefix-filtering on the backend-side it will pass the desired prefix string as probe argument (The prefix completion styles basic, emacs21 and emacs22 do that). If the completion style does not want to have prefix-filtering it will pass the empty string as the probe argument to the backend (The styles partial-completion, initials, flex and orderless pass the empty string to the backend).

[yyoncho]

@minad

I still see some value if each component tries to follow the APIs as closely as possible.

If you ask me, better to throw away this completion implementation and create a simpler async specification(e. g. following what company-mode or other editors have) instead of having the current cryptic spec/implementation.

[kiennq]

No, it respects the completion styles or rather you are confusing the different levels here. The backend/completion-table implementation sits one level below the completion styles. all-completions belongs to this backend level.

I think you missed my point, all it does is doing prefix matching, then certainly it's not respecting the user's completion-styles here.
I think that we should just call completion-all-completions (which see the completion-styles) instead of forwarding using complete-with-action here. See my latest change in the PR.

[minad]

@yyoncho

If you ask me, better to throw away this completion implementation and create a simpler async specification(e. g. following what company-mode or other editors have) instead of having the current cryptic spec/implementation.

Well, then propose it on emacs-devel. I agree with you that the current spec is baroque and complicated (and confusing as this discussion shows). But one can work with it and so far it went well enough in my use cases. Note that completion tables serve a dual purpose since they can complete candidates and filter so this complexity is hard to argue away. Then there is the big issue with backward compatibility. There are hundreds of completing-read calls in Emacs. For capfs it is not that bad, but capfs and completing-read share a part of their specification. Of course one could question why capfs and completing-read share their specification - one could decouple this. But then one loses support for completion-styles. If you look at Company, it currently promotes to use capfs instead of implementing custom backends. I think this is a good idea since this allows the backends to work with different UIs. You could of course invent your own API here and then develop a frontend which works specifically and only with the frontend of your choice, for example you could add a Company backend. There has been company-lsp, why did you abandon it? I think that was the right choice but it would still be interesting to hear the reasoning.

Regarding async - you are right that there is some potential for improvement. In the case of completing-read one can implement asynchronous completion tables as I am doing in Consult. For Capfs I've not yet thought about a solution, but I assume that one could do something similar on the level of the completion table. The only component which is basically missing is a generic possibility for the completion table to inform the frontend that candidates became available.

[yyoncho]

Well, then propose it on emacs-devel.

At some point I will, this is not my top priority right now.

I agree with you that the current spec is baroque and complicated (and confusing as this discussion shows). But one can work with it and so far it went well enough in my use cases. Note that completion tables serve a dual purpose since they can complete candidates and filter so this complexity is hard to argue away. Then there is the big issue with backward compatibility. There are hundreds of completing-read calls in Emacs. For capfs it is not that bad, but capfs and completing-read share a part of their specification. Of course one could question why capfs and completing-read share their specification - one could decouple this. But then one loses support for completion-styles. If you look at Company, it currently promotes to use capfs instead of implementing custom backends. I think this is a good idea since this allows the backends to work with different UIs. You could of course invent your own API here and then develop a frontend which works specifically and only with the frontend of your choice, for example you could add a Company backend.

IMHO it all can be done without breaking the existing code.

There has been company-lsp, why did you abandon it? I think that was the right choice but it would still be interesting to hear the reasoning.

lsp-mode was criticized for the need to install one more package and also the company-lsp maintainer was inactive and we didn't have write permissions for their repo. Also, it has the issues that eglot currently has - it was calculating the prefixes incorrectly and we started implementing the capf to correct those. Later that turned into a saga.

[yyoncho]

Oh, and forgot the most important part - move that completion framework in elpa.

[kiennq]

There has been company-lsp, why did you abandon it? I think that was the right choice but it would still be interesting to hear the reasoning.

lsp-mode was criticized for the need to install one more package and also the company-lsp maintainer was inactive and we didn't have write permissions for their repo. Also, it has the issues that eglot currently has - it was calculating the prefixes incorrectly and we started implementing the capf to correct those. Later that turned into a saga.

IMO, capf is not that bad, it's only missing good documentations and async callback. We work around the async requirement by doing canceling on user typing already, the left part is for good documentation.

[minad]

What is the status here and in #2975?

[yyoncho]

@minad - are you fine with the changes in that PR? I was planning to merge it once you approve it?

[minad]

@yyoncho Yes, please merge the PR. It fixes this issue.

[yyoncho]

I approved it, I will give some time to @kiennq in case smt else is missing and then I will proceed wth the merge.