cider-enable-flex-completion is not a sufficient replacement for cider-company-enable-fuzzy-completion

[alexander-yakushev]

After I enabled cider-enable-flex-completion instead of cider-company-enable-fuzzy-completion, I noticed that my completion was acting up. Certain prefixes that used to complete correctly now failed to complete, and weird suggestions were popping up. The weird suggestions are understandable because of flex aggressive fuzzy policy, but that's not that big of an issue. The issue is the following completion scenarios supported by compliment that don't work in flex:

1. Namespace completion by first letters of namespaces:

• (cider-complete "cji") => (#("clojure.java.io" 0 1 (ns nil type "namespace")))
• Tab with flex => no completions.

2. FQ classname completions by first word:

• (cider-complete "InputSt") => (#("java.io.InputStream" 0 1 (ns nil type "class")) #("java.io.InputStreamReader" 0 1 (ns nil type "class")) ...
• Tab with flex => no completions.

3. Fuzzy completion behind namespace prefix:

• (cider-complete "str/sl") => (#("str/split-lines" 0 1 (ns "clojure.string" type "function")))
• Tab with flex => no completions.

There are a couple of others but they are not as important as these.

Additionally, Emacs achieves this flex behavior by sending two requests to the backend: one with the normal prefix and one with empty prefix (""). Since we have no upper bound on the number of candidates we return, this may add some extra overhead/delay for the completion because CIDER has to deserialize all that output.

[alexander-yakushev]

Overall, the less the frontend (editor) tries to be smart about the completion candidates, the better. We can control the UX match better on the backend. And if there are users that want flex-like fuzzy matching, we can also add that to Compliment and hide behind a toggle. That is still much more efficient and makes more sense than filtering through everything on the frontend side.

[vemv]

Thanks for the issue!

We should tread carefully here because we shouldn't necessarily be smarter than (or reinvent) Emacs' mechanisms.

Most of all, it's a concern of simplicity - Emacs has an abundance of styles, and there's also an abundance of Emacs packages building up on those, so creating our own standards can possibly make things even messier for everyone involved.

I don't have a specific suggestion/stance except: we can try first hard at adhering at Emacs standards and squeezing performance out of them.

If we document reasoning/facts reflecting that that's impossible, we could keep exploring solutions.

[vemv]

Quick q - would it make sense to pass the completion style to Compliment?

In my mind it would make sense - then Compliment could infer what's wanted and return fewer / more relevant results.

[alexander-yakushev]

I'll repeat just in case: the problem is Emacs not displaying valid completion candidates returned by Compliment. Some of those are really important (like completing a class by its short name).

What you've proposed, if I understand correctly, would be to pass say flex to Compliment and an empty prefix and so that Compliment understands that Emacs will do the filtering and it should return everything it knows. But that would mean, for example, returning 600 thousand loaded classnames and I doubt that would be an efficient thing to do.

[vemv]

What if we passed flex and what the user has typed?

So it would include (among other stuff) these 3 things:

• What Emacs wants us to complete - ""
• The style - "flex"
• What CIDER knows that the user has actually typed - "InputSt"

[alexander-yakushev]

We should tread carefully here because we shouldn't necessarily be smarter than (or reinvent) Emacs' mechanisms.

I think we should be smarter than Emacs mechanisms. I don't see mainline Emacs as the frontier in completion solutions; e.g., company-mode steps away from it in many regards and there is a reason we use company and not just completion-at-point.

[alexander-yakushev]

What if we passed flex and what the user has typed?

Then Compliment doesn't really need to know that it was flex that triggered it, the returned candidates would be the same.

[alexander-yakushev]

I think I understood what you mean. So we would return the same "preferred" result twice, for normal prefix, and for the empty prefix, and let Emacs show the candidates that we want to show in the empty prefix case. I mean, that would work, but it feels like a bigger hack to me than what cider-company-enable-fuzzy-completion was doing.

[vemv]

company-mode steps away from it in many regards and there is a reason we use company and not just completion-at-point.

Well we (certainly you and me) use Company but many others user Corfu and whatnot.

That's where adhering to standards show their value - supporting a large divesity of users

Then Compliment doesn't really need to know that it was flex that triggered it, the returned candidates would be the same.

Then we have a potential solution, right? Have CIDER pass the actual user input as an additional k-v

(passing flex might be worthwhile for further trimming results, but I'd understand if you didn't want that duplicated work)

[vemv]

let Emacs show the candidates that we want to show in the empty prefix case.

Yes

I mean, that would work, but it feels like a bigger hack to me than what cider-company-enable-fuzzy-completion was doing

It seems a clean, low-effort, low-complection solution to me. Pass additional properties from the frontend and act on them on the backend.

To me (as someone who strives to support a diverse set of users) there's high value in adhering to Emacs standards, so I'd vote for this solution unless there was a huge factual drawback.

[alexander-yakushev]

Actually, I tried it again, and I see two identical messages are sent to nrepl for each completion attempt:

(-->
  id                        "570"
  op                        "complete"
  session                   "ecd63e82-c777-435b-8dbd-16dcf3b3a01b"
  time-stamp                "2024-05-05 21:42:27.829532000"
  context                   "nil"
  enhanced-cljs-completion? "t"
  prefix                    "InputS"
)
(<--
  id          "570"
  session     "ecd63e82-c777-435b-8dbd-16dcf3b3a01b"
  time-stamp  "2024-05-05 21:42:27.832275000"
  completions ((dict "candidate" "java.io.InputStream" "type" "class") ...)
  status      ("done")
)
(-->
  id                        "571"
  op                        "complete"
  session                   "ecd63e82-c777-435b-8dbd-16dcf3b3a01b"
  time-stamp                "2024-05-05 21:42:27.855899000"
  context                   "nil"
  enhanced-cljs-completion? "t"
  prefix                    "InputS"
)
(<--
  id          "571"
  session     "ecd63e82-c777-435b-8dbd-16dcf3b3a01b"
  time-stamp  "2024-05-05 21:42:27.858973000"
  completions ((dict "candidate" "java.io.InputStream" "type" "class") ...)
  status      ("done")
)

How would passing extra arguments to Compliment help here?

[vemv]

Please describe what's wrong in the nrepl interaction above

[alexander-yakushev]

The problem is that Emacs sends two identical requests to the backend (with prefix "InputS"). Compliment returns the correct result both times, but Emacs refuses to render it.

[vemv]

Why does that happen?

[alexander-yakushev]

Do you ask me? :)

However, if the prefix is all-lowercase, then it sends the second request with an empty prefix:

(-->
  id                        "658"
  op                        "complete"
  session                   "ecd63e82-c777-435b-8dbd-16dcf3b3a01b"
  time-stamp                "2024-05-05 21:55:04.272861000"
  context                   "(__prefix__)
"
  enhanced-cljs-completion? "t"
  prefix                    "cji"
)
(<--
  id         "659"
  session    "ecd63e82-c777-435b-8dbd-16dcf3b3a01b"
  time-stamp "2024-05-05 21:55:04.293445000"
  status     ("done" "no-eldoc")
)
(-->
  id                        "661"
  op                        "complete"
  session                   "ecd63e82-c777-435b-8dbd-16dcf3b3a01b"
  time-stamp                "2024-05-05 21:55:04.353659000"
  context                   "(__prefix__)
"
  enhanced-cljs-completion? "t"
  prefix                    ""
)
(<--
  id          "661"
  session     "ecd63e82-c777-435b-8dbd-16dcf3b3a01b"
  time-stamp  "2024-05-05 21:55:04.387442000"
  completions ((dict "candidate" "*" "ns" "clojure.core" "type" "function") ...)
  status      ("done")
)

[alexander-yakushev]

Basically, if CIDER can control what is sent to Compliment, then you don't really have to pass flex to it and the prefix in some other field – you can just perform the regular query. The trick is to make Emacs show everything that Compliment returned, not some arbitrary subset of it.

[vemv]

Do you ask me? :)

Okay, we'd have to look into this first - could plausibly be a bad implementation on our end

Without this information there's not really much decision-making to do.

[alexander-yakushev]

Okay, we'd have to look into this first - could plausibly be a bad implementation on our end

I thought it was Emacs completion engine that controls it. If not, then it would make things easier, I suppose.

[alexander-yakushev]

Without this information there's not really much decision-making to do.

As long as cider-company-enable-fuzzy-completion exists, one doesn't have to be made promptly. Just something to keep in mind before accidentally removing it. Possibly "undeprecate" it since this part of the docs now shows incorrect information: https://docs.cider.mx/cider/usage/code_completion.html#fuzzy-candidate-matching. CC @bbatsov.

[vemv]

Fixing the issue would seem a good course of action - having churn in the docs can easily be confusing.

We can add a ;; DO NOT DELETE https://github.com/clojure-emacs/cider/issues/3653 comment in the meantime

For clarity - would you be interested into digging into the Elisp part?

[alexander-yakushev]

Maybe later – looks like quite a bit digging is required.

We can add a ;; DO NOT DELETE #3653 comment in the meantime

I think it's fine, enough people have the context now.

[bbatsov]

I thought it was Emacs completion engine that controls it. If not, then it would make things easier, I suppose.

Yeah, I'm reasonably sure the Emacs completion engine controls it. But there's always the chance we've messed something up on our end. 😅

As long as cider-company-enable-fuzzy-completion exists, one doesn't have to be made promptly. Just something to keep in mind before accidentally removing it. Possibly "undeprecate" it since this part of the docs now shows incorrect information: https://docs.cider.mx/cider/usage/code_completion.html#fuzzy-candidate-matching. CC @bbatsov.

Sure. Feel free to update the docs to reflect your findings. I guess I haven't used the new flex completion, so I didn't notice the issues with it. (I switched from Company to Corfu a while ago and I was too lazy to customize it to my liking so I went back to the basics)

[bbatsov]

I see we also have a bunch of notes about missing functionality here:

;; defines a completion style named `cider' (which ideally would have been named `cider-fuzzy').
;; note that there's already a completion category named `cider' (grep for `(metadata (category . cider))` in this file),
;; which can be confusing given the identical name.
;; The `cider' completion style should be removed because the `flex' style is essentially equivalent.
;; (To be fair, `flex' was introduced in Emacs 27, 3 years in after our commit 04e428b
;;  which introduced `cider-company-enable-fuzzy-completion')
(add-to-list 'completion-styles-alist
             '(cider
               cider-company-unfiltered-candidates
               cider-company-unfiltered-candidates
               "CIDER backend-driven completion style."))

;; Currently CIDER completions only work for `basic`, and not `initials`, `partial-completion`, `orderless`, etc.
;; So we ensure that those other styles aren't used with CIDER, otherwise one would see bad or no completions at all.
;; This `add-to-list` call can be removed once we implement the other completion styles.
;; (When doing that, please refactor `cider-enable-flex-completion' as well)
(add-to-list 'completion-category-overrides '(cider (styles basic)))

That might be related to the problems that @alexander-yakushev has observed.

[vemv]

Yes, surely if we implemented the other styles, other issues would become more evident.

[vemv]

Yeah, I'm reasonably sure the Emacs completion engine controls it. But there's always the chance we've messed something up on our end. 😅

Title / OP in #3006 hints that our current state isn't quite polished.

[bbatsov]

Ah, yeah - I had forgotten about it.