Allow using completion-styles in Helm and implementation of own helm style

[joaotavora]

Expected behavior

Helm's completion engine can use SLY's completion functions as a backend. The original issue is here: joaotavora/sly#214

Actual behavior (from emacs-helm.sh if possible, see note at the bottom)

The value for completion-at-point-functions is sly-complete-symbol which does the completion itself. This does not trigger the problem. It only happens in other situations such as when SLY's sly-read-symbol-name calls completing-read which Helm overrides via completing-read-function. Actually it used to happen because I've turned off that overriding in that particular context context.

But the root problem remains and I guess some users would like to use Helm's UI for completing SLY symbols. However, when trying

(helm-comp-read "foo " (sly--completion-function-wrapper sly-complete-symbol-function))

Nothing interesting happens. sly--completion-function-wrapper is supposed to produce a reasonably well-behaved completion-function/table. It works well with SLY's own sly-complete-symbol

But it probably isn't well behaved enough, or not to Helm's expectations. I want you to help me understand what's missing.

In particular SLY's lower-level sly-flex-completions and sly-simple-completions functions, which request completions over the network, will return an empty list of completions when the pattern is the empty string, i.e. they only start operating once you feed them at least a character.

Perhaps Helm expects the full completion list to be handed to it up front. But that is impractical because it's frequent that a running SLY session has many thousands of symbols available. Is there a way for Helm to behave more incrementally, a little bit like company does?

Also sly-flex-completions will perform "scatter-match" completion instead of "prefix" completion, which is what sly-simple-completions does.

Steps to reproduce (recipe)

./emacs-helm.sh
(add-to-list 'load-path "~/Source/to/sly")
(require 'sly-autoloads)
M-x sly ; this requires a `lisp` program somewhere in your path, possibly sbcl

Backtraces if any (M-x toggle-debug-on-error)

N/A

Describe versions of Helm, Emacs, operating system, etc.

MELPA helm-20190627.758
A reasonably fresh Emacs 27
Windows 10

Are you using emacs-helm.sh to reproduce this bug (yes/no):

yes

[thierryvolpiatto]

Thanks to report, I will install sly and try to understand what's going on ASAP.

[thierryvolpiatto]

Ok, I think I found what's wrong.
What you expect is after starting Sly TAB trigger helm completion when helm-mode is enabled, right?
It is not working because you override locally the value of completion-in-region-function, if you don't set it, helm completion is working fine.
You have to comment the line number 340 in sly-completion.el and eval the function BEFORE starting the REPL. See screenshot below.

[thierryvolpiatto]

OTH.

[thierryvolpiatto]

Also I noticed that when I hit "," from the REPL I have an ido completion, you need completing-read as value for sly-completing-read-function to have helm completion. A helm user wanting ido completion somewhere have just to customize helm-completing-read-handlers-alist for the needed commands.

[joaotavora]

OTH

"Off the hook" Lol :-) OK mostly. I think there will be some complications somehow somewhere, I'll test with your advice.

Also I noticed that when I hit "," from the REPL I have an ido completion, you need completing-read as value for sly-completing-read-function to have helm completion. A helm user wanting ido completion somewhere have just to customize helm-completing-read-handlers-alist for the needed commands.

That's a different problem, but thanks for reminding me.

[joaotavora]

think there will be some complications somehow somewhere, I'll test with your advice.

As I feared, a deeper problem persists. First of all, thanks for promptly installing SLY and reproducing this, it is an effort not every package maintainer will make.

Now, the root problem is that Helm is caching whatever completions it gets from the completion-at-point function (capf). In the scenario where capf is and non-prefix is used, the completion UI cannot do that. If you have access to it, I would refer you to how company-mode deals with this.

But even before that, you can try this out: notice that once you complete "sly", you get a completion list of, say, 100 symbols. If you add a letter to "sly", such as "slyt" you now get 73 symbols. Fine so far. Delete the "t", you get the 100 again, now delete the "y" and notice that you stay at the very same 100, while you should now have other symbols at your disposal. (I understand the 100 is somehow a default cap in Helm, but it doesn't change the problem: the symbol list isn't refreshed).

In other words, the Lisp process needs to be contacted every time a character is added or removed from the pattern string. It is only in the reasonably simple "prefix"-matching situation that caching and client-side matching can be used. But even so, Helm must either get the full completion list upfront, or contact the process every time the pattern is widened.

Being such a popular package, surely there is some similar use-case in say, https://github.com/clojure-emacs/cider.

I hope I have managed to explain myself.

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

think there will be some complications somehow somewhere, I'll test with your advice. As I feared, a deeper problem persists. First of all, thanks for promptly installing SLY and reproducing this, it is an effort not every package maintainer will make. Now, the root problem is that Helm is caching whatever completions it gets from the completion-at-point function (capf). In the scenario where capf is and non-prefix is used, the completion UI cannot do that. If you have access to it, I would refer you to how company-mode deals with this.

Company-mode doesn't have to deal with this as it uses as input the text at point in current-buffer i.e. it have no minibuffer. The question is what is completion? Is it completing what is under cursor in current-buffer or is it completing what we have in minibuffer? Helm actually is doing the former, in our example it gives the completion on "sly-" and only "sly-" until you exit helm and change the text at point. You can try in any elisp buffer with e.g. "def" as text at point, you will have completion against "def" and nothing else until you change what is at point.

But even before that, you can try this out: notice that once you complete "sly", you get a completion list of, say, 100 symbols. If you add a letter to "sly", such as "slyt" you now get 73 symbols. Fine so far. Delete the "t", you get the 100 again, now delete the "y" and notice that you stay at the very same 100, while you should now have other symbols at your disposal. (I understand the 100 is somehow a default cap in Helm, but it doesn't change the problem: the symbol list isn't refreshed).

Yes, it is same as described above.

In other words, the Lisp process needs to be contacted every time a character is added or removed from the pattern string. It is only in the reasonably simple "prefix"-matching situation that caching and client-side matching can be used. But even so, Helm must either get the full completion list upfront, or contact the process every time the pattern is widened.

As said above this is questionable. Actually we have once in Helm to C-g, modify input in curent-buffer and hit TAB again, do we want something else? i.e. Taking in account what we have in minibuffer instead of what is at point in current-buffer once Helm is started.

Being such a popular package, surely there is some similar use-case in say, https://github.com/clojure-emacs/cider.

Depend of the UI they use i.e. a minibuffer or not.

…

I hope I have managed to explain myself. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.*

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

What may help is quitting helm as soon as minibuffer-contents length is < to initial input which would allow user modify its input and hit TAB again when done.

[joaotavora]

The question is what is completion? Is it completing what is under cursor in current-buffer or is it completing what we have in minibuffer?

So Helm assumes that whatever text it starts out with in the minibuffer is correct and immutable until the next time you invoke Helm? And likewise the the list of completions corresponding to that text. Is that set of (apparently 100 max by default) the largest set of completions that will ever be attained in that particular Helm invocation/session?

If what I'm asking is against Helm's philosophy, it's perfectly OK. Let me just ask, to clarify. Helm is an "incremental completion and narrowing framework", right? Do you think there there is any way that this framework can be used to present the user the choice of one among, say, 1 million symbols, especially given that those 1 million symbols do not reside in the same memory as the Emacs process?

Both SLY's native interface (very close to Emacs's and the company-mode interface allow this, without forcing the user to exit that interface. Does Helm, too? (notice that this is not the same as saying that 1 million symbols are visible or present in the interface at any given time).

[joaotavora]

What may help is quitting helm as soon as minibuffer-contents length is < to initial input which would allow user modify its input and hit TAB again when done.

But don't you think it's a reasonable expectation by the user that the set of possible completions is enlarged as he backspaces in the Helm interface, just as it does when the user does so after more characters have been entered beyond "initial input"?

[thierryvolpiatto]

At first yes, but after thinking about it no.
Emacs completion-in-region expect completing something against initial input, adding the missing characters to what is at point.
If you have at point "sly" completion-in-region will complete "sly" with anything else starting with "sly", if you change your mind and change the minibuffer contents to what is at point e.g. "def" and Helm behave as you are asking, completion in region mechanism would be confused and insert e.g. "slydefun".
Remember that what we are talking about is a generic completion i.e. same function should handle sly, elisp completion etc...
So for a generic completion, I am not sure we should do more than what we are doing actually (apart maybe quitting when minibuffer differs from input).
But a specialized package may be created to complete in sly as you are expecting.
What do you think?

[thierryvolpiatto]

Apart this SLY looks really cool!

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

The question is what is completion? Is it completing what is under cursor in current-buffer or is it completing what we have in minibuffer? So Helm assumes that whatever text it starts out with in the minibuffer is correct and immutable until the next time you invoke Helm?

Only in the context of completing text at point.

And likewise the the list of completions corresponding to that text. Is that set of (apparently 100 max by default) the largest set of completions that will ever be attained in that particular Helm invocation/session?

No, see `helm-candidate-number-limit`.

If what I'm asking is against Helm's philosophy, it's perfectly OK. Let me just ask, to clarify. Helm is an "incremental completion and narrowing framework", right?

Yes, but what we are speaking about here is generic completion which limits capabilities.

Do you think there there is any way that this framework can be used to present the user the choice of one among, say, 1 million symbols, especially given that those 1 million symbols do not reside in the same memory as the Emacs process?

Yes, but Helm will compute only the `helm-candidate-number-limit` of them matching helm-pattern.

Both SLY's native interface (very close to Emacs's and the company-mode interface allow this, without forcing the user to exit that interface.

As I said because there is no minibuffer involved. The interface is not accessed because your cursor is still in your buffer, not in a minibuffer. company-mode is an auto-complete interface, while Emacs native completion is fully manual (needs to hit TAB at each time you change input) and Helm is partially auto completing as display change as you add characters to original input. The difference with Helm and company is that Helm takes the initial input and compute candidates against it, while company compute the initial input in real time as it changes.

Does Helm, too? (notice that this is not the same as saying that 1 million symbols are visible or present in the interface at any given time).

Completing against million symbols ? Yes.

…

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

Completing against million symbols ? Yes.

But to do so without exiting the Helm interface, you need to fetch the million symbols upfront, yes?

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

Completing against million symbols ? Yes. But to do so without exiting the Helm interface, you need to fetch the million symbols upfront, yes?

Not if you complete against the prefix, which is completion, I have the impression you want to compute all candidates like if input were empty string.

…

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

Not if you complete against the prefix, which is completion, I have the impression you want to compute all candidates like if input were empty string.

No, it's very simple: I want to be able to explore the whole domain, the whole set/space of completions, in the following way:

1. Without exiting the completion interface, or whatever you want to call it.
2. Without ever having to compute the whole set upfront.

Prefix or non-prefix does not enter into it. I just found it very surprising that once you invoke helm on abc and then backspace back to ab you stay in the Helm interface but don't see the completions that start with abd. My simple question is: is this by design or am I doing something wrong? If it's by design just say so :) I don't use Helm but perhaps that is what your users expect/or how you think it should behave.

[thierryvolpiatto]

What you are asking (1.) is not completion, so I can't make this happen in Helm as it will no more fit with the requirements of a generic command.
(2.) can be done by dynamically computing candidates with probably a function instead of a plain list of candidates precomputed.

[thierryvolpiatto]

With last Helm, now completion behave like this:

• Prefix is highlighted and read-only
• When trying to delete backward char prefix helm is completing against, a message is sent on first hit on DEL and second hit delete the last char of what you have at point in current-buffer quitting helm at the same time, letting user correct its initial input and hitting TAB again to start a new completion.

[joaotavora]

What you are asking (1.) is not completion

In your personal view of what completion is. company behaves exactly like this. Is company a completion engine or is it something else? Maybe it's a question that doesn't need an answer: let's call it a different UI.

with probably a function instead of a plain list of candidates precomputed

Great idea. But what do you mean "probably"? In fact that idea has already been had, and, what a coincidence ;-) company uses it!

With last Helm, now completion behave like this:

I think these are improvements on the current situation. I don't understand the use case of letting the user delete the initial pattern and suggesting he press TAB. In my opinion it should either be totally forbidden, or the re-fetch should be automatic. Don't you think this is a simpler interface?

Anyway remember that you shouldn't be doing these improvements motivated by Sly, which, for the time being now, uses either its own UI or company. Think about what your current users would expect (i'm not one of them ;-) )

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

What you are asking (1.) is not completion In your personal view of what completion is. company behaves exactly like this.

No, as explained before company behave the same and emacs vanilla completion too, the only difference is they have NO minibuffer, i.e. the input is fetched directly from what the user write in current buffer whereas helm is taking this same input and put it in minibuffer to provide completion.

Is company a completion engine or is it something else? Maybe it's a question that doesn't need an answer: let's call it a different UI.

Company, auto-complete => auto complete engine. Helm => semi auto complete engine. Emacs vanilla => Manual completion.

with probably a function instead of a plain list of candidates precomputed Great idea. But what do you mean "probably"? In fact that idea has already been had, and, what a coincidence ;-) company uses it!

Helm could also do this but for a specialized command to complete dynamically, here we are speaking of generic completion for helm-mode.

With last Helm, now completion behave like this: I think these are improvements on the current situation. I don't understand the use case of letting the user delete the initial pattern and suggesting he press TAB. In my opinion it should either be totally forbidden, or the re-fetch should be automatic. Don't you think this is a simpler interface?

You have yo understand that Helm is not company or auto-complete, (see above comparisons) and I could write something doing what you are asking but NOT in the context of a generic function.

Anyway remember that you shouldn't be doing these improvements motivated by Sly, which, for the time being now, uses either its own UI or company. Think about what your current users would expect (i'm not one of them ;-) )

Of course, I am not going to change the actual behavior which is IMO the right thing, eshell completion behave also like this among others and is happily used by helm users, helm-slime completion behave also like this. I suggest someone write a helm package for sly behaving like you expect if you think it is the right think to do. To resume the situation for the initial purpose of this bug report: To have a generic completion using Helm interface (using completion-at-point) all you have to do is not using the sly function for completion-in-region. To have dynamic completion with Helm you have to write a specialized package providing a new command for TAB replacing completion-at-point.

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.*

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

No, as explained before company behave the same and emacs vanilla completion too, the only difference is they have NO minibuffer, i.e. the input is fetched directly from what the user write in current buffer whereas helm is taking this same input and put it in minibuffer to provide completion.

Helm is taking this same input and put it in minibuffer to provide completion.

I don't understand why it makes a difference where typing happens. The end goal for the user is to complete the thing he started, presumably with as few keystrokes as possible, with as much visual feedback as possible, and as little visual intrusion as possible. Different UI's have different approaches. Helm happens uses the minibuffer, fine by me.

I also don't understand what semiauto/auto/manual mean in your case. Certainly none of these UI engines complete something "automatically", i.e. with some kind of IA engine. They all show behave slightly differently on certain keystrokes, and show information in different areas of the screen.

Helm could also do this but for a specialized command to complete dynamically, here we are speaking of generic completion for helm-mode.

A helm-sly (or company-sly, or x-sly) would be a waste of effort. I have another LSP extension, eglot that uses the same concepts, I'm not going to repeat that effort, and I don't think anyone should have to. You (and I) should be coding to a common interface, not to a specific backend/frontend.

The good news is that interface already exists, see info node 20.6.7 Programmed Completion. The bad news is it not the best (for historical/backward compatibility reasons), and not wonderfully documented either. I'm frequently in touch with Stefan Monnier about this. Let's bother him here again, @monnier. See also the variable completion-styles, it's another part of the interface that frontends should respect.

Now SLY doesn't yet respect these interfaces fully (and neither does company-capf by the way). But it respects parts of it and I am improving support for it. If Helm were to respect it a bit more, it would work with SLY and other backends that respect it.

To have dynamic completion with Helm you have to write a specialized package providing a new command for TAB replacing completion-at-point.

No, this is precisely what I want to avoid. No speciazlied nothings, life is too short.

[joaotavora]

Now SLY doesn't yet respect these interfaces fully

I've done some work in a side branch (https://github.com/joaotavora/sly/tree/scratch/fix-214-rework-completion) and it should be more compliant now. To illustrate (and to prove that it really does work) here are 4 scenarios:

(1) basic completion (what you call "manual" completion).

(2) company (what you call "fully auto")

(3) icomplete (an ido.el replacement in emacs core)

(4) SLY's own UI

(5) Helm, not quite working

[thierryvolpiatto]

You can try with this branch https://github.com/emacs-helm/helm/tree/completion_in_region_dynamic.
Not sure it works everywhere.

[joaotavora]

You can try with this branch https://github.com/emacs-helm/helm/tree/completion_in_region_dynamic.
Not sure it works everywhere.

Thanks very much for attempting this. Working properly with Helm is a big plus for any package.

Now, I tried it, but it didn't work. Perhaps I'm not testing correctly. I checked out your branch, git clean -fdx, make EMACS_COMMAND=..., then ./emacs-helm and did basically what I do in the last .gif. I get the same behaviour except, in the completion-at-point case, it doesn't even get to the Helm interface.

How are you testing?

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

You can try with this branch https://github.com/emacs-helm/helm/tree/completion_in_region_dynamic. Not sure it works everywhere. Thanks very much for attempting this. Working properly with Helm is a big plus for any package. Now, I tried it, but it didn't work. Perhaps I'm not testing correctly. I checked out your branch, git clean -fdx, make EMACS_COMMAND=..., then ./emacs-helm and did basically what I do in the last .gif. I get the same behaviour except, in the completion-at-point case, it doesn't even get to the Helm interface. How are you testing?

Tested with completion-at-point in emacs-lisp buffers and completing-read-multiple, these are working. With Sly I only have [no matches] for now, don't know why it is not working here.

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.*

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

Tested with completion-at-point in emacs-lisp buffers and completing-read-multiple, these are working.

That's a good start. I will debug SLY a bit today.

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

Tested with completion-at-point in emacs-lisp buffers and completing-read-multiple, these are working. That's a good start. I will debug SLY a bit today.

I could get it working with Sly by passing explicitely pattern to dynamic function, but this is defeating helm matching functions (I can workaround multimatch but fuzzy is broken). I noticed also that flexmatching is not working with helm whereas it is working with sly-symbol-completion-mode.

…

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

As I expected, trying to pass pattern (minibuffer-contents) to a dynamic function is a regression in Helm because it defeat all helm matching functions. In the example you show above e.g. icomplete, it is simple because the completion is basic, and helm is working fine like this, but as soon as you enter a more complex pattern, the function doesn't understand it, we would need a super function handling alone all the helm features, which is not possible. So with the actual behavior of the Master branch, I think we have a good compromise, loosing only the ability of changing on the fly the initial input, but well when you start to complete on e.g. "def" I hardly see why you would change your mind as soon you hit TAB to complete on something else.
Below a screenshot illustrating pattern than can be used with helm:

[monnier]

> a `flex-no-space` completion-style or a `helm-flex` completion style. I understand the usefulness of the first, but not the second.

The second could be tailor-made to mix the two styles in any way we see fit. I haven't thought about it in detail, so maybe it wouldn't be any better than just combining `flex-no-space` and `helm`. Stefan

[joaotavora]

I think flex-no-space is the way to go here. Or just advising users that mixing and matching styles doesn't always work as expected.

On a tangent, I'm quite suprised helm doesn't provide a flex style itself. I assumed it provides many modern and sophisticated things.

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

I think flex-no-space is the way to go here. Or just advising users that mixing and matching styles doesn't always work as expected.

Hmm, no I think allowing mixing styles (perhaps only some styles) is very interesting, it's what I do in Helm since long time now (multi matching and fuzzy).

On a tangent, I'm quite suprised helm doesn't provide a flex style itself.

It does, it is called fuzzy matching, I have a branch with a fuzzy style, but I would prefer using flex in the future, at least for completion, the flex algo seems better than the helm one (distance between matches vs contiguous matches), and also it comes for free :-), thanks for this!

…

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

monnier <notifications@github.com> writes:

>> a `flex-no-space` completion-style or a `helm-flex` completion style. > I understand the usefulness of the first, but not the second. The second could be tailor-made to mix the two styles in any way we see fit. I haven't thought about it in detail, so maybe it wouldn't be any better than just combining `flex-no-space` and `helm`.

Yes `flex-no-space` and `helm` is what I would like to have, would be perfect.

…

Stefan — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.*

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

On Thu, Nov 21, 2019 at 5:33 PM Thierry Volpiatto ***@***.***> wrote: monnier ***@***.***> writes: >>> a `flex-no-space` completion-style or a `helm-flex` completion style. >> I understand the usefulness of the first, but not the second. > > The second could be tailor-made to mix the two styles in any way we see > fit. I haven't thought about it in detail, so maybe it wouldn't be any > better than just combining `flex-no-space` and `helm`. Yes `flex-no-space` and `helm` is what I would like to have, would be perfect.

Thierry, presuming you already have some experience with completion styles, you are welcome to make this change to minibuffer.el. I'm a bit busy right now (you probably are too, but just in case you aren't). João

[thierryvolpiatto]

monnier <notifications@github.com> writes:

>> So as explained above the candidates are buffers like *helm foo* and >> helm-foo.el, if I enter as pattern "helm" flex matches both buffers but if >> the pattern is "helm " it matches only *helm foo* (as expected), I had >> expected that as soon as I enter the space after "helm", flex stop matching, >> i.e. returns nil and let helm style kicking in. > Yup, I think this makes sense. But let's ask @monnier and @dgutov (amd more people?) In the case where `flex` is not followed by `helm`, I think it's good that flex doesn't stop matching just because the user typed a space.

Yes.

But indeed when it's followed by `helm`, maybe it would make sense for `flex` to delegate to helm in such a case.

Yes, what I want.

So I think this basically argues for the introduction of either a `flex-no-space` completion-style or a `helm-flex` completion style.

Maybe not making a new style, but just having a variable to tell flex to fail as soon at it sees a space in pattern would be good, it would be easy for helm to let-bind it.

…

Stefan — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.*

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

On Thu, Nov 21, 2019 at 5:33 PM Thierry Volpiatto ***@***.***> wrote: > > monnier ***@***.***> writes: > > >>> a `flex-no-space` completion-style or a `helm-flex` completion style. > >> I understand the usefulness of the first, but not the second. > > > > The second could be tailor-made to mix the two styles in any way we see > > fit. I haven't thought about it in detail, so maybe it wouldn't be any > > better than just combining `flex-no-space` and `helm`. > > Yes `flex-no-space` and `helm` is what I would like to have, would be > perfect. Thierry, presuming you already have some experience with completion styles, you are welcome to make this change to minibuffer.el. I'm a bit busy right now (you probably are too, but just in case you aren't).

Is this ok for you: diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el index 6e72eb73f99..ba9283e87b0 100644 --- a/lisp/minibuffer.el +++ b/lisp/minibuffer.el @@ -3494,6 +3494,11 @@ that is non-nil." ;;; "flex" completion, also known as flx/fuzzy/scatter completion ;; Completes "foo" to "frodo" and "farfromsober" +(defcustom completion-flex-nospace nil + "Make flex style fails when a space is found in pattern." + :version "27.1" + :type 'boolean) + (put 'flex 'completion--adjust-metadata 'completion--flex-adjust-metadata) (defun completion--flex-adjust-metadata (metadata) @@ -3539,29 +3544,31 @@ which is at the core of flex logic. The extra (defun completion-flex-try-completion (string table pred point) "Try to flex-complete STRING in TABLE given PRED and POINT." - (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds) - (completion-substring--all-completions - string table pred point - #'completion-flex--make-flex-pattern))) - (if minibuffer-completing-file-name - (setq all (completion-pcm--filename-try-filter all))) - ;; Try some "merging", meaning add as much as possible to the - ;; user's pattern without losing any possible matches in `all'. - ;; i.e this will augment "cfi" to "config" if all candidates - ;; contain the substring "config". FIXME: this still won't - ;; augment "foo" to "froo" when matching "frodo" and - ;; "farfromsober". - (completion-pcm--merge-try pattern all prefix suffix))) + (unless (and completion-flex-nospace (string-match-p " " string)) + (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds) + (completion-substring--all-completions + string table pred point + #'completion-flex--make-flex-pattern))) + (if minibuffer-completing-file-name + (setq all (completion-pcm--filename-try-filter all))) + ;; Try some "merging", meaning add as much as possible to the + ;; user's pattern without losing any possible matches in `all'. + ;; i.e this will augment "cfi" to "config" if all candidates + ;; contain the substring "config". FIXME: this still won't + ;; augment "foo" to "froo" when matching "frodo" and + ;; "farfromsober". + (completion-pcm--merge-try pattern all prefix suffix)))) (defun completion-flex-all-completions (string table pred point) "Get flex-completions of STRING in TABLE, given PRED and POINT." - (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds) - (completion-substring--all-completions - string table pred point - #'completion-flex--make-flex-pattern))) - (when all - (nconc (completion-pcm--hilit-commonality pattern all) - (length prefix))))) + (unless (and completion-flex-nospace (string-match-p " " string)) + (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds) + (completion-substring--all-completions + string table pred point + #'completion-flex--make-flex-pattern))) + (when all + (nconc (completion-pcm--hilit-commonality pattern all) + (length prefix)))))) ;; Initials completion ;; Complete /ums to /usr/monnier/src or lch to list-command-history.

…

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

On Thu, Nov 21, 2019 at 5:33 PM Thierry Volpiatto ***@***.***> wrote: > > monnier ***@***.***> writes: > > >>> a `flex-no-space` completion-style or a `helm-flex` completion style. > >> I understand the usefulness of the first, but not the second. > > > > The second could be tailor-made to mix the two styles in any way we see > > fit. I haven't thought about it in detail, so maybe it wouldn't be any > > better than just combining `flex-no-space` and `helm`. > > Yes `flex-no-space` and `helm` is what I would like to have, would be > perfect. Thierry, presuming you already have some experience with completion styles, you are welcome to make this change to minibuffer.el. I'm a bit busy right now (you probably are too, but just in case you aren't).

Ok pushed, let me know if something seems wrong, thanks.

…

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

So now with the use of the new variable completion-flex-nospace in helm (it is let bounded) helm and flex work beautifully with e.g. switch-to-buffer :-).

[joaotavora]

@thierryvolpiatto thanks. It's a decent approach, but it still doesn't solve the use case where I don't the helm interface but still want to use some helm matching styles, right?

This would require an actual flex-nospace style. However, and since you already pushed this, I think a good compromise is to keep completion-flex-nospace but default it to t. I think most people don't expect flex to match spaces anyway.

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

@thierryvolpiatto thanks. It's a decent approach, but it still doesn't solve the use case where I don't the helm interface but still want to use some helm matching styles, right?

Of course, this is an unrelated issue. The problem is that there is no UI in emacs to use such pattern (AFAIK).

This would require an actual flex-nospace style.

Making a new style, thus nearly the same as flex just for this?

However, and since you already pushed this,

It's easy to write the new style by reusing the try/all flex completion function with completion-flex-nospace around (probably defcustom for completion-flex-nospace is not needed in this case).

I think a good compromise is to keep completion-flex-nospace but default it to t. I think most people don't expect flex to match spaces anyway.

Yes.

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.*

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

Thierry Volpiatto <notifications@github.com> writes:

João Távora ***@***.***> writes: > @thierryvolpiatto thanks. It's a decent approach, but it still doesn't solve the use case where I don't the helm interface but still want to use some helm matching > styles, right? Of course, this is an unrelated issue. The problem is that there is no UI in emacs to use such pattern (AFAIK).

Hmm? Really? I can't use your Helm style with icomplete or Company or bare completion-at-point? I was under the impression that I could. Or do you mean that I can but it's not very useful without the UI to go with it?

> This would require an actual flex-nospace style. Making a new style, thus nearly the same as flex just for this?

I don't see the problem, it's not like a style takes up much space (the implementation would be shared 100%). But indeed, if you say that helm styles are Helm-UI-only (a shame in my opinion), then it's not needed.

> However, and since you already pushed this, It's easy to write the new style by reusing the try/all flex completion function with completion-flex-nospace around (probably defcustom for completion-flex-nospace is not needed in this case). > I think a good compromise is to keep completion-flex-nospace but > default it to t. I think most people don't expect flex to match spaces > anyway. Yes.

Good. I'll do that change. João

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

Thierry Volpiatto ***@***.***> writes: > João Távora ***@***.***> writes: > >> @thierryvolpiatto thanks. It's a decent approach, but it still doesn't solve the use case where I don't the helm interface but still want to use some helm matching >> styles, right? > > Of course, this is an unrelated issue. The problem is that there is no > UI in emacs to use such pattern (AFAIK). Hmm? Really? I can't use your Helm style with icomplete or Company or bare completion-at-point?

icomplete: Maybe if you have the ability to enter spaces, last time I tried I couldn't. company: No. completion-at-point: Surely No.

I was under the impression that I could. Or do you mean that I can but it's not very useful without the UI to go with it?

No you can't, well helm style will kick in with the prefix but you will not be able to do much more.

>> This would require an actual flex-nospace style. > > Making a new style, thus nearly the same as flex just for this? I don't see the problem, it's not like a style takes up much space (the implementation would be shared 100%). But indeed, if you say that helm styles are Helm-UI-only (a shame in my opinion)

The shame is that there is no one UI in emacs vanilla able to handle this, what can I do on helm side? nothing. Even if you are not a fan of helm, perhaps you should give it a try to understand better what helm is able to do and what it is not able to do.

Good. I'll do that change.

Ok, thanks.

…

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

The shame is that there is no one UI in emacs vanilla able to handle
this, what can I do on helm side? nothing.

Makes sense. But you can explain exactly what the minimal requirements of such an UI is. To be clear, an UI where the styles provided by Helm make sense (and why it is a good thing).

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

The shame is that there is no one UI in emacs vanilla able to handle this, what can I do on helm side? nothing. Makes sense. But you can explain exactly what the minimal requirements of such an UI is.

Being able to enter a pattern with spaces at prompt, and yes you need a prompt even for completion-at-point i.e. the text at point should be inserted into a prompt for further edit, but as I said the best is using helm in various situations to have a better idea.

…

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[monnier]

> Hmm? Really? I can't use your Helm style with icomplete or Company or > bare completion-at-point? icomplete: Maybe if you have the ability to enter spaces, last time I tried I couldn't.

icomplete shouldn't have any effect on whether or not you can enter a space. But if you can't enter a space when `helm` completion style is in use then it's probably a bug (but likely a bug in `minibuffer-complete-word` or `helm` style rather than in `icomplete`).

company: No. completion-at-point: Surely No.

I understand that in-buffer completion is likely not to work when the intended input pattern includes a space, since the capf function will probably be confused by the space unless we're completing against things that can indeed contain spaces. And of course the same can happen for helm patterns that don't use spaces but use regexp operators. The same problem plagues the `partial-completion` style which allows you to use the `*` character as an explicit glob pattern but when used for completion-at-point in many cases the `capf` only considers "letters" in the input as being part of the input pattern. IIUC Helm solves this problem by making it so you first activate it, then enter the pattern (this part takes place in the minibuffer), so there's no ambiguity about the special chars (space or other) being part of the input pattern. This requires having a notion of entering and leaving a completion-in-progress state. `completion-at-point` does have such a notion, but it's defined to leave this state in a completely automated/implicit way (by calling capf to check if we're still within the same completable element), so we'd have to change it to make this "end of completion" a bit less eager. BTW for minibuffer completion, I can't see any reason why `helm` style can't be fully used without using Helm itself. Am I missing something? Stefan

[thierryvolpiatto]

monnier <notifications@github.com> writes:

>> Hmm? Really? I can't use your Helm style with icomplete or Company or >> bare completion-at-point? > icomplete: Maybe if you have the ability to enter spaces, last time I > tried I couldn't. icomplete shouldn't have any effect on whether or not you can enter a space.

Seems it is possible to enter a space for find-file when icomplete-mode is active, but not for M-x, don't know what icomplete handle else.

But if you can't enter a space when `helm` completion style is in use then it's probably a bug (but likely a bug in `minibuffer-complete-word` or `helm` style rather than in `icomplete`).

May be, I tried to enter a space with find-file displayed in icomplete and helm style didn't kick in, don't know why.

> company: No. > completion-at-point: Surely No. I understand that in-buffer completion is likely not to work when the intended input pattern includes a space, since the capf function will probably be confused by the space unless we're completing against things that can indeed contain spaces. And of course the same can happen for helm patterns that don't use spaces but use regexp operators.

No matter spaces or not spaces, helm use regexp, always except when fuzzy matching of course (but forget fuzzy, let focus on what helm style does).

The same problem plagues the `partial-completion` style which allows you to use the `*` character as an explicit glob pattern but when used for completion-at-point in many cases the `capf` only considers "letters" in the input as being part of the input pattern. IIUC Helm solves this problem by making it so you first activate it, then enter the pattern (this part takes place in the minibuffer),

The initial pattern taken from text at point, extracted from START and END is first inserted as initial input and then user can refine it as needed.

so there's no ambiguity about the special chars (space or other) being part of the input pattern.

Yes.

This requires having a notion of entering and leaving a completion-in-progress state.

This is transparent, no notion is needed.

`completion-at-point` does have such a notion, but it's defined to leave this state in a completely automated/implicit way (by calling capf to check if we're still within the same completable element), so we'd have to change it to make this "end of completion" a bit less eager. BTW for minibuffer completion, I can't see any reason why `helm` style can't be fully used without using Helm itself. Am I missing something?

The problem is the same here, minibuffer is just like any other buffer, when you press TAB, completion takes what it has at point and complete it, you have no chance to add something else. Would be great you give a try to helm to see what is does exactly.

…

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

@joaotavora @monnier , does it make a difference for you if using add-face-text-property instead of put-text-property for setting completions-first-difference in completion-pcm--hilit-commonality?
The problem is this face (completely usefulness in helm) is overriding the possible face actually in use in helm display like actually in helm-org:

but after applying patch below using add-face-text-property leading star appear fontified properly:

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 779c3c88ae8..2cef08fd9f0 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -3180,9 +3180,10 @@ one-letter-long matches).")
                               'face 'completions-common-part
                               str)
            (if (> (length str) pos)
-               (put-text-property pos (1+ pos)
-                                  'face 'completions-first-difference
-                                  str))
+               (add-face-text-property
+                pos (1+ pos)
+                'face 'completions-first-difference
+                str))
            (unless (zerop (length str))
              (put-text-property
               0 1 'completion-score

More generally I wonder why you are bothering showing this in completions...

[joaotavora]

I think the patch makes sense: using put-text-property there is a bug, I think. (did I add it?)

More generally I wonder why you are bothering showing this in completions...

I agree, and there's a long long bikeshedding thread about that that didn't end on particularly happy note.

[thierryvolpiatto]

João Távora <notifications@github.com> writes:

I think the patch makes sense: using put-text-property there is a bug, I think.

It is also used for `completions-common-part` may be we should use `add-face-text-property` as well here?

(did I add it?)

Not sure, but it seems you only merge the scoring code inside this function.

More generally I wonder why you are bothering showing this in completions... I agree, and there's a long long bikeshedding thread about that that didn't end on particularly happy note.

Ah, I see... Also a lot of code handle prefix+suffix where most of the time suffix is irrelevant, I will add a note on this later.

…

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

It is also used for completions-common-part may be we should use
add-face-text-property as well here?

Probably. yes.

Not sure, but it seems you only merge the scoring code inside this function.

That's my perception as well, which means you should also quickly ping Stefan about this change.

most of the time suffix is irrelevant,

I think Stefan and some other users of completion will modify "most of the times" to "sometimes". IOW that code is there for reasons.

[monnier]

@joaotavora @monnier , does it make a difference for you if using `add-face-text-property` instead of `put-text-property` for setting `completions-first-difference` in `completion-pcm--hilit-commonality`?

I don't think it's The Right Solution to this problem (a variation of which manifests itself in Company, where they want to highlight the same things as in *Completions* but they want to use different faces for that for reasons for color-schemes), but I have no objection to such a workaround, tho please see below.

The problem is this face (completely usefulness in helm)

[ Side note: I disagree that it's completely useless in Helm; you (and maybe even most users of Helm) may not care for it, but there are people who might find it helpful. ]

- (put-text-property pos (1+ pos) - 'face 'completions-first-difference - str)) + (add-face-text-property + pos (1+ pos) + 'face 'completions-first-difference + str))

This will append the face `face` rather than the face `completions-first-difference`, so it will break existing uses. I believe you should use either (add-face-text-property pos (1+ pos) 'completions-first-difference nil str)) or (add-face-text-property pos (1+ pos) 'completions-first-difference 'append str)) depending on whether you want to append/prepend the new face (I presume you want to `append` since you fundamentally want to demote it; tho prepending would be a safer change). Not sure if this fix will still result in the behavior you want, or will re-introduce the problem (depends on the other face you're using), so buyer beware.

More generally I wonder why you are bothering showing this in completions...

For prefix style completions, it's often more useful than showing the "common-part" (which is already super-evident without any highlighting and which many people would prefer to "hide" more than to highlight). Stefan

[thierryvolpiatto]

monnier <notifications@github.com> writes:

> @joaotavora @monnier , does it make a difference for you if using > `add-face-text-property` instead of `put-text-property` for setting > `completions-first-difference` in `completion-pcm--hilit-commonality`? I don't think it's The Right Solution to this problem (a variation of which manifests itself in Company, where they want to highlight the same things as in *Completions* but they want to use different faces for that for reasons for color-schemes), but I have no objection to such a workaround, tho please see below. > + (add-face-text-property > + pos (1+ pos) > + 'face 'completions-first-difference > + str)) This will append the face `face` rather than the face `completions-first-difference`, so it will break existing uses.

Yes of course, my bad.

I believe you should use either (add-face-text-property pos (1+ pos) 'completions-first-difference nil str)) or (add-face-text-property pos (1+ pos) 'completions-first-difference 'append str)) depending on whether you want to append/prepend the new face (I presume you want to `append` since you fundamentally want to demote it; tho prepending would be a safer change).

Ok, I used prepending it works fine with emacs -Q (M-x do-vi TAB TAB) and helm. Thanks.

…

-- Thierry Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Pushed to master.
With emacs -Q:

[thierryvolpiatto]

Closing now this long thread, many thanks to all for their help, specially to @joaotavora and @monnier .
Feel free to open a new issue if needed.