Plans?

[alphapapa]

Hi,

I just stumbled across this a few days ago... and don't remember how. :) Anyway, I see that you're still working on it. I just wondered, what are your plans in the mid-long term? Thanks. :)

[Fuco1]

This project exists mostly only because I don't like helm very much. It's a fine project, works most of the time, but is too bloated with nonsense and too many things were hardcored/made to work for the author (which is nothing wrong, I'm doing the same).

So my plans are mostly to port over features I find useful while keeping the architecture as flexible as possible to allow future interfaces to be built on top of this.

I'm not planning to make this "huge", but if it attracts attention on its own so be it.

[alphapapa]

Cool, I'll keep an eye on it. I love Helm, and Thierry does a great job on it, but it is indeed large and complex.

I'd be interested in trying to make a few add-on commands for it. Do you think it's ready for that, or does it need more time to mature? (Maybe I'm just lazy and don't want to have to fix things if you make breaking changes. ;)

[Fuco1]

The architecture described in https://github.com/Fuco1/sallet/blob/master/dev-notes.org is pretty much fixed and won't change unless something substantial happens. If you want to hack on go ahead :) I'll be happy to answer the questions.

[Fuco1]

Well, I now have to retract my above command. Over the last couple months I've started rewriting this whole thing to be fully concurrent and asynchronous. I'm using emacs-deferred as the backend and it works wonderfully. The user experience is phenomenal, you can keep typing and interact with the prompt while the results keep appearing on the screen.

It will need quite a lot of work but I am very enthusiastic about the resulting UX.

[alphapapa]

Oh, that sounds very interesting! But how does it work for in-process sources? e.g. helm-org-rifle runs on Org buffers that are open in the current process, and spinning up a second Emacs process for every change to the query (as would be required to run it as an async source in Helm) would ruin performance, because all of the Org files would have to be loaded in the external process. And the user's config has to be loaded in the external process too, since Emacs can't just fork a copy of itself.

I'm hoping that with Emacs 26 I can do something with threading, but AFAIK in Emacs 25, async requires external processes, right?

If I'm wrong and missing something, please do tell, because that would be great. :)

[alphapapa]

I looked at concurrent.el (didn't know of it before) and I see that it has some macros that run each sexp with a delay, and I guess that doesn't use an external process. Is that what you're doing? If so, that sounds very interesting. I guess you have to manually cancel the rendering of a source if the query changes before it finishes?

I couldn't find anything related to that in the repo, so I guess you haven't pushed a branch yet. Please let me know when you do so I can study it. :)

[Fuco1]

It works by having a queue of things to do on the idle timer. I employ a hack where I limit each "coroutine" to only run for 10 miliseconds then yield back to the input loop. In case the user inputs anything the queue is cancelled and a new "thread" is spawned where the things update. Because you can't really notice 10 miliseconds delay in typing everything works very smooth and as soon as you are not doing anything to Emacs the performance loss is quite negligible.

I haven't pushed it yet because I have it laying around in different files in different projects :D (I was first trying to do some "yielding cooperation library" but decided against it as it is quite difficult to do in a general enough way given emacs's capabilities.

[alphapapa]

Hm, that sounds very interesting. How do you limit them to run for 10 ms? Just check current-time? And how do you preserve state between invocations of the coroutine?

helm-org-rifle is actually not Helm-only--the main logic is independent of it, and it has a non-Helm, occur-like mode as well. There's one function that returns the results, so if I could return one result at a time from it instead of the whole list, that might work--but then the state would be lost, including current buffer, position, etc. I guess I could move the actual re-search-forward into another function and return one at a time, or just return the position of a result, which would preserve the state between individual calls to re-search-forward, but that still doesn't cover yielding back to the input loop.

Trying to think of a queue system, I guess I could have one item in the queue for each buffer being searched, and store the state for each buffer in the item, along with results that the search function could push to. I guess that would work, but then rendering the results would be a lot more complicated, as I'd have to keep track of which result items had already been rendered, or else they'd all be re-rendered for each result.

Whew...it starts to get a lot more complicated very quickly. I look forward to seeing what you come up with, because if I could use it, it would help a lot. helm-org-rifle does suffer from some delays when searching several large buffers.

[Fuco1]

Closures can preserve state which is quite super useful (like the current buffer etc). For the results and the state I need to feed to further stages I just emit that in the result as a plist. I've built a monad-like interface on top of it (something like State monad in Haskell if you are familiar with it) so it kind of works pretty seamlessly.

For the limiting I just check the current-time yes, it's pretty silly but it works quite well.

I will try to clean it up and put on some branch so you can check it out. But right now writing the sources is rather difficult and case-by-case basis. I also haven't figured out asyncio (external processes) sources yet, but it should be relatively simple too.

[Fuco1]

An example

;; -*- lexical-binding: t -*-

(defun my-make-generator ()
  (let ((search-start 0))
    (lambda ()
      (incf search-start)
      (message "start at %d" search-start))))

(defvar my-generator (my-make-generator))

(funcall my-generator)

eval the last line => magic

Make sure lexical-binding is t in that buffer.

[alphapapa]

Ah, yes, I hadn't thought of using closures. That would be a significant restructuring of my code, but it might be worth it.

One question, though: I know closure-oriented code is a common Lisp pattern (pardon the pun), and I've read some of On Lisp, so I think I have a basic idea of how to use them--but are they performant enough in Elisp compared to byte-compiled, non-closure-oriented code?

Thanks for your guidance here!

[Fuco1]

Well, I'm benchmarking this on a 15k line org file with a helm-occur-like searches such as a which basically matches every line. There is no noticable lag and everything feels smooth. As I keep typing more it just narrows in a very pleasant way.

As for closures, they are hella slow in elisp but in this usecase I think that is not a problem because usually you narrow to a fairly small amount of candidates and if the checks are ordered well enough you can very quickly go through a lot of invalid candidates. Also I'm counting on them becoming more optimized in the future, plus moore's law and all.

[alphapapa]

Ah, I see, so the closure stuff is narrowing a list of candidates that has already been provided? That's different than my use case, which is building the list of candidates rather than narrowing it.

For example, helm-org-in-buffer-headings is rather slow on a large buffer, because it has to get every heading in the buffer first, which is much slower than narrowing the resulting list. But helm-org-rifle is faster because it uses re-search-forward to go directly to potential matches, skipping across headings that don't match. So it doesn't use any of Helm's built-in narrowing or matching facilities (which, now and then, Thierry suggests I rewrite my code to use, but that would defeat the benefits of not using them--not that they aren't great and fast, just not for this use case).

Also I'm counting on them becoming more optimized in the future, plus moore's law and all.

Hmmmmm, you're not a Guile programmer, by any chance, are you? =)

[Fuco1]

The closure can generate or filter or whatever. I actually have 4 stages:

generator => matcher => sorter => renderer

Each of them has its own internal buffer (as a closed over variable) and generates/processes just 10ms worth of stuff every iteration. It is separated so in stages so that in principle you could shell out in each stage if necessary (I've written a very fast implementation of flx in C for example which the matcher could shell out to, even holding the same process between stages so the startup cost is mitigated).

[alphapapa]

Wow, sophisticated. :) Very interesting. I look forward to where Sallet is going!