need a way to get an environment from a callback

[tromey]

For emacs-ffi I need a way to get an environment from a callback that doesn't have any obvious connection to Lisp. There doesn't seem to be any way to do this currently; the runtime passed to my init function is destroyed when my function returns, so there's no way to stash either a runtime or an environment for later use.

Without this, ffi closure support can't work.

[phst]

The only possible ways how module code can be called are the initialization functions and functions registered with make_function. All such functions receive an environment pointer, which you can use while the corresponding functions are running. Can you given a simple example for a call sequence that is not covered by this model?

[phst]

OK, I took a very quick look at the emacs-ffi code. Seems the only issue is how to correctly pass the environment pointer around. I'd suggest the following: Remove the env member from closure_description, and make it mandatory that the first argument to each closure is a valid env pointer. Would that work?

[tromey]

It's not possible to require a closure to be called in any particular way. They are generally called by random C library code -- they are callbacks.

It's true that it is hard to see how this situation could arise. You have to sneak something into the Emacs event loop. But, I think it can be done. For example, using ffi, I could make a new Gtk window and attach some Lisp function as an event handler. I think perhaps this window would participate in the existing event loop. I haven't tried this though.

[phst]

You could store active environment pointers in a thread-local stack. However you then really have to make sure that any environment function is (indirectly) called from Emacs. Anything else is outside of Emacs's execution model and is definitely undefined behavior.

[tromey]

If the callback happens "spontaneously" from the emacs point of view, then there is no way to stash the pointer.

[phst]

There is no place in the Emacs execution model for "spontaneous" callbacks: code is executed as a result of input events, timers, process sentinels, etc., but never "spontaneously". Allowing such callbacks would imply a significant change in the Emacs execution model, probably requiring significant time for discussion. This will certainly not happen before the feature freeze for 25.1.

[tromey]

It can happen if one uses FFI to make a new window, then attaches Lisp callbacks to that window. This isn't all that far-fetched. The solution is simple: keep a runtime alive indefinitely (which I understood to be the initial design anyway); and let code request (and destroy) an environment from the runtime.

[tromey]

Another spot having a similar issue is free_closure_desc in the ffi. This is a free function attached to a user-pointer that wants to dereference some global references.

[aaptel]

@tromey while certainly useful, can it be done safely? I have to look more into it but as Phillip said it seems unsafe, you would run into race conditions in the Emacs core.

[aaptel]

Can we come up with a safe subset of ELisp that is safe (yet useful) to use in callbacks. As long as it's documented I don't see a problem (seems tedious tho).

[tromey]

I think the dangerous situations are running elisp in other threads or from signal handlers. From callbacks it should be ok.

[phst]

This issue is more complex than the request implies. If you create a window (in the Gtk sense) in the Emacs process, you do something that is "impossible": normally only Emacs can create windows, and all windows are under Emacs's control. It would need to be carefully considered what the meaning of such an "impossible" operation should be. The typical answer (in C) would be undefined behavior (similar to introducing a data race), and I'm inclined to apply this here as well: if you do something that is observable from Emacs but is impossible in Lisp (as described in the Emacs Lisp manual), the behavior is undefined. Otherwise this would open a huge can of worms: suddenly you'd have to define what happens if an impossible situation occurs.

[tromey]

I think you are overstating the case. It's easy to define what happens, provided (1) it happens on the main thread, and (2) it doesn't happen from a signal handler. It's no different from a process filter.

[phst]

Some discussion happened on emacs-devel. Eli's answer was

In the future, we might add
facilities that would allow a module register a file descriptor
through which events could flow into the Emacs event queue. But that
is not yet available, and I'm not sure we'd want to allow it.

Closing this for now as it would require more discussion and most likely won't happen in the near future.

[aaptel]

Well it's settled for now then.