Feature request: Callbacks into Python

[bpcreech]

I'm writing this down now as an idea for future work. I probably won't be able to get to this soon, but I might later, or someone else might be able to contribute it.

PyMiniRacer runs code in the V8 sandbox. The JS code, by design, has no (intended!) way to reach out and touch the system it runs in. I.e., it cannot make network calls, read and write files, etc. It would be nice if a PyMiniRacer user could write a custom JavaScript API in Python and present it to JS code. For inspiration, the NodeJS standard library is full of such APIs: a bunch of things from network I/O to file I/O to subprocess management, all specific to the Node server and useful for JS code. (Of course, the NodeJS library is implemented in C++ and exposed to JS. Here we're talking about implementing an API in Python and exposing it to JS. Also creating a standard library as extensive as NodeJS's would be an enormous undertaking; this issue is just talking about making it possible to create JS-callable APIs in Python, not about creating any specific "PyMiniRacer standard library".)

Obviously using such a feature would expose the PyMiniRacer user to security concerns if running unstrusted JS code as it would create a breach of the sandbox. However, it could be useful if running trusted code, or if the user is very careful about the APIs they expose. We wouldn't want to include any "PyMiniRacer standard library" by default (unless its contents were totally safe; e.g., we currently ship a setTimeout and clearTimeout which just simplify what one can already do with Atomics.waitAsync, and that's it).

Unfortunately, as of this writing there's no way for JavaScript under PyMiniRacer to call back into Python, and thus no way for a PyMiniRacer user to create a Python-native API and expose it to JS. One could do a hacky solution like polling a "work queue" (e.g., a JS array) from Python and servicing it in Python, but there's no true callback mechanism. The one place PyMiniRacer actually already supports callbacks from JS is Promise objects, which you can create in JS and await in Python. However, those are one-shot by design and thus would be very awkward to convert into a general-purpose callback API.

Here's a sketch of how a general-purpose callback API might work:

async def read_arbitrary_file(name: str) -> str:  # args and return can be any of PythonJSConvertedTypes
    with aiofiles.open(name) as f:
        return await f.read()

# Wrap and install the API:
mr = MiniRacer()
wrapped_api = mr.wrap_api(my_api)
mr.eval("this")["read_arbitrary_file"] = wrapped_api

promise = mr.eval("""
let get_reversed_contents = async (name) => {
    let contents = await read_arbitrary_file(name);  # note the API is async in JS!
    return contents.split("").reverse().join("");
};
get_reversed_contents("/usr/share/dict/words")
""")
print(promise.get()) # print the dictionary backwards!

A design key point is that at least on the JS side, if not also the Python side, the API must be async. This is because callbacks from V8 hold the V8 Isolate lock, so the most we should do in a callback (especially an arbitrary one like proposed here) is to create and return a promise. (We want to avoid the mistake of attempting to re-enter V8 from within the callback. That would deadlock, at least under PyMiniRacer's current design.) We can then fulfill the promise in another thread which doesn't hold the v8 isolate lock. (This is somewhat similar to how the NodeJS standard library works; NodeJS APIs generally either return Promises or take in callback functions. They don't usually give you synchronous results.)

Implementation idea: To make this work, the mr.wrap_api Python API would call C++ code which instantiates and returns generic C++ callback function. That C++ callback function would do nothing other than create a Promise, then call a Python callback with the arguments array plus the Promise resolve+reject functions, then return the Promise to the JS caller. The Python callback would just decode the arguments and resolve+reject and stuff them onto a work queue (probably an asyncio.Queue). Then another loop, probably an asyncio loop (TODO: managed by the PyMiniRacer user? or magically by PyMiniRacer?) would service that queue by popping the args+resolve+reject, calling the user-specified Python callback with the args, and stuffing the return (or exception) into the resolve or reject function. In that way, JS would be immediately triggering work in Python, by way of a asyncio.Queue wakeup, but Python would not do any serious work on V8's message loop thread.

[bpcreech]

Other impl idea:

class CallTeleporter {
    constructor() {
        this.incoming_promises = [Promise.withResolvers()];
    }

    call_api() {
        let { outgoing_result_promise, outgoing_result_resolve, outgoing_result_reject }  = Promise.withResolvers();
        // resolve an incoming-call-promise by telling it about the new waiting API call:
        this.incoming_promises[this.incoming_promises.length-1][1](arguments, outgoing_result_resolve, outgoing_result_reject);
        // set up for next incoming call
        this.incoming_promises.push(Promise.withResolvers());
        return outgoing_result_promise;
    }
 
   bound_call() {
       return (...arguments) => this.call_api(...arguments);
    }
}

Then Python can just do;

async def wrap_api(the_api):
    teleporter_object = mr.eval("CallTeleporter();")
    async def service_api(): 
        while True:
            incoming_call_promise = teleporter_object["incoming_call_promises"].pop(0)[0]
            args, resolve, reject = await incoming_call_promise
            try:
                result = await the_api(*args)
                resolve(result) 
            except Exception as exc:
                reject(exc)
    return teleporter_object["bound_call"](), asyncio.Task(service_api)

# Then user code is responsible for "installing" the call to teleporter_object somewhere their JS code can find it,
# and running the asyncio.Task on an asyncio loop.

This would require no new C++ code. This is what I meant by it being awkward to create N-use callbacks just one-shot promises, but maybe it's sufficiently simpler than writing new C++ code that we should just do it this way.

[bpcreech]

Alternate implementation idea 3:

Create a Python/C API, mr_create_callback, which just creates a single JS callback function, implemented in C++, which does nothing but pack up the arguments and call the specified Python callback function. The function does not bother to convey a return value or exceptions. Note: we shouldn't expose this API directly to PyMiniRacer users. As with the sketch above, it's important that the Python callback function not re-enter JS (that is, it shouldn't attempt to all MiniRacer.eval as part of the callback), and the easiest way to ensure that is to not let PyMiniRacer users directly work with this API.

That would be sufficient, I think, to replace the current Promise implementation (promise_attacher.{h,cc}) and create a generalized callback API.

E.g., for Promises you do something like (within py_mini_racer.py):

class _Context:
    ...
    def promise_to_sync_future(self, promise: JSPromise) -> _SyncFuture:  # this function exists today; we're rewriting it here
        """Create and attach a Python _SyncFuture to a JS Promise."""

        future = _SyncFuture()

        resolved_callback_id = next(self._next_callback_id)
        rejected_callback_id = next(self._next_callback_id)

        def remove_callbacks():  # these are single-use callbacks!
            self._active_callbacks.pop(resolved_callback_id)
            self._active_callbacks.pop(rejected_callback_id)

        def on_resolved(value: PythonJSConvertedTypes) -> None:
            remove_callbacks()
            future.set_result(value)

        def on_rejected(exc: Exception) -> None:
            remove_callbacks()
            future.set_exception(exc)

        self._active_callbacks[resolved_callback_id] = on_resolved
        self._active_callbacks[rejected_callback_id] = on_rejected

        resolved_js_function = self._wrap_value(self._dll.mr_create_callback(self._ctx, resolved_callback_id)).to_python()
        rejected_js_function = self._wrap_value(self._dll.mr_create_callback(self._ctx, rejected_callback_id)).to_python()

        # does this actually work? seems too good to be true:
        promise["then"](resolved_js_function, rejected_js_function)

        return future

^ That replaces the existing feature to attach Python callbacks to JS promises. The generalized async function callback, on the other hand, which would create new Promises and stuff inbound calls into an asyncio.Queue (and maybe a queue.Queue), would look vaguely like the first implementation idea above, except it could reuse the same C++ API.

In other words, if we add some artifice on the Python and JS side we can build a basic mr_create_callback Python/C API into both a general-purpose Promise attacher and a Python-native JS API creator without exposing this interface directly to PyMiniRacer users.

[bpcreech]

Okay #46 generalizes how Promises do callbacks, creating a facility (_Context._make_js_callback) which I think we can reuse to implement arbitrary JS-to-Python callbacks with no further C++ changes.

Still needs the Python-side changes in the initial comment; i.e., a new class (WrappedPythonFunction?) to wrap the user-supplied Python callback function, creating and wiring up JS callbacks to a queue.Queue and/or an asyncio.Queue or asyncio.Task and generating JS Promises to send back to the JS side.