gh-142598: Add new type of executable to support external JITs #142550
+528
−19
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#105727 introduced the ability to have different types of executables. During the discussion of that there was mention of potentially adding support for JITs to be able to plug in as well. There's a similar disucssion of the idea over here: faster-cpython/ideas#575
This extends the existing set of executors so that PEP-523 JITs can hook into the interpreter frames and create their own "lightweight" frames which live on the stack with some minimal amount of initialization.
Currently the initialization required is probably more than idea:
FRAME_OWNED_BY_THREADvarious things look atstackpointer.Presumably the next step after this PR would be to add a new frame owner which is complete but doesn't require these additional fields to be initialized.
The JIT executors are created with a fixed object that includes a callback to reify the frame, a code object, and an optional piece of state. The reifier is called in various places where the JIT needs to populate the frame - for example accessing globals, builtins, or instr_ptr. Currently the API just provides a single callback with no information on what's required but you could imagine we could have a finer grained control if it was desired - but reification of the frame is generally going to happen in the slow path so simplicity seems the best here.
The code object isn't strictly necessary but because profilers and other external introspection tools frequently want a minimal number of hops to determine a functions identity it's tracked here instead of making them head back to the function object and then read the code from there.