GH-113464: A copy-and-patch JIT compiler

[brandtbucher]

'Twas the night before Christmas, when all through the code
Not a core dev was merging, not even Guido;
The CI was spun on the PRs with care
In hopes that green check-markings soon would be there;
The buildbots were nestled all snug under desks,
Even PPC64 AIX;
Doc-writers, triage team, the Council of Steering,
Had just stashed every change and stopped engineering,

When in the "PRs" tab arose such a clatter,
They opened GitHub to see what was the matter.
Away to CPython they flew like a flash,
Towards sounds of PROT_EXEC and __builtin___clear_cache.
First LLVM was downloaded, unzipped
Then the Actions were running a strange new build script,
When something appeared, they were stopped in their tracks,
jit_stencils.h, generated from hacks,
With their spines all a-shiver, they muttered "Oh, shit...",
They knew in a moment it must be a JIT.

More rapid than interpretation it came
And it copied-and-patched every stencil by name:
"Now, _LOAD_FAST! Now, _STORE_FAST! _BINARY_OP_ADD_INT!
On, _GUARD_DORV_VALUES_INST_ATTR_FROM_DICT!
To the top of the loop! And down into the call!
Now cache away! Cache away! Cache away all!"
But why now? And how so? They needed a hint,
Thankfully, Brandt gave a great talk at the sprint;
So over to YouTube the reviewers flew,
They read the white paper, and the blog post too.

And then, after watching, they saw its appeal
Not writing the code themselves seemed so unreal.
And the platform support was almost too easy,
ARM64 Macs to 32-bit PCs.
There was some runtime C, not too much, just enough,
Basically a loader, relocating stuff;
It ran every test, one by one passed them all,
With not one runtime dependency to install.
Mostly build-time Python! With strict static typing!
For maintenance ease, and also nerd-sniping!

Though dispatch was faster, the JIT wasn't wise,
And the traces it used still should be optimized;
The code it was JIT'ing still needed some thinning,
With code models small, and some register pinning;
Or new calling conventions, shared stubs for paths slow,
Since this JIT was brand new, there was fruit hanging low.
It was awkwardly large, parsed straight out of the ELFs,
And they laughed when they saw it, in spite of themselves;

A configure flag, and no merging this year,
Soon gave them to know they had nothing to fear;
It wasn't much faster, at least it could work,
They knew that'd come later; no one was a jerk,
But they were still smart, and determined, and skilled,
They opened a shell, and configured the build;
--enable-experimental-jit, then made it,
And away the JIT flew as their "+1"s okay'ed it.
But they heard it exclaim, as it traced out of sight,
"Happy JIT-mas to all, and to all a good night!"

• Issue: JIT Compilation #113464

[brandtbucher]

Alright, I think I've addressed everyone's comments. Thanks for the reviews... this PR is looking better than ever!

If anyone is planning on reviewing this, please do so now. Otherwise, my plan is to merge this first thing on Monday.

(I've confirmed that the two CI failures are failing on main too, and aren't JIT-related).

[brandtbucher]

Well, due to a GitHub UI glitch, the merge button errored out when I pushed it, leaving the PR open. But never fear... it was indeed merged (twice)!

Thanks again, everybody. :)

[gvanrossum]

Congrats! Well done.

[Yhg1s]

As Release Manager, and after talking this through with the Steering Council, I'm going to have to ask: where is the PEP for this, or some other kind of document explaining the design decisions and the discussions around it? Why was it merged now, without a PEP or wide discussion among Core Developers, especially since the immediate performance benefit is pretty much non-existent?

We have the PEP process for a reason. For longevity of CPython, it's important to not just add new features, but also bring along other Core Developers in their design -- especially since we all implicitly adopt the maintenance of it. Even optional bits, because they are all part of the support surface (especially when they're desirable features). We also have a standing policy not to experiment in main, and while I understand the upside of merging this early and then iterating on it, especially given how it meshes with other improvements to uops and what not, this feels like it's close to doing just that: experimenting in main. Merging it and iterating makes it much harder to revert it later, should that turn out to be necessary. That's not to say that it mustn't be merged, but it should be considered an exception and done with some deliberation with the RM or the SC.

How much follow-up work relies on this being merged, now? How difficult would it be to revert now and merge again later?

[brandtbucher]

Thanks for voicing your concerns.

As Release Manager, and after talking this through with the Steering Council, I'm going to have to ask: where is the PEP for this, or some other kind of document explaining the design decisions and the discussions around it? Why was it merged now, without a PEP or wide discussion among Core Developers, especially since the immediate performance benefit is pretty much non-existent?

There isn't a PEP for this. If I were to write one today, it would basically cover all of the same points as the talk I gave at the October sprint (slides, paper, blog post).

Despite the absence of a PEP, I feel that this was widely-discussed with core devs. Just to recap the timeline...

I first introduced this patch at the core dev sprint last year, on October 10th, in a 48-minute talk and Q&A. During the Q&A, I said:

If this is something people would like to see in main, off by default, we can get it in main very easily. I know we aren't crazy about experiments in main, which is kind of why I'm floating the idea now.

The video was shared with all core devs that same day. In both the in-person and follow-up Discourse discussion, I personally perceived the feedback to be very positive (much more positive than I anticipated). Maybe I misread the room, but the response convinced me that this wasn't very hard-to-understand or controversial (except for parts of the build process, in particular the LLVM requirement).

Twenty days later, on October 30th, I made the following comment on Discourse, visible to all committers:

An update: my current goal is to merge this initial implementation (behind a build flag, off by default) before the American Thanksgiving holiday. That’s on Thursday, November 23rd… which means I’ll probably be opening the PR around Friday the 17th. Active development can then continue on main (which seemed okay with people at the sprint).

Nobody objected. That deadline came and went, and a month later the PR itself was published as a draft for 24 days, then open for an additional 11 days before merging (which I announced 3 days in advance).

Merging it and iterating makes it much harder to revert it later, should that turn out to be necessary. That's not to say that it mustn't be merged, but it should be considered an exception and done with some deliberation with the RM or the SC.

How much follow-up work relies on this being merged, now? How difficult would it be to revert now and merge again later?

Part of the reason why there's a negligible performance impact right now is because it purposefully omits lots of "obvious" performance improvements in order to keep the initial JIT implementation as small and easy-to-review as possible. I'd rather have each one of them be reviewed on their own merits than develop a more-and-more sophisticated JIT that we all feel pressure to merge before the beta freeze.

I don't believe these changes would make reverting any more difficult, since they'll probably be isolated to new files (git revert f6d9e59 works fine right now on main).

Should one of us open an issue on the SC tracker?

[Yhg1s]

I'm glad to hear you think the discussion was wide-spread enough. I'm not really concerned that the JIT wasn't researched or reviewed enough, although I will say that I tuned out the discussion about your presentation on discuss.python.org because I was expecting a PEP once it was fleshed out (and I don't believe I was the only one).

I do believe a PEP is warranted, even at this stage. A PEP isn't just for making decisions, it's also for recording the history of the decisions (and the process around it), and to set boundaries and limitations to a particular feature. It's where people (core developers and others) will look for answers to questions they have about this feature. A PEP that summarises (or links to) arguments previously made is fine. I (and the SC) would like it to address questions like:

• What's the story for inspecting the runtime with the JIT enabled, like with perf or C debuggers (not PEP 669 debuggers, which I assume are unaffected)?
• What are the effects on and limitations of the JIT in terms of code size, binary size, memory use, etc? I remember in October we talked about 32-bit offsets which I know is going to be problematic in certain situations.
• What other downsides, negative effects and subtle semantic differences can people expect when they enable this in their builds?
• How are we going to determine whether this should be promoted from an experimental feature to one we recommend, or even one we make the default?

[brandtbucher]

Okay, makes sense. I'll start on a PEP.

[brandtbucher]

At long last, the PEP is up: https://discuss.python.org/t/pep-744-jit-compilation