Skip to content

Add details about JIT build infrastructure and updating dependencies to Tools/jit #141167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+31 −0
Open

Add details about JIT build infrastructure and updating dependencies to Tools/jit #141167

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0af3db2
Add details about bumping LLVM dep in cpython-bin-deps
savannahostrowski Nov 6, 2025
74d876b
Merge branch 'main' into jit-deps
savannahostrowski Nov 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Tools/jit/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ Otherwise, just configure and build as you normally would. Cross-compiling "just

The JIT can also be enabled or disabled using the `PYTHON_JIT` environment variable, even on builds where it is enabled or disabled by default. More details about configuring CPython with the JIT and optional values for `--enable-experimental-jit` can be found [here](https://docs.python.org/dev/using/configure.html#cmdoption-enable-experimental-jit).

## Miscellaneous
If you're looking for information on how to update the JIT build dependencies, see [JIT Build Infrastructure](jit_infra.md).

[^pep-744]: [PEP 744](https://peps.python.org/pep-0744/)

[^why-llvm]: Clang is specifically needed because it's the only C compiler with support for guaranteed tail calls (`musttail`), which are required by CPython's continuation-passing-style approach to JIT compilation. Since LLVM also includes other functionalities we need (namely, object file parsing and disassembly), it's convenient to only support one toolchain at this time.
28 changes: 28 additions & 0 deletions Tools/jit/jit_infra.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# JIT Build Infrastructure

This document includes details about the intricacies of the JIT build infrastructure.

## Updating LLVM

When we update LLVM, we need to also update the LLVM release artifact for Windows builds. This is because Windows builds automatically pull prebuilt LLVM binaries in our pipelines (e.g. notice that `.github/workflows/jit.yml` does not explicitly download LLVM or build it from source).

To update the LLVM release artifact for Windows builds, follow these steps:
1. Go to the [LLVM releases page](https://github.com/llvm/llvm-project/releases).
1. Download x86_64 Windows artifact for the desired LLVM version (e.g. `clang+llvm-21.1.4-x86_64-pc-windows-msvc.tar.xz`).
1. Extract and repackage the tarball with the correct directory structure. For example:
Copy link
Contributor

@diegorusso diegorusso Nov 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More generic question: can this be avoided at all?

IIUC, we are doing all of this just to set the right directory within the archive. Can we actually deal with the original directory structure in CPython?

Copy link
Member Author

@savannahostrowski savannahostrowski Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some thoughts here: https://github.com/python/cpython/pull/141167/files#r2504767448

...but TL;DR: yes, we could potentially automate the creation of LLVM release artifacts in cpython-bin-deps and make tarball extraction smarter. I'd tackle this in pieces and as a separate issue though!

Copy link
Contributor

@diegorusso diegorusso Nov 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean, can we avoid to use cpython-bin-deps at all? Can we deal with whatever structure the original tar ball has in https://github.com/llvm/llvm-project/releases ?
What we are doing is a just a repackaging with a different directory structure. Cannot we do it at all and fetch the original tarball from the llvm project?
Here I'm debating the existence of that extra project and the existence of this documentation :)

Copy link
Member Author

@savannahostrowski savannahostrowski Nov 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, there's extra context here. In order for the JIT to be in the Windows release, this was necessary as all dependencies for Windows release builds have to be in cpython-bin-deps. That's why this is the status quo. More on this in #137604 (comment).

That said, Emma, Itamar and I are working on a PEP to improve this experience. See https://discuss.python.org/t/pre-pep-redesigning-cpython-source-binary-dependencies/101824

```bash
tar -xf clang+llvm-21.1.4-x86_64-pc-windows-msvc.tar.xz
mv clang+llvm-21.1.4-x86_64-pc-windows-msvc llvm-21.1.4.0
tar -cf - llvm-21.1.4.0 | pv | xz > llvm-21.1.4.0.tar.xz
```
The tarball must contain a top-level directory named `llvm-{version}.0/`.
1. Go to [cpython-bin-deps](https://github.com/python/cpython-bin-deps).
1. Create a new release with the updated LLVM artifact.
- Create a new tag to match the LLVM version (e.g. `llvm-21.1.4.0`).
- Specify the release title (e.g. `LLVM 21.1.4 for x86_64 Windows`).
- Upload the asset (you can leave all other fields the same).

### Other notes
- You must make sure that the name of the artifact matches exactly what is expected in `Tools/jit/_llvm.py` and `PCbuild/get_externals.py`.
- We don't need multiple release artifacts for each architecture because LLVM can cross-compile for different architectures on Windows; x86_64 is sufficient.
- You must have permissions to create releases in the `cpython-bin-deps` repository. If you don't have permissions, you should contact one of the organization admins.
Loading