✨ feat(pypy): run the C core on PyPy, publish 3.10 + 3.11 wheels

[gaborbernat]

turbohtml ships no pure-Python fallback, so PyPy users get nothing today. The C core needs little of what makes most C extensions unportable: it names no private _Py* API, builds its types with PyType_FromModuleAndSpec, keeps its state in module state, and uses multi-phase init. Of 66 translation units, 65 compiled against cpyext on the first try. 🎯

The 66th exposed three cpyext contracts that differ from CPython's. None of them produces a compiler error, a CPython behavior change, or a byte of different CPython machine code. PyPy corrupts data or dies. PyUnicode_CopyCharacters does not exist. PyUnicode_FromFormat returns a string cpyext left in the legacy wstr representation, so PyUnicode_KIND, PyUnicode_DATA and PyUnicode_GET_LENGTH are undefined on it; under NDEBUG their assertions compile away and GET_LENGTH answers one past the code point count. That is how sanitize("<script>x</script>") came to weave NUL bytes into its escaped tags. A 2-byte buffer cannot reach cpyext. cpyext materializes one by decoding it as UTF-16, which eats a leading U+FEFF as a byte-order mark, byte-swaps the rest of the string after a leading U+FFFE, folds a surrogate pair into the single code point it encodes, and aborts the interpreter on a lone surrogate through the strict error handler. A CPython str carries a lone surrogate, and both HTML input and the WHATWG tokenizer have to preserve one. escape("\ud800<") took the process down with SIGABRT.

src/turbohtml/_c/core/pycompat.h holds the whole adaptation, and every name in it expands to the CPython spelling on CPython. That is what keeps the CPython build unchanged. 64 of the 65 translation units compile to byte-identical machine code against main at -O3, so the PGO and LTO layouts the release wheels depend on cannot shift. The odd one out is dom/node.c, which carries the bug fix below. On PyPy the core builds any result too wide for Latin-1 at 4-byte kind, where cpyext is exact, while CPython keeps the narrowest kind its str equality demands. escape() takes a separate 2-byte path there, because its block copies assume the output width matches the input's.

Two more gaps needed handling. cpyext hands sq_item a raw negative index instead of adding sq_length first, so node[-1] answered the first child. While chasing that I found a CPython bug, fixed in its own commit: PySequence_GetItem adds sq_length but never rechecks, so node[-len - 1] still arrived negative and node_item returned first_child where list raises IndexError.

cpyext also ignored Py_TPFLAGS_DISALLOW_INSTANTIATION until PyPy 7.3.21, which let Document() and its thirteen siblings construct without a tree and then segfault on first use, so pure Python code could crash the interpreter. 7.3.21 honors the flag and prints a debug line to stdout for every type that carries it. Both cost nothing to avoid, because cpyext seals a type through an explicit tp_new as well, the branch the flag would otherwise shadow. Setting that slot instead of the flag seals every PyPy with CPython's own error message and leaves 7.3.21 quiet, since its debug line fires only for a type carrying the flag. Element("div") and Text("hi") still build, because a subtype declaring its own tp_new overrides the inherited one. That removes the need for a version floor, and PyPy 3.10 joins the wheel matrix.

I rejected the approach the closest analogues take. MarkupSafe and msgpack detect PyPy at build time and ship a pure-Python fallback instead, because cpyext turns their PyUnicode_KIND fast path into a decode-and-allocate. Cython disables the same family of fast paths on PyPy with CYTHON_USE_UNICODE_INTERNALS=0. That trade does not exist here, since no Python implementation of this core exists to fall back to. HPy offers no kind-dispatched buffer access and has not released since 2023.

docs/explanation/interpreters.rst states what PyPy costs, from a table the bench harness generates rather than from figures typed into prose. tox -e bench -- interpreters builds the working tree under CPython 3.14 and PyPy 3.11, runs the same seven operations over the same vendored corpora in each, and writes docs/explanation/bench/interpreters.json for the existing bench-table directive to render. Parsing a whole document stays within a small factor. Anything that hands Python an object per node runs an order of magnitude slower and past it: walking every descendant, collecting visible text, serializing a tree. The JIT recovers none of it, because the time goes to cpyext rather than to bytecode. turbohtml is a poor reason to choose PyPy, and the page says so.

Reviewers should look hardest at core/pycompat.h and at the codegen-neutrality claim. I checked it by compiling every translation unit from main and from this branch at -O3 -g0 -DNDEBUG and comparing normalized objdump -d output. cibuildwheel's pypy and pypy-eol groups together select pp310 and pp311. The PyPy wheels skip PGO, since training a profile by driving turbohtml from PyPy measures cpyext instead of the C hot paths. On PyPy 7.3.19, 7.3.21 and 7.3.23 the library produces byte-identical output to CPython 3.13 across the operations I compared, down to lone surrogates, byte-order marks and surrogate pairs, and every sealed type refuses construction with the same message on all four interpreters. ⚙️

Three upstream cpyext defects fell out of this work. Each reproduces on the current 7.3.23, violates a documented CPython contract, and is filed with a standalone reproducer: pypy#5524 for the non-canonical PyUnicode_FromFormat result, pypy#5525 for the UTF-16 materialization of 2-byte buffers, and pypy#5526 for the unadjusted negative sq_item index. The sealing flag and the 7.3.21 debug line were already reported as pypy#5318 and pypy#5388, and both are fixed upstream. The tp_new seal here works whichever release a user runs. Every one of the five is linked from the code that works around it, in core/pycompat.h, dom/node.c and docs/explanation/interpreters.rst.

[codspeed-hq]

Merging this PR will not alter performance

✅ 91 untouched benchmarks
⏩ 18 skipped benchmarks¹

---

_{Comparing gaborbernat:feat/pypy-support (a152763) with main (c80b7bc)}

Footnotes

1. 18 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩