v0.0.60: writing a C extension properly

R09, writing a C extension properly, which is the last lesson in M8. That milestone now has all seventeen of its lessons written and merged, C01 through C08 and R01 through R09, and what remains in it is blueprints, one app and four animations.

This is the first lesson in the whole set that writes C rather than reading it. Four rules an extension owes the runtime, each one broken on purpose and then fixed in front of you.

The error path first. Two nearly identical C functions, both packing their argument into a tuple and hashing it, one of which drops the tuple on the way out through the failure and one of which does not. A thousand failing calls leave two thousand references behind in one and none in the other, and the process is perfectly happy either way.

Then the collector. One box type compiled three ways: no GC flag, the flag plus a tp_traverse that reports its type and forgets its field, and the flag plus a complete one. Only the third is ever freed out of a cycle. The one in the middle is registered with the collector, gets asked what it is holding, gives an incomplete answer, and leaks exactly as thoroughly as the one that was never registered at all.

Then teardown, where tp_clear breaks links for the collector and tp_finalize is where your cleanup goes, running exactly once whether the object died on its reference count or inside a cycle. Then the single line at the bottom of the file, Py_MOD_GIL_USED or Py_MOD_GIL_NOT_USED, which a free threaded build reads and acts on.

The last section is a leak test, because four rules that fail silently need one. It is CPython's -R flag with the interesting parts removed, about a dozen lines, and it catches both broken boxes and clears the correct one.

Two Tier 1 recordings, bringing the total to 45. CPython's own leak hunter on the debug build, over four small tests that all pass an ordinary run and two of which it fails. And on the free threaded build, one shared object from CPython's test suite loaded three ways, where one of the loads turns the lock back on for the whole process and says so.

Also six diagrams, three glossary terms, and sixteen citations into the pinned tree.

Eighty lessons now, 630 claims, 2575 citations, 242 glossary terms and 45 Tier 1 experiments.

v0.0.59, when the interpreter stops

R08, when the interpreter stops, in #189.

The last lesson about the runtime as a thing with a lifetime. R01 asked what has already happened before your first line, and this one asks what is still to come after your last one. Every cell starts a child interpreter, because a notebook cannot watch its own shutdown.

The cells walk the order of the end with sys.is_finalizing() next to each step, show atexit callbacks coming back newest first and one registered during shutdown that never runs, ask a late __del__ what it can still reach and find its module globals intact but sys.modules empty, let a callback and a finalizer both raise while the exit status stays at zero, catch the single generation two collection, and leave a subinterpreter open so its warning turns up.

The case worth the lesson is the daemon thread. Pass time.sleep and your finalizers run. Pass a function defined in your own module and its stack frame holds your module's globals, so the module dict is never cleared and nothing in it is freed. The debug recording puts a number on it: 12680 references still alive at the end, against zero for the same objects held any other way.

Two Tier 1 recordings on a release build and a debug build, twelve citations, six diagrams, and three glossary terms. Also a stale count fixed in R07, which said seven sample extensions where the table has eight.

M8 now has R09 left to write.

v0.0.58, the stable ABI

R07, the stable ABI, which is the lesson about what stands between a compiled extension and a working module.

There are two gates. The first is the file name, and it is decided before anything is opened, from a fixed list of tags compiled into the interpreter. The lesson drives it by putting empty files in temporary directories and asking the finder what it sees, which is enough to show why abi3t exists: a free threaded build refuses abi3 and takes abi3t, an ordinary 3.15 build takes both, and 3.14 has never heard of abi3t.

The second gate is PyABIInfo, twelve bytes and new in 3.15. Because PyABIInfo_Check is a plain exported function, the lesson builds the struct in ctypes and calls the real check, so the four refusals it collects are CPython's own error messages rather than a paraphrase.

Two Tier 1 recordings run both gates on a release build and on a build made with --disable-gil. Six file name tags become four, four refusals become six.

Six diagrams, three glossary terms, and ten notebook cells that all run under Pyodide.

R08 and R09 left on M8.

v0.0.57, the C API tiers

R06, the C API tiers, is the sixth runtime lesson and the first one in the book that looks at CPython from the C side rather than from inside Python.

The C API is three directories and two macros. Include/ is open to any extension. Include/cpython/ needs Py_LIMITED_API to be undefined, and the mechanism is an #ifndef around an #include at the bottom of each public header. Include/internal/ starts nearly every file with three lines that stop the compiler with an error unless you define Py_BUILD_CORE. On 3.15 that third directory is 148 files and 42552 lines, which is more than the other two put together.

The lesson walks the Py_LIMITED_API guard as a preprocessor stack, so the numbers are measured: 580 functions a limited build may call, 186 the guard takes away, 974 in the other two directories. It then makes the case that the functions are the cheap part. The struct layouts go too, and with no fields to read Py_TYPE becomes a call and Py_DECREF becomes a call to _Py_DecRef.

None of the tiers survive into the binary, and that turns out to be on purpose. Inside the internal headers, 530 names are spelled PyAPI_FUNC and 493 of them resolve through ctypes.pythonapi on the release build, against 757 spelled plain extern of which 3 do. Both spellings sit in the same files, often two lines apart, and 168 comments name which bundled shared extension needs each export. Two Tier 1 recordings put that on record, and the free threaded one finds six more names, which are the ones behind Py_GIL_DISABLED.

_PyDict_SizeOf gets called by hand and returns exactly what dict.__sizeof__ returns, 16 bytes short of sys.getsizeof because that adds the collector header.

Six diagrams, four glossary terms, and five differs= notes checked against a real 3.14 run. Half the cells need the headers, which a browser tab does not have, so they sit behind one flag and say so; the other half only needs ctypes, so all eleven probed cells run under Pyodide.

Merged in #185.

v0.0.56, lazy imports

One lesson in this one, R05 on lazy imports, plus a fix to R03 that the browser probe turned up along the way.

PEP 810 landed in 3.15 and added one soft keyword to the import statement. lazy import json binds the name straight away and leaves the finding, reading and running of the module until the first time something reads that name back. R05 weighs what an unused import costs, shows the five line LazyLoader dance you had to write before 3.15, and disassembles four import statements to show that a lazy import compiles to the same IMPORT_NAME opcode as a plain one with the mode carried in two spare bits of the argument.

From there it works in a temporary directory on sys.path, so every module it looks at is one the cell above wrote. The placeholder gets inspected without being resolved, resolved by a single bare name read, and watched escaping into another variable where it resolves on the read rather than on the copy. Six spellings of the keyword get compiled to see which the symbol table refuses. The behaviour gets turned on without the keyword, once with a filter and once with a __lazy_modules__ list. A module raises during resolution and the two part traceback gets read.

Three Tier 1 recordings back the measured parts. Deferring is worth about three quarters of a run on a file that imports twelve modules and uses one. The other two show something I did not expect: waking a placeholder takes the interpreter wide import lock rather than the per module name lock an ordinary import takes, and holds it for the whole resolution. On a free threaded build, four threads importing four different modules the ordinary way keep 3.60 cores busy, and four threads waking four different deferred imports keep 0.98. That is filed as issue #184.

The R03 fix is unrelated to lazy imports. The Emscripten filesystem keeps whole second timestamps, so a directory mtime does not always move when a file is written into it and FileFinder keeps serving a stale listing, which made one cell fail at random in the browser. It now drops the finder out of sys.path_importer_cache and says why.

Also here: six diagrams, three glossary terms, rows in both READMEs, and refreshed citations, claims and probe recordings.

The repo is now at 76 lessons, 229 glossary terms, 601 claims and 37 Tier 1 experiments, all of it green under just check and all 76 lessons running end to end on Pyodide.

Merged: #183.

v0.0.55, frozen modules

R04, the fourth of the nine runtime lessons, and the one about the modules that live inside the binary. R03 finished on a fact it did not explain: import os never opens os.py. This is why.

The problem is a real chicken and egg. The import system is written in Python, in Lib/importlib/_bootstrap.py, so there is no way to import it. CPython cuts the loop by compiling that file during its own build, marshalling the code object, and writing the bytes into the binary as a C array. The lesson proves it rather than asserting it. The compiled module body of _frozen_importlib contains zero IMPORT_NAME opcodes, which is exactly what makes it loadable with nothing running, while _frozen_importlib_external, loaded second and by then with an import system to use, contains eight. init_importlib hands sys and _imp in as arguments, which is why _bootstrap.py never writes import sys.

Ten cells, and each one watches rather than describes. The thirty three frozen names split into the three arrays frozen.c keeps them in, which the flag treats differently: three for the import system that no setting can remove, nineteen for what a bare startup needs, eleven hello world modules for the test suite. A frozen code object pulled out of the binary and weighed. os asked five questions about where it came from. The flag turned off inside the running interpreter, so you can watch os move from FrozenImporter to SourceFileLoader and back without leaving the notebook. A startup run twice under -v, counting the files the second run reads that the first one does not.

The facts worth keeping. A frozen module knows perfectly well which file it would have opened: find_spec works the path out from sys._stdlib_dir and parks it on the spec as loader_state, and the loader copies it onto __file__, so the spec says frozen and __file__ says a real path and both are true. That is the whole reason a traceback through frozen code is readable, because linecache sees a filename starting with <frozen and reads __file__ out of the globals instead. The cost is not where you would guess: both paths end in the same marshal.loads over the same bytes, so what freezing removes is the finder search and the file read in front of it, and since every file involved is an already compiled .pyc, it is not saving compilation either.

Two Tier 1 recordings put a number on it, and the interesting part is that the two builds disagree about the default. initconfig.c turns frozen modules off under Py_DEBUG, so the program was rewritten to ask for on and off explicitly on every child and report the default as a measured fact rather than an assumption. Freezing gives back 14.2 percent of a release startup and 9.9 percent of a debug one. Both measurements alternate the two cases round by round, because running one case forty times and then the other measures the page cache.

Four new glossary terms: frozen module, import bootstrap, loader state and module alias. Six diagrams and twenty one citations. Both READMEs updated. GLOSSARY.md is 226 terms and CLAIMS.md is 589 claims across 75 lessons.

Nine of the ten cells run end to end in a browser. The tenth needs a second process, so it prints a line saying so.

v0.0.54 R03, what import does

R03, the third of the nine runtime lessons, and the one that takes import apart. It is not a keyword doing something the language will not explain. It compiles to a call to an ordinary builtin, that builtin is written in Python in a file you can open, and every step it takes is reachable from inside the language.

Ten cells, and each one watches rather than describes. The four spellings of the statement compiled and read back opcode by opcode, which settles that import a.b binds a and that a relative import is the empty string at a level above zero. A finder put on the front of sys.meta_path that answers nothing and writes down every question, turning one statement into three visible searches, outermost first. The three finders asked for the same three names side by side. A circular import in a temporary directory, caught reading a module halfway through its own body. A module served out of a string by fourteen lines of class, with no file on disk anywhere. The three caches an import passes through, including the one that keeps a directory you have just created invisible.

The facts worth keeping. IMPORT_NAME looks __import__ up in builtins every single time, which is why replacing it works. A dotted import is one search per part, and everything after the first part is looked for in the parent package's __path__ rather than on sys.path. A name can have more than one answer and the earlier finder wins, so import os never opens os.py, because FrozenImporter gets asked before PathFinder. The module object goes into sys.modules before its body runs, which is both why circular imports work at all and exactly what decides how much of a half loaded module the other side can see, and if the body raises the entry is taken back out again.

Two Tier 1 recordings settle the import lock, which almost everybody has wrong. It is one lock per module name and has been since 3.3, not one lock for the process. Four threads importing four different modules keep 1.02 cores busy on a release build and 3.63 on a free threaded one, so what serialises them is the GIL. Four threads importing the same module keep almost exactly one core busy on both builds and the body runs once, which is the per module lock doing its job.

Four new glossary terms: module spec, meta path finder, path entry finder and module lock. Six diagrams and seventeen citations. Both READMEs updated. GLOSSARY.md is 222 terms and CLAIMS.md is 580 claims across 74 lessons.

All ten cells also run end to end in a browser with nothing skipped, which no earlier R lesson managed.

v0.0.53 R02, where the state lives

R02, the second of the nine runtime lessons, and the one that puts a frame around everything before it. A running Python is three nested things: a runtime that there is one of per process, interpreters inside it, and threads inside those. Every fact this book has taught you belongs to exactly one of the three levels, and the level decides who can see a change when you make one.

Eight cells, and every section works the same way: make a second interpreter, ask both of them the same question, and watch for where the answers stop matching. Nine objects asked for their address on both sides. A search for the exact integer at which the two of them stop agreeing. The interpreter list with its ids counting upwards and never coming back. The recursion limit, a warnings filter and sys.modules changed on one side and read on the other. signal.signal tried from the main thread, from another thread and from inside a second interpreter. Two threads each holding a different exception at the same instant.

The facts worth keeping. The shared pile is small and it is not luck: None, the booleans, the small ints, the one character strings, the fixed identifier strings and the static type objects are fields of the runtime struct rather than allocations, which is why nothing can free them and why every interpreter points at the same bytes. Where the small int run ends is a #define, not a policy, and it moved from 256 on 3.14 to 1024 on 3.15, which the cell measures rather than asserts. An interpreter owns its own sys.modules, its own builtins, its own import lock, its own warnings filters and its own recursion limit. Signals are the exception, because the operating system has never heard of interpreters, so there is one handler table per process and the test for who may write to it is two conditions in one line of C. The exception being handled is per thread, which two threads inside an except block at the same instant make obvious.

The lesson also says out loud why no cell calls sys._current_frames from inside a second interpreter: on a build with the GIL that aborts the process, because the function materialises frame objects for other interpreters' frames and each interpreter allocates from its own pools. That is issue #179, found while writing this.

Two Tier 1 recordings, one release and one free threaded. An operating system thread takes a few hundred microseconds and a whole interpreter takes about fifty times that, plus a couple of megabytes of resident memory for as long as you hold it. The free threaded build charges more for both, and the reason is the allocator rather than noise.

Two new glossary terms, runtime state and static object. Six diagrams and fifteen citations. Both READMEs updated. GLOSSARY.md is 218 terms and CLAIMS.md is 571 claims across 73 lessons.

v0.0.52 R01, before your first line

R01, the first of the nine runtime lessons, and a change of subject. Everything up to here has been about what happens when your code runs. This one is about what has already happened by the time it starts.

Eight cells, every one of them asking a child interpreter rather than the notebook, because a notebook has been up for minutes and cannot be a fair witness about its own startup. What is in sys.modules before your first line and where each of those modules came from. Every import that runs before your first line, with its own time. Seven children disagreeing about sys.flags.optimize. The whole path configuration with an exists check next to each entry. sys.path[0] compared across -c, a script, -m and -P. And ten timed starts of an interpreter that runs nothing at all.

The facts worth keeping. With site out of the way none of the startup modules were read from a file, because they are either C compiled into the binary or Python bytecode frozen into it. Startup is two halves on purpose and the first half has no import system, which is why a configuration mistake is a fatal error with a plain C string rather than a traceback. The command line and the environment settle a disagreement by taking the higher number rather than the nearer one, so PYTHONOPTIMIZE=2 survives a later -O. sys.path is produced by a Python program frozen into the binary and handed eleven C functions to stand in for the os.path it cannot import. And the front of sys.path is pushed on after startup is over, which is the whole mechanism behind a local random.py shadowing the standard library.

Two Tier 1 recordings, the first release against debug pair in the book. The debug build starts in 56.5 ms against 26.5 ms, and most of the extra is not the assertions. Release reports 17 frozen modules and none from a file. Debug reports 3 frozen and 14 from a file, because a debug build turns frozen modules off so that you can step through the real Lib/os.py. The import bill goes from 11.6 ms to 30.8 ms with it.

A new glossary group for startup and shutdown: two phase initialisation, path configuration, safe path. Six diagrams and sixteen citations. Both READMEs updated. GLOSSARY.md is 216 terms and CLAIMS.md is 564 claims across 72 lessons.

v0.0.51 C08 sending work to another interpreter

C08, the last of the eight concurrency lessons, and the practical half of C04.

C04 built a second interpreter and found that the two share almost nothing. C08 asks what follows from that: how do you give one of them a job, and what does the handover cost. Which callables can cross and why a recursive function cannot, the three route fallback chain that an argument goes through, what comes back when the far side raises, which standard library modules refuse to load in a subinterpreter at all, how many queue round trips a second you can afford, and InterpreterPoolExecutor.

Two Tier 1 recordings split the same two workloads three ways on both builds. On a build with the lock, four interpreters take the arithmetic job from 507 ms to 208 ms while four threads do nothing. On a build without the lock the ranking flips and threads win at 179 ms against 359 ms. The job whose argument is a two hundred thousand item list is ruined on both, 6 ms one at a time against 255 ms across four interpreters, because there the crossing is the whole job. The verdict is not a ranking, it is a question about the shape of your work.

Also in this release: a correction to C07's closing paragraph, which described C04's material rather than C08's.

Three glossary terms, six diagrams and twelve citations. Both READMEs updated. GLOSSARY.md is now 213 terms and CLAIMS.md is 557 claims across 71 lessons.