pycc-next v0.3.0

Packages, stdlib, everyday code. Programs of several modules and packages compile as
one unit; dataclasses, enums, functools.cache, deque/Counter/defaultdict and
itertools compile natively; a real PyPI package (diff-match-patch) builds unchanged
as --ext modules and passes its own test suite in CI; strict mode compiles 161/200
real-world programs unchanged (was 124) and --non-strict runs 98.1% of their
top-level statements natively (was 95.1%). Prebuilt archives for Linux x86_64, Linux
aarch64 and macOS 26+ on Apple silicon.

Benchmarks (python3 bench/run.py, best of 7, shared 4-core box): fib 19.9x faster
than CPython / 2.0x of Rust, mandelbrot 43.9x / 0.96x, nbody 29.4x / 2.6x, sieve 25.0x
/ 1.9x, spectral_norm 46.1x / 1.7x, collections_bfs 6.8x / 2.9x, numpy kernels
stencil2d 677x / 2.3x and prefix_sum_np 78x / 1.7x.

Added

• A real PyPI package, unchanged: bench/package_check.py (CI-gated on Linux x86_64)
  builds every module of diff-match-patch 20241021 with --ext --non-strict and runs the
  package's own test suite with CPython against the compiled modules; every function
  and method compiles (v0.3 accept 4).
• --ext: a package's __init__.py builds to pkg/__init__.abi3.so (-o pkg/__init__),
  imported as the package; under --foreign-relative-imports a module's __name__ is
  its dotted name (pkg.mod); the module docstring is __doc__; a class's literal
  constants (INSERT = 1) are attributes of its published type.
• Non-strict --ext: module-level variables and imports are module attributes
  (mod.X, from mod import X, a package __init__'s re-exports).
• Non-strict: o[a:b] = v on a CPython object; a class attribute initialized with any
  expression (PATTERN = re.compile(...)), evaluated once where the class runs; a
  method that can run off its end returns None; an in-module call fills an object
  parameter's None or scalar default; a builtin exception built from an object message
  (ValueError("bad: " + obj)).
• Non-strict typing: an instance attribute nothing types is an object; an attribute
  stored values of two types becomes an object; a local first bound to None and later
  to something else is an object; a helper parameter or return inferred both as an
  object and as a scalar is an object; a helper returning a tuple or None returns an
  object; a container held by a name that flows into an object tuple becomes an object.
• Linux aarch64: a pycc-vX.Y.Z-aarch64-linux.tar.gz archive (glibc 2.39+), built,
  tested and smoke-tested on GitHub's ubuntu-24.04-arm runner next to the other two;
  CI runs the full test suite and the corpus gate on Linux aarch64.
• itertools natively in strict mode (and in --non-strict and --ext code), with no
  embedded CPython: count, islice, chain, chain.from_iterable, accumulate
  (operator functions, max/min, two-argument lambdas of those, initial=),
  pairwise, groupby (with a key), product (with repeat=), permutations,
  combinations and combinations_with_replacement. Each returns a lazy iterator
  implemented in the runtime, consumed by for, list(), sum(), next(),
  comprehensions and every other generator consumer; inputs are lists, strs, ranges
  and generators, pulled only as far as CPython pulls them.
• collections.deque natively: deque(), deque(iterable[, maxlen]), append,
  appendleft, pop, popleft, extend, extendleft, rotate, indexing, len,
  in, index, count, remove, iteration, copy, +/*, equality and repr
  (deque([...], maxlen=n)); deque[T] annotations. Both ends are O(1) amortized, and a
  BFS over a deque runs ~10x faster than CPython.
• collections.Counter and collections.defaultdict natively: Counter(),
  Counter(list|str|dict), c[k] reading 0 for a missing key, most_common([n]),
  update, subtract, elements, total, +/-/|/&, equality and CPython's
  repr order; defaultdict(int|float|str|list|set|dict|lambda|function) with the
  factory run on a missing-key read, and nested defaultdicts. Counter[K] and
  defaultdict[K, V] annotations, and defaultdict(list) typed from its uses.
  bench/py/collections_bfs.py (a deque BFS plus a Counter) runs ~9x faster than
  CPython.
• Dict | (union) between two dicts.
• A list, set or dict comprehension over a generator ([x for x in gen()]).
• An unannotated generator helper whose yields the solver cannot type from their terms
  alone (a yielded tuple display, a value read from a local container) is typed from its
  body once its parameters are known; this includes every lifted generator expression.
• Programs of several modules and packages compile as one unit (natively in strict and
  --non-strict mode, and as an --ext module): import m, import pkg.sub [as s],
  from pkg.sub import f [as g], relative imports and __init__.py; m.f(...),
  m.CONST and m.Class(...) are direct native calls and reads. Each module has its own
  namespace and __name__, and its body runs once at its first import, in CPython's
  import order. An unresolved module or name, a module used as a value and an import
  cycle are compile errors (T0021, C0001, E0108).
• dataclasses.dataclass, natively in strict mode: @dataclass and
  @dataclass(...) with init/repr/eq/order/frozen/unsafe_hash, field
  defaults, field(default=..., default_factory=..., init=, repr=, compare=),
  __post_init__, inherited fields, generated __init__/__repr__/__eq__/
  ordering/__hash__, and dataclasses.FrozenInstanceError on assignment to a frozen
  field. An --ext dataclass takes keyword arguments and prints as in CPython.
• enum: IntEnum beside Enum/StrEnum; members print, str(), repr() and
  format exactly as CPython's (Color.RED, <Color.RED: 1>, 2 for an IntEnum),
  Color(value) looks a member up (ValueError: 4 is not a valid Color), case Color.RED: patterns, members as set elements and dict keys, IntEnum ordering.
• functools.cache / functools.lru_cache (bare, (), maxsize=None, maxsize=N
  with least-recently-used eviction, maxsize=0) on module-level functions with
  int/float/bool/str parameters, and f.cache_clear(). Any other decorator
  still runs in CPython under --non-strict.
• Classes: print/str()/f-strings call a class's __str__ or __repr__, the
  repr() builtin and f-string !r/!s work, ==/!=/</<=/>/>= between
  instances call the class's dunders, == without __eq__ and is compare identity,
  and a call of a constructor with defaults may omit them. --ext classes install
  __repr__/__str__ as tp_repr/tp_str.
• hash() of a str, a float and a tuple holding them.
• Strict mode infers more of real-world untyped code (the corpus, v0.3 accept 3):
  • a return the solver cannot type from its terms (a returned tuple display such as
    return up, down, a value computed from a local it did not type) is typed from the
    function's body once its parameters are, and inference runs again with it, so its
    callers and the parameters they feed are typed too;
  • an empty list whose producer reads an unannotated function's parameter or return
    (result.append(x % num)) is typed once those are inferred;
  • an empty list bound to a name whose element nothing reads, produces or compares
    (L = [] only rebound, measured or printed) is a list[int]: no operation can
    observe it;
  • an operator over two operands already typed is typed at once, so a.count(m - 1)
    types the parameter it is passed to.
• The builtins exit()/quit() and from sys import exit are sys.exit (run through
  the embedded CPython, so finally blocks run and the exit status and message are
  CPython's).
• A module-level X = sys.stdin / X = sys.stdout alias: X.readline(),
  X.write(s) and the other stream operations are the native ones.
• More template idioms of competitive-programming code compile natively: a module-level
  if on a flag bound once to a literal (DEBUG = 0) or on sys.gettrace() is the
  branch that runs; next(sys.stdin) (a line, StopIteration at EOF); a parameter
  whose default is a builtin conversion no call passes (def rl(t=int)); x in c of a
  constant tuple; a, b = x, y typing both targets for inference;
  g[i].sort(reverse=True); a def returning sys.stdin.buffer.readline() read only
  as numbers.
• Non-strict mode: the inference solver runs again with what it settled written in, and
  a return it leaves open is typed from the body when every return agrees, so less
  code becomes a CPython object.
• A tuple whose elements share one type: t[k] with a computed index (int, bool and
  float elements; IndexError: tuple index out of range outside [-len, len)), and
  for x in t / a comprehension over it (any element a list can hold).
• A generator expression reading a module-level loop variable (all(q % p for p in primes) inside for q in ...) is typed; graph[u].append(v) types the empty list
  graph's items start as ({i: [] for i in range(n)}); non-strict mode runs
  any/all of a generator of CPython objects natively.
• Item access on an instance of a module class defining __getitem__/__setitem__/
  __delitem__ (through self, or a name bound only to calls of the class) calls those
  methods, and their unannotated parameters are inferred from those calls.
• list() and dict() with no argument are [] and {}; set.update(g) and
  list.extend(g) take a generator; a local rebound from a scalar (temp = ord(c)) to an
  empty list is a separate variable; non-strict mode passes a generator to a CPython
  object's method as an iterator over it.
• The real-world corpus (v0.3 accept 3): strict mode passes 161/200, non-strict mode runs
  98.1% of top-level statements natively.

Changed

• Non-strict mode: chr() runs in CPython, whose str holds a lone surrogate
  (chr(0xD800)) that pycc's UTF-8 str cannot.
• d[k] += n on a dict of int values (a Counter's c[x] += 1) is one lookup, and a
  str caches its hash as CPython does: a word-counting loop runs ~4x faster than
  CPython.
• Strict mode, like non-strict mode, does not compile an unannotated definition nothing
  in a native program's entry module names: it can never run and no call site could type
  it (previously T0001). An --ext module keeps every definition.
• A dependency module's top-level names no longer share the entry module's namespace:
  two modules may define the same name, and an --ext artifact no longer exports the
  public functions of the project modules it imports (only the entry module's own).

Fixed

• print(), str() or an f-string of a value that may be None (int | None) passed
  pycc check and crashed the backend; it is refused with C0001.
• A __str__ whose signature is not (self) -> str was called for print()/str() and
  its result used as a str; the conversion is now refused (C0001), as CPython raises
  TypeError.
• A tuple boxed inside a loop (list(...) of a generator of tuples, a tuple stored in a
  container) grew the stack on every iteration and crashed after ~10^5 iterations.
• A comprehension over a fresh list (a call's result, a str expression) leaked that
  list.
• A fresh native value passed to CPython leaked once per call: a str (a literal, str(i),
  a concatenation, an f-string), a list, a bigint int or a class instance as a positional
  or keyword argument of a call on a CPython function or method, or a str subscript key
  of a CPython object (json.dumps(str(i)) grew ~47 bytes per iteration; it now runs in
  constant memory, strict and non-strict).
• Non-strict: a function returning a tuple with a non-scalar element (return text[:n], len(text) - n) crashed the backend in a build with fallback code (and --ext), whose
  export thunk flattened the tuple as if it were a struct of scalars; such a return is now
  typed from the body like any other instead of falling back to a CPython object.

Known gaps

• See docs/DEVIATIONS.md for every known difference from CPython 3.13.