Skip to content

PyLite 1.2.0

Latest

Choose a tag to compare

@sashokfestival sashokfestival released this 16 Nov 09:22

Three things changed since 1.1.0. A run now ends at its deadline whatever the script calls, and nothing of it keeps computing after Run returns. Memory a script holds is charged where it was not, so MaxAllocBytes bounds what the process keeps. And many answers, error types and texts moved to CPython 3.14's. The engine no longer limits how many runs go at once: that is the host's to decide.

dotnet add package PyLite --version 1.2.0

Every fix has its test, listed in docs/fixes.md. What still differs from CPython is in docs/deviations.md, and embedding the engine is described in the new hosting guide.

Upgrading from 1.1.0: hosts

what changed what to do
The engine no longer limits how many runs execute at once, or their memory together. EngineOptions.MaxConcurrentRuns and GlobalMemoryCeilingBytes are [Obsolete] and ignored A host that relied on them keeps its own limit around Run and sizes DefaultLimits for its parallelism; remove the properties (CS0618 under warnings-as-errors)
Run refuses a new run when Windows can no longer promise its stack, its MaxAllocBytes and 200 MB, and when its thread cannot start: HostError with LimitName "FreeMemory" or "ThreadStart" Treat both as "try later"
ScriptRunner.GetErrorLimit() and ScriptError.LimitName name a refusal ("FreeMemory", "ThreadStart", "CircuitBreaker") Tell "try later" from a failing host function by the name, not the message
Run(null) returns a HostError instead of throwing ArgumentNullException Read the result
RunCompleted fires once for every Run, also for a syntax error, a refusal and a leak A counter of runs may grow
Host failures end as HostError with a per-run copy of the exception; methods and properties built with SlotDescriptor are fenced like host functions; a host function's Thread.Abort ends the run, not the thread Do not match on EngineFault for them
Memory the process could not get while compiling or running is HostError / "FreeMemory", no longer EngineFault Treat it as "try later"; keep logging the EngineFault event
A source not in the compile cache compiles off the calling thread (1.1.0 compiled on the caller's stack and could refuse print(1) deep in it) Nothing; the answer no longer depends on the caller's stack
The compile cache keys on source and name and holds at most 2 million characters of source
JSON inputs are parsed before any prelude or provider code runs; ScalarsIn coercion is stricter (a sign and at most 4,300 ASCII digits make an int) Bad input fails earlier
The clock is validated: UtcNow must have DateTimeKind.Utc A clock returning DateTime.Now was silently wrong and is now a HostError
Host callbacks run in the invariant culture A callback that formats with the current culture sets its own
Function tables are copied and frozen on the caller's thread; HostFunctionTable.Set with a null function throws ArgumentNullException Later changes to a table do not reach a request already made
A token already cancelled when Run is called runs nothing; a run refused before it starts reports the refusal even when the source has a syntax error Read Error.Kind as before
A limit hit while a finally runs after an abort keeps the first abort's LimitName A host that maps LimitName to an action sees the cause that ended the run
A run is reported leaked only when it is stuck in host code past deadline + grace and made no progress for 5 s
Error types and texts follow CPython 3.14: re.error is re.PatternError, a missing module is ModuleNotFoundError, an unexpected indent is IndentationError, ZeroDivisionError says division by zero A host that matches Message, PythonType or Line should check them
Step and allocation charges moved, by up to about 30% for re matching, json and XML building; time.sleep charges one step per quantum Budgets sized on old runs may need another look
New: ScriptRunner.SetLimitIntBits, ScriptRunner.ConfigureSharedEngine

A run ends on its deadline

In 1.1.0 a few single calls into .NET could not be interrupted: the run overran its deadline by seconds, and past deadline + 5.25 s the engine gave up on the runner thread, which kept a core busy.

  • Big-int arithmetic runs in the engine's own kernel on the run's thread and checks the deadline every few milliseconds: Karatsuba multiply and square, Knuth division, Lehmer gcd, divide-and-conquer str(), sliding-window pow(b, e, m). There are no helper threads any more. It is faster too: at 4K bits a square ×1.9, divmod ×1.7, 3**20000 ×1.9; at a million bits it multiplies 14× faster than .NET's BigInteger.
  • Formatting: a float or Decimal formatted with a huge precision (format(x, '.1000000f')) pads zeros instead of building 10**precision, and a result past MaxStrChars is refused before it is built.
  • Regexes: the ones the engine runs on script data (csv.Sniffer, textwrap) obey the run's RegexPerOpTimeoutMs. A pattern that starts with a large count, such as re.compile('a{100000}'), compiles at once, and so does one whose nested loops .NET merges into such a count ((?:(?:a{60,}){60,}){20}): .NET used to spend seconds building a search table for them in one call the deadline could not stop.
  • Long work for one step: 28 kinds of loops that did long work for the one step they charged now end on their deadline. A method or constructor call is checked against the deadline when it returns; long string compares and searches, set and ChainMap walks, codec error handlers, case mappings of astral text, XPath and XML walks are charged as they go; keyword binding, dataclass fields, tee, deque deletion, StringIO reads, heapq.merge and long csv line terminators no longer take quadratic time.
  • No stack overflow: four scripts ended the host process at default limits (a 3,000-level [*[*[*…]]] while compiling, str() of an exception wrapped 200,000 times, repr of a deep itertools.repeat chain, <= of deeply nested Counters). They now end in a SyntaxError or a RecursionError.
  • Bounded where CPython is not: a tree that contains itself (a.append(a)) ends on MaxCollectionItems when walked; an XML tag with more than 10,000 attributes is refused; struct.unpack('0s' * n), combinations_with_replacement, product(repeat=n), slice assignment and list.insert end on MaxCollectionItems.

Memory that is now charged

  • Outputs are checked against MaxStrChars as they grow (ascii, format precisions, print, json escapes, strftime, str.format, f-strings), and lists grown from iterators are charged and capped (f(*it), random.choices, batched, XML extend). json.dumps and repr of a large list no longer keep two work-stack entries per item (850 MB and 1 GB for a 5-million-item list before).
  • Compiled regexes are charged what .NET keeps of them while they are alive: the pattern, its search table, its runner and, for a loop of groups, its backtracking stack. A script that keeps 50,000 compiled patterns in a list meets a 32 MB MaxAllocBytes; patterns a loop compiles and drops are given back once collected. A match object is charged 64 bytes and 32 a group, so a script that makes millions of many-group matches needs a larger MaxAllocBytes than before.
  • XML: an element without attributes holds no attrib dict until a script asks for it: a tree of plain elements needs a third of the memory and charges a quarter less, an attribute-heavy one charges up to half more, which is closer to what it holds. ElementPath's parent maps are charged while held and given back when a search ends; a loop that adds a row and finds row[last()] builds no map per search (6,000 rows on 2.4M steps; 18M before).
  • A method call on a namedtuple no longer keeps the run's type in the engine's script cache, and ScriptError.ScriptTraceback is capped at 262,144 characters.

Scripts that may notice

Answers that now equal CPython 3.14's, so a script that relied on the old one sees a difference:

  • Floats read and print as in CPython: a decimal text, an int and a Decimal become the nearest double, and repr gives the shortest digits that read back. repr was wrong for about 2% of everyday values (9.37631e-113 printed as 9.376309999999999e-113), so some floats, random() among them, print a different last digit.
  • str(), repr(), f-strings, %d and format() of an int of more than 4,300 decimal digits raise ValueError, and so does int() of such a string. The dialect has no sys, so the limit cannot be raised; hex(), bin(), to_bytes() and Decimal are not limited.
  • math.hypot, dist, sumprod, expm1, sin, cos, tan and sum of floats: last digits. Decimal ** n, //, % and divmod: the last digit of a power, whole digits of a 29-digit dividend.
  • re: IGNORECASE folds as sre does (the Kelvin sign matches k, ı and İ match i, ſ matches s; under re.A only ASCII letters fold); \w, \W, \b, \B follow Unicode 16; \123 after a group is an octal character and \10 with ten groups is group 10; in re.sub templates \0 is a character, not group 0, and \g<name> of a missing group is an IndexError; \UHHHHHHHH names its character; a bytes pattern is a TypeError naming the dialect.
  • Arguments that were dropped silently are a TypeError: an unknown keyword to dozens of functions and methods, a parameter given by position and by name, an extra positional (json.dumps(1, 2)), a keyword to an exception class. Many functions take their first parameters by name (Decimal(value='1') used to be Decimal('0')).
  • typing: a subscription keeps its arguments (repr(Sequence[int])), Optional[int] and Union[int, str] are the int | None union, which isinstance takes, and TypedDict('TD', {...}) makes a class.
  • Syntax: from __future__ import annotations is a no-op instead of a ModuleNotFoundError; a class header's arguments are read as a call's (class C(metaclass=M) is refused by name); a statement that starts match -, match + or match * with match a variable is read as a match statement; __debug__ cannot be bound.
  • str.encode error handlers see a whole run: 'a\xe9\xe9b'.encode('ascii', 'replace') is b'a??b'. base64.b32decode refuses lower case without casefold=True; bytes.fromhex skips all ASCII whitespace. Codec names take CPython's aliases (cp65001, latin, us, ...).
  • ElementTree: ET.fromstring of bytes decodes by the XML declaration's encoding (every byte document was read as UTF-8); itertext() includes the text of comments and processing instructions; [@a!='v'] no longer matches elements without a; find('') and whitespace in a path answer as CPython's; tostring writes text='' as <x /> and keeps < in HTML attributes; iterparse hands every record to a loop that clears the root after each one (it saw 1 of 10).
  • collections: Counter's ==, del, in-place operators and repr order; OrderedDict | dict and defaultdict | dict keep the subclass; ChainMap | mapping works; an iterator of reversed(deque) raises RuntimeError after the deque changes.
  • Iterators: a script iterator that ends and then yields again is asked again by filter, map, zip, enumerate and the others CPython asks again; copy of a list, tuple, str, bytes or range iterator copies it, and copy of a generator-like object, a hash object or a csv reader raises TypeError.
  • traceback.format_exception and its kind honour limit; an exception the engine raises inside except has __context__; a traceback frame inside a module names the module's file.
  • Character predicates (isprintable, isalpha, isidentifier, ...) and repr of new characters follow Unicode 16.
  • string.capwords(s, '') raises ValueError: empty separator; gzip.compress(mtime=) refuses a value outside 0..2^32-1; strptime refuses a week the ISO year does not have; uuid1() draws its node once per run.
  • A missing module is ModuleNotFoundError (still an ImportError); IOError, TimeoutError, FileNotFoundError, PermissionError, ConnectionError, Warning, UserWarning and others exist for except clauses; e.args can be assigned.

New

  • math.erf, erfc, gamma and lgamma; int | str unions; Decimal formats e, g, n and %; map(strict=True), statistics.LinearRegression, correlation(method='ranked'); struct format 'e'.
  • functools.lru_cache and cache functions have cache_info(), cache_clear(), cache_parameters() and __wrapped__; functools.wraps sets __wrapped__; an exception has __suppress_context__.

Tests and documents

  • About 8,600 xUnit tests (7,000 in 1.1.0), a CPython 3.14 oracle corpus among them, 90 differential scripts against IronPython 3.4.2 and 274 adversarial probes (130 in 1.1.0).
  • New: the probe bank, tools/PyLite.Stress/bank: about 1,200 scripts written by the reviewers of this release, each pinned to its outcome, output, steps and charged memory, with 1.1.0's steps and memory as a ceiling where 1.1.0 ran them. A changed answer, a new false limit or a hidden slowdown fails the gate.
  • docs/test-report.md is the full gate run on the release commit.
  • Benchmarks against IronPython 3.4.2, three passes on the machine that measured 1.1.0, are in benchmarks/results/1.2.0: money in decimal ×66–78 faster, JSON ×15–18, CSV ×4.0–4.5; string joining, slower in 1.1.0, is now a little faster and sorted with a key, slower in 1.1.0, about equal; datetime arithmetic is ×1.06–1.18 slower. PyLite allocates less than 1.1.0 on eleven of the twelve workloads.