Repository navigation
Three things changed since 1.1.0. A run now ends at its deadline whatever the script calls, and nothing of it keeps computing after Run returns. Memory a script holds is charged where it was not, so MaxAllocBytes bounds what the process keeps. And many answers, error types and texts moved to CPython 3.14's. The engine no longer limits how many runs go at once: that is the host's to decide.
dotnet add package PyLite --version 1.2.0
Every fix has its test, listed in docs/fixes.md. What still differs from CPython is in docs/deviations.md, and embedding the engine is described in the new hosting guide.
Upgrading from 1.1.0: hosts
| what changed | what to do |
|---|---|
The engine no longer limits how many runs execute at once, or their memory together. EngineOptions.MaxConcurrentRuns and GlobalMemoryCeilingBytes are [Obsolete] and ignored |
A host that relied on them keeps its own limit around Run and sizes DefaultLimits for its parallelism; remove the properties (CS0618 under warnings-as-errors) |
Run refuses a new run when Windows can no longer promise its stack, its MaxAllocBytes and 200 MB, and when its thread cannot start: HostError with LimitName "FreeMemory" or "ThreadStart" |
Treat both as "try later" |
ScriptRunner.GetErrorLimit() and ScriptError.LimitName name a refusal ("FreeMemory", "ThreadStart", "CircuitBreaker") |
Tell "try later" from a failing host function by the name, not the message |
Run(null) returns a HostError instead of throwing ArgumentNullException |
Read the result |
RunCompleted fires once for every Run, also for a syntax error, a refusal and a leak |
A counter of runs may grow |
Host failures end as HostError with a per-run copy of the exception; methods and properties built with SlotDescriptor are fenced like host functions; a host function's Thread.Abort ends the run, not the thread |
Do not match on EngineFault for them |
Memory the process could not get while compiling or running is HostError / "FreeMemory", no longer EngineFault |
Treat it as "try later"; keep logging the EngineFault event |
A source not in the compile cache compiles off the calling thread (1.1.0 compiled on the caller's stack and could refuse print(1) deep in it) |
Nothing; the answer no longer depends on the caller's stack |
| The compile cache keys on source and name and holds at most 2 million characters of source | |
JSON inputs are parsed before any prelude or provider code runs; ScalarsIn coercion is stricter (a sign and at most 4,300 ASCII digits make an int) |
Bad input fails earlier |
The clock is validated: UtcNow must have DateTimeKind.Utc |
A clock returning DateTime.Now was silently wrong and is now a HostError |
| Host callbacks run in the invariant culture | A callback that formats with the current culture sets its own |
Function tables are copied and frozen on the caller's thread; HostFunctionTable.Set with a null function throws ArgumentNullException |
Later changes to a table do not reach a request already made |
A token already cancelled when Run is called runs nothing; a run refused before it starts reports the refusal even when the source has a syntax error |
Read Error.Kind as before |
A limit hit while a finally runs after an abort keeps the first abort's LimitName |
A host that maps LimitName to an action sees the cause that ended the run |
| A run is reported leaked only when it is stuck in host code past deadline + grace and made no progress for 5 s | |
Error types and texts follow CPython 3.14: re.error is re.PatternError, a missing module is ModuleNotFoundError, an unexpected indent is IndentationError, ZeroDivisionError says division by zero |
A host that matches Message, PythonType or Line should check them |
Step and allocation charges moved, by up to about 30% for re matching, json and XML building; time.sleep charges one step per quantum |
Budgets sized on old runs may need another look |
New: ScriptRunner.SetLimitIntBits, ScriptRunner.ConfigureSharedEngine |
A run ends on its deadline
In 1.1.0 a few single calls into .NET could not be interrupted: the run overran its deadline by seconds, and past deadline + 5.25 s the engine gave up on the runner thread, which kept a core busy.
- Big-int arithmetic runs in the engine's own kernel on the run's thread and checks the deadline every few milliseconds: Karatsuba multiply and square, Knuth division, Lehmer gcd, divide-and-conquer
str(), sliding-windowpow(b, e, m). There are no helper threads any more. It is faster too: at 4K bits a square ×1.9,divmod×1.7,3**20000×1.9; at a million bits it multiplies 14× faster than .NET'sBigInteger. - Formatting: a float or
Decimalformatted with a huge precision (format(x, '.1000000f')) pads zeros instead of building 10**precision, and a result pastMaxStrCharsis refused before it is built. - Regexes: the ones the engine runs on script data (
csv.Sniffer,textwrap) obey the run'sRegexPerOpTimeoutMs. A pattern that starts with a large count, such asre.compile('a{100000}'), compiles at once, and so does one whose nested loops .NET merges into such a count ((?:(?:a{60,}){60,}){20}): .NET used to spend seconds building a search table for them in one call the deadline could not stop. - Long work for one step: 28 kinds of loops that did long work for the one step they charged now end on their deadline. A method or constructor call is checked against the deadline when it returns; long string compares and searches, set and
ChainMapwalks, codec error handlers, case mappings of astral text, XPath and XML walks are charged as they go; keyword binding, dataclass fields,tee,dequedeletion,StringIOreads,heapq.mergeand long csv line terminators no longer take quadratic time. - No stack overflow: four scripts ended the host process at default limits (a 3,000-level
[*[*[*…]]]while compiling,str()of an exception wrapped 200,000 times,reprof a deepitertools.repeatchain,<=of deeply nestedCounters). They now end in aSyntaxErroror aRecursionError. - Bounded where CPython is not: a tree that contains itself (
a.append(a)) ends onMaxCollectionItemswhen walked; an XML tag with more than 10,000 attributes is refused;struct.unpack('0s' * n),combinations_with_replacement,product(repeat=n), slice assignment andlist.insertend onMaxCollectionItems.
Memory that is now charged
- Outputs are checked against
MaxStrCharsas they grow (ascii, format precisions,print,jsonescapes,strftime,str.format, f-strings), and lists grown from iterators are charged and capped (f(*it),random.choices,batched, XMLextend).json.dumpsandreprof a large list no longer keep two work-stack entries per item (850 MB and 1 GB for a 5-million-item list before). - Compiled regexes are charged what .NET keeps of them while they are alive: the pattern, its search table, its runner and, for a loop of groups, its backtracking stack. A script that keeps 50,000 compiled patterns in a list meets a 32 MB
MaxAllocBytes; patterns a loop compiles and drops are given back once collected. A match object is charged 64 bytes and 32 a group, so a script that makes millions of many-group matches needs a largerMaxAllocBytesthan before. - XML: an element without attributes holds no attrib dict until a script asks for it: a tree of plain elements needs a third of the memory and charges a quarter less, an attribute-heavy one charges up to half more, which is closer to what it holds. ElementPath's parent maps are charged while held and given back when a search ends; a loop that adds a row and finds
row[last()]builds no map per search (6,000 rows on 2.4M steps; 18M before). - A method call on a namedtuple no longer keeps the run's type in the engine's script cache, and
ScriptError.ScriptTracebackis capped at 262,144 characters.
Scripts that may notice
Answers that now equal CPython 3.14's, so a script that relied on the old one sees a difference:
- Floats read and print as in CPython: a decimal text, an int and a
Decimalbecome the nearest double, andreprgives the shortest digits that read back.reprwas wrong for about 2% of everyday values (9.37631e-113printed as9.376309999999999e-113), so some floats,random()among them, print a different last digit. str(),repr(), f-strings,%dandformat()of an int of more than 4,300 decimal digits raiseValueError, and so doesint()of such a string. The dialect has nosys, so the limit cannot be raised;hex(),bin(),to_bytes()andDecimalare not limited.math.hypot,dist,sumprod,expm1,sin,cos,tanandsumof floats: last digits.Decimal ** n,//,%anddivmod: the last digit of a power, whole digits of a 29-digit dividend.re: IGNORECASE folds as sre does (the Kelvin sign matchesk,ıandİmatchi,ſmatchess; underre.Aonly ASCII letters fold);\w,\W,\b,\Bfollow Unicode 16;\123after a group is an octal character and\10with ten groups is group 10; inre.subtemplates\0is a character, not group 0, and\g<name>of a missing group is anIndexError;\UHHHHHHHHnames its character; a bytes pattern is aTypeErrornaming the dialect.- Arguments that were dropped silently are a
TypeError: an unknown keyword to dozens of functions and methods, a parameter given by position and by name, an extra positional (json.dumps(1, 2)), a keyword to an exception class. Many functions take their first parameters by name (Decimal(value='1')used to beDecimal('0')). typing: a subscription keeps its arguments (repr(Sequence[int])),Optional[int]andUnion[int, str]are theint | Noneunion, whichisinstancetakes, andTypedDict('TD', {...})makes a class.- Syntax:
from __future__ import annotationsis a no-op instead of aModuleNotFoundError; a class header's arguments are read as a call's (class C(metaclass=M)is refused by name); a statement that startsmatch -,match +ormatch *withmatcha variable is read as a match statement;__debug__cannot be bound. str.encodeerror handlers see a whole run:'a\xe9\xe9b'.encode('ascii', 'replace')isb'a??b'.base64.b32decoderefuses lower case withoutcasefold=True;bytes.fromhexskips all ASCII whitespace. Codec names take CPython's aliases (cp65001,latin,us, ...).- ElementTree:
ET.fromstringof bytes decodes by the XML declaration'sencoding(every byte document was read as UTF-8);itertext()includes the text of comments and processing instructions;[@a!='v']no longer matches elements withouta;find('')and whitespace in a path answer as CPython's;tostringwritestext=''as<x />and keeps<in HTML attributes;iterparsehands every record to a loop that clears the root after each one (it saw 1 of 10). collections:Counter's==,del, in-place operators and repr order;OrderedDict | dictanddefaultdict | dictkeep the subclass;ChainMap | mappingworks; an iterator ofreversed(deque)raisesRuntimeErrorafter the deque changes.- Iterators: a script iterator that ends and then yields again is asked again by
filter,map,zip,enumerateand the others CPython asks again;copyof a list, tuple, str, bytes or range iterator copies it, andcopyof a generator-like object, a hash object or acsvreader raisesTypeError. traceback.format_exceptionand its kind honourlimit; an exception the engine raises insideexcepthas__context__; a traceback frame inside a module names the module's file.- Character predicates (
isprintable,isalpha,isidentifier, ...) andreprof new characters follow Unicode 16. string.capwords(s, '')raisesValueError: empty separator;gzip.compress(mtime=)refuses a value outside 0..2^32-1;strptimerefuses a week the ISO year does not have;uuid1()draws its node once per run.- A missing module is
ModuleNotFoundError(still anImportError);IOError,TimeoutError,FileNotFoundError,PermissionError,ConnectionError,Warning,UserWarningand others exist forexceptclauses;e.argscan be assigned.
New
math.erf,erfc,gammaandlgamma;int | strunions;Decimalformatse,g,nand%;map(strict=True),statistics.LinearRegression,correlation(method='ranked');structformat'e'.functools.lru_cacheandcachefunctions havecache_info(),cache_clear(),cache_parameters()and__wrapped__;functools.wrapssets__wrapped__; an exception has__suppress_context__.
Tests and documents
- About 8,600 xUnit tests (7,000 in 1.1.0), a CPython 3.14 oracle corpus among them, 90 differential scripts against IronPython 3.4.2 and 274 adversarial probes (130 in 1.1.0).
- New: the probe bank,
tools/PyLite.Stress/bank: about 1,200 scripts written by the reviewers of this release, each pinned to its outcome, output, steps and charged memory, with 1.1.0's steps and memory as a ceiling where 1.1.0 ran them. A changed answer, a new false limit or a hidden slowdown fails the gate. - docs/test-report.md is the full gate run on the release commit.
- Benchmarks against IronPython 3.4.2, three passes on the machine that measured 1.1.0, are in benchmarks/results/1.2.0: money in
decimal×66–78 faster, JSON ×15–18, CSV ×4.0–4.5; string joining, slower in 1.1.0, is now a little faster andsortedwith a key, slower in 1.1.0, about equal; datetime arithmetic is ×1.06–1.18 slower. PyLite allocates less than 1.1.0 on eleven of the twelve workloads.