Repository navigation
Releases
PyLite 1.2.0
Three things changed since 1.1.0. A run now ends at its deadline whatever the script calls, and nothing of it keeps computing after Run returns. Memory a script holds is charged where it was not, so MaxAllocBytes bounds what the process keeps. And many answers, error types and texts moved to CPython 3.14's. The engine no longer limits how many runs go at once: that is the host's to decide.
dotnet add package PyLite --version 1.2.0
Every fix has its test, listed in docs/fixes.md. What still differs from CPython is in docs/deviations.md, and embedding the engine is described in the new hosting guide.
Upgrading from 1.1.0: hosts
| what changed | what to do |
|---|---|
The engine no longer limits how many runs execute at once, or their memory together. EngineOptions.MaxConcurrentRuns and GlobalMemoryCeilingBytes are [Obsolete] and ignored |
A host that relied on them keeps its own limit around Run and sizes DefaultLimits for its parallelism; remove the properties (CS0618 under warnings-as-errors) |
Run refuses a new run when Windows can no longer promise its stack, its MaxAllocBytes and 200 MB, and when its thread cannot start: HostError with LimitName "FreeMemory" or "ThreadStart" |
Treat both as "try later" |
ScriptRunner.GetErrorLimit() and ScriptError.LimitName name a refusal ("FreeMemory", "ThreadStart", "CircuitBreaker") |
Tell "try later" from a failing host function by the name, not the message |
Run(null) returns a HostError instead of throwing ArgumentNullException |
Read the result |
RunCompleted fires once for every Run, also for a syntax error, a refusal and a leak |
A counter of runs may grow |
Host failures end as HostError with a per-run copy of the exception; methods and properties built with SlotDescriptor are fenced like host functions; a host function's Thread.Abort ends the run, not the thread |
Do not match on EngineFault for them |
Memory the process could not get while compiling or running is HostError / "FreeMemory", no longer EngineFault |
Treat it as "try later"; keep logging the EngineFault event |
A source not in the compile cache compiles off the calling thread (1.1.0 compiled on the caller's stack and could refuse print(1) deep in it) |
Nothing; the answer no longer depends on the caller's stack |
| The compile cache keys on source and name and holds at most 2 million characters of source | |
JSON inputs are parsed before any prelude or provider code runs; ScalarsIn coercion is stricter (a sign and at most 4,300 ASCII digits make an int) |
Bad input fails earlier |
The clock is validated: UtcNow must have DateTimeKind.Utc |
A clock returning DateTime.Now was silently wrong and is now a HostError |
| Host callbacks run in the invariant culture | A callback that formats with the current culture sets its own |
Function tables are copied and frozen on the caller's thread; HostFunctionTable.Set with a null function throws ArgumentNullException |
Later changes to a table do not reach a request already made |
A token already cancelled when Run is called runs nothing; a run refused before it starts reports the refusal even when the source has a syntax error |
Read Error.Kind as before |
A limit hit while a finally runs after an abort keeps the first abort's LimitName |
A host that maps LimitName to an action sees the cause that ended the run |
| A run is reported leaked only when it is stuck in host code past deadline + grace and made no progress for 5 s | |
Error types and texts follow CPython 3.14: re.error is re.PatternError, a missing module is ModuleNotFoundError, an unexpected indent is IndentationError, ZeroDivisionError says division by zero |
A host that matches Message, PythonType or Line should check them |
Step and allocation charges moved, by up to about 30% for re matching, json and XML building; time.sleep charges one step per quantum |
Budgets sized on old runs may need another look |
New: ScriptRunner.SetLimitIntBits, ScriptRunner.ConfigureSharedEngine |
A run ends on its deadline
In 1.1.0 a few single calls into .NET could not be interrupted: the run overran its deadline by seconds, and past deadline + 5.25 s the engine gave up on the runner thread, which kept a core busy.
- Big-int arithmetic runs in the engine's own kernel on the run's thread and checks the deadline every few milliseconds: Karatsuba multiply and square, Knuth division, Lehmer gcd, divide-and-conquer
str(), sliding-windowpow(b, e, m). There are no helper threads any more. It is faster too: at 4K bits a square ×1.9,divmod×1.7,3**20000×1.9; at a million bits it multiplies 14× faster than .NET'sBigInteger. - Formatting: a float or
Decimalformatted with a huge precision (format(x, '.1000000f')) pads zeros instead of building 10**precision, and a result pastMaxStrCharsis refused before it is built. - Regexes: the ones the engine runs on script data (
csv.Sniffer,textwrap) obey the run'sRegexPerOpTimeoutMs. A pattern that starts with a large count, such asre.compile('a{100000}'), compiles at once, and so does one whose nested loops .NET merges into such a count ((?:(?:a{60,}){60,}){20}): .NET used to spend seconds building a search table for them in one call the deadline could not stop. - Long work for one step: 28 kinds of loops that did long work for the one step they charged now end on their deadline. A method or constructor call is checked against the deadline when it returns; long string compares and searches, set and
ChainMapwalks, codec error handlers, case mappings of astral text, XPath and XML walks are charged as they go; keyword binding, dataclass fields,tee,dequedeletion,StringIOreads,heapq.mergeand long csv line terminators no longer take quadratic time. - No stack overflow: four scripts ended the host process at default limits (a 3,000-level
[*[*[*…]]]while compiling,str()of an exception wrapped 200,000 times,reprof a deepitertools.repeatchain,<=of deeply nestedCounters). They now end in aSyntaxErroror aRecursionError. - Bounded where CPython is not: a tree that contains itself (
a.append(a)) ends onMaxCollectionItemswhen walked; an XML tag with more than 10,000 attributes is refused;struct.unpack('0s' * n),combinations_with_replacement,product(repeat=n), slice assignment andlist.insertend onMaxCollectionItems.
Memory that is now charged
- Outputs are checked against
MaxStrCharsas they grow (ascii, format precisions,print,jsonescapes,strftime,str.format, f-strings), and lists grown from iterators are charged and capped (f(*it),random.choices,batched, XMLextend).json.dumpsandreprof a large list no longer keep two work-stack entries per item (850 MB and 1 GB for a 5-million-item list before). - Compiled regexes are charged what .NET keeps of them while they are alive: the pattern, its search table, its runner and, for a loop of groups, its backtracking stack. A script that keeps 50,000 compiled patterns in a list meets a 32 MB
MaxAllocBytes; patterns a loop compiles and drops are given back once collected. A match object is charged 64 bytes and 32 a group, so a script that makes millions of many-group matches needs a largerMaxAllocBytesthan before. - XML: an element without attributes holds no attrib dict until a script asks for it: a tree of plain elements needs a third of the memory and charges a quarter less, an attribute-heavy one charges up to half more, which is closer to what it holds. ElementPath's parent maps are charged while held and given back when a search ends; a loop that adds a row and finds
row[last()]builds no map per search (6,000 rows on 2.4M steps; 18M before). - A method call on a namedtuple no longer keeps the run's type in the engine's script cache, and
ScriptError.ScriptTracebackis capped at 262,144 characters.
Scripts that may notice
Answers that now equal CPython 3.14's, so a script that relied on the old one sees a difference:
- Floats read and print as in CPython: a decimal text, an int and a
Decimalbecome the nearest double, andreprgives the shortest digits that read back.reprwas wrong for about 2% of everyday values (9.37631e-113printed as9.376309999999999e-113), so some floats,random()among them, print a different last digit. str(),repr(), f-strings,%dandformat()of an int of more than 4,300 decimal digits raiseValueError, and so doesint()of such a string. The dialect has nosys, so the limit cannot be raised;hex(),bin(),to_bytes()andDecimalare not limited.math.hypot,dist,sumprod,expm1,sin,cos,tanandsumof floats: last digits.Decimal ** n,//,%anddivmod: the last digit of a power, whole digits of a 29-digit dividend.re: IGNORECASE folds as sre does (the Kelvin sign matchesk,ıandİmatchi,ſmatchess; underre.Aonly ASCII letters fold);\w,\W,\b,\Bfollow Unicode 16;\123after a group is an octal character and\10with ten groups is group 10; inre.subtemplates\0is a character, not group 0, and\g<name>of a missing group is anIndexError;\UHHHHHHHHnames its character; a bytes pattern is aTypeErrornaming the dialect.- Arguments that were dropped silently are a
TypeError: an unknown keyword to dozens of functions and methods, a parameter given by position and by name, an extra positional (json.dumps(1, 2)), a keyword to an exception class. Many functions take their first pa...
PyLite 1.1.0
PyLite 1.1.0 adds the parts of Python that integration scripts most often needed (with, zipfile, enum, class constants, super(), io.BytesIO), makes sorting and several hot paths faster, closes every resource-limit gap that an audit with hostile scripts found, and puts the whole test suite into this repository.
dotnet add package PyLite --version 1.1.0
Upgrading from 1.0.0
Everything below is additive or a fix, but these can change what an existing host or script sees:
- Host API: the module implementation classes (
ReModule,JsonModule,CsvModule, ...,FloatRepr,NumericHash) areinternal; they were public by accident and never documented. - Limits that now hold:
SleepTotalMs(a sleep past it fails the run; each sleep used to be cut at 5 s silently), a per-runRegexPerOpTimeoutMs(patterns used a fixed 250 ms),MaxCollectionItemsfordictandset, and the hard capsMaxDataDepth10,000 andMaxIntBits4,000,000, whichEngineOptions.DefaultLimitsis clamped to as well.int()of an over-long string ends asMaxIntBits(it wasMaxSteps). - Scripts:
json.dumpsof too-deep data raisesRecursionError(wasValueError);type(e)of an exception is its class; a@dataclasssubclass runs its generated__init__even when a base wrote one; builtins and methods refuse an extra argument or an unknown keyword they used to ignore; aDecimalliteral too long for the backend is rounded instead of raisingOverflow;def/class/importof anonlocalname bind that name, not a global.
New in the dialect
with: one or several managers, the parenthesized form, classes with__enter__/__exit__;io.StringIO/io.BytesIOare managers. A resource limit still ends the run whatever__exit__answers.io.BytesIO: an in-memory binary file with CPython's surface (read, readline(s), iteration, write, writelines, seek, tell, truncate, getvalue);json.load,ET.parseandElementTree.writework with it.io.StringIOgainedtruncate,writelines, thereadable/writable/seekableflags,next(f), and zero padding past the end; both buffers are capped atMaxStrChars.zipfile: read, write and append ZIP archives in anio.BytesIO(STORED and DEFLATED, ZIP64 records,open(name, 'w'), comments,testzip,printdir); archives are interchangeable with .NET'sZipArchive. Zip bombs end the run on the budget: inflation is charged and stops at the recorded size, entries are capped, overlapped entries are refused.- Class bodies take constants (
NEW = 'new', read asStatus.NEWandself.NEW, shadowed by subclasses, settable on the class) and docstrings;__match_args__,__slots__and__hash__ = Nonework. Exception subclasses may carry constants too (class ApiError(Exception): CODE = 500,e.CODE). super()andsuper(Class, self)in methods,__class__,self.__class__;type(obj)of a class instance is its class, andtype(e) is ValueErrorholds for an exception.super().__init__(...)reaches a base's__init__, the one@dataclassgenerated, orobject.__init__.enum:Enum,IntEnum,StrEnum,auto(),@unique,Enum('Name', 'A B C'), and theclass Status(str, Enum)/(int, Enum)form. Members work inmatch, as dict keys and in sets;IntEnum/StrEnummembers compute and compare as ints/strs andjson.dumpswrites their values.contextlib.suppress,nullcontext,closing;decimal.localcontext()(alsoprec=/rounding=),decimal.setcontextanddecimal.Context(prec, rounding): a rounding block restores the previous context however it ends.suppresscannot keep a run alive past a resource limit.string.Template(substitute,safe_substitute,$name/${name}/$$,get_identifiers,is_valid, subclasses with their owndelimiter,idpattern,braceidpattern,flagsorpattern) and theemail.utilsdate helpers (formatdate,format_datetime,parsedate,parsedate_tz,parsedate_to_datetime,mktime_tz): CPython's own code, running as script code.Decimal('1_000.5')andFraction('1_000/3')accept PEP 515 underscores, as CPython does. New builtinEOFError;obj.attr += xworks on engine objects that allow the attribute.
Faster
- Sorting (
sorted,list.sortand everything built on them) is 1.3–3× faster: presorted and reversed data take n−1 comparisons, int/str/float keys are compared without boxing. A comparison or key that raises no longer leaves the list empty. - Dict and set comprehensions over a range, a set or a dict are about twice as fast and allocate 45–60% less for large results; every dict/set insert probes once instead of twice.
re:match()is anchored atpos(a failing match on a long string no longer scans it: ×150 on 10 KB); group reads are 10–30% faster.- A function whose body is one
return <expr>runs on the path lambdas use: about 20% faster per call, tracebacks unchanged. - Against IronPython 3.4.2 on the same scripts (docs/performance.md, samples in benchmarks/): money in
decimal×65–73 faster, JSON ×13–14 faster, a new engine in a warm process ×187–226 faster;sortedwith akey×1.08–1.36 slower (it was ×2.6–3.3 in 1.0.0).
Resource limits
From an audit with hostile scripts:
- A host that raises
MaxDataDepthcan no longer have the process killed by a stack overflow (reprof ~200k nested deques); the run ends asStackDepth, andMaxDataDepthhas a hard cap of 10,000.EngineOptions.DefaultLimitsobeys the engine hard caps also for a run that sets no limits of its own. SleepTotalMsworks: a sleep past it fails the run at once, real sleep matches the request, andRunStatistics.SleepMsis reported.RegexPerOpTimeoutMsset for a run is honoured (a run's value can only lower the engine'sDefaultLimits). Adictorsetstops onMaxCollectionItemslike a list.- Nested
functools.partialis charged (100k levels ran 34 s and took 1.3 GB).int()of more digits thanMaxIntBitsholds ends asMaxIntBits, and7**350000or2**999999, which fit, are no longer refused.MaxIntBitshas a hard cap of 4,000,000 bits. int.bit_length()/bit_count()of a huge int,DecimalandFractionliterals of a million digits,pow(b, e, m)andpow(b, -1, m)with huge numbers,math.isqrt(now CPython's algorithm), Fraction arithmetic with huge parts andint.from_bytes/to_bytes(now linear:to_bytesof a 786k-bit int took 12 s, now 0.8 s) no longer run past the deadline. A single long big-int operation no longer delays the end of an aborted run by its own length: it runs on a helper thread while the run polls its deadline.- Outputs are capped as they grow, not built first:
str.replace(''),%-formatting,json.dumps,re.subwith a function,ET.tostring,textwrap.indent,bytes.join,itertools.tee,calendar.month. Grouping digits of a huge int, draining a set/dict withpop()/popitem()anddel a[::2]are linear;str.expandtabsno longer hits the memory limit spuriously;format(x, '.100000f')no longer grows memory shared by every later run.
Fixes
- A wrong argument is a catchable error, never an internal engine error:
random.choice(),ET.fromstring(b'\xff'),uuid.UUID(version='4'),b'..'.replace(a, b, None)and others. Builtins, container methods andmathrefuse an extra argument or a keyword they do not take (math.sqrt(4, 5),'x'.strip(chars='x')were silently accepted);pow(x, y, mod=),bytes(s, encoding=),replace(count=),base64.b64encode(altchars=),b64decode(validate=),binascii.hexlify(sep=)andhmac.new(msg=)now use the keyword.isinstance/issubclassno longer fault when a class is checked against a builtin type. def/class/import/except asof anonlocalname binds that name (it bound a global), anddelof a captured ornonlocalvariable works.- As CPython:
1 // 0.1 == 9.0,-6.0 % 3 == 0.0,float('-0.0')keeps its sign,bytesfind/count/replace with an empty needle,type(p) is Pfor a namedtuple,lru_cache(partial(...)),calendar.IllegalMonthError(13),statistics.harmonic_mean/linear_regressionof iterators,asdictof a cyclic dataclass is aRecursionError. - As CPython:
json.dumpsof data nested pastMaxDataDepthraisesRecursionError(it wasValueError); adefault=hook that returns the object it was given isValueError: Circular reference detected, one that returns a new object every time is thatRecursionError. - As CPython, found by the differential corpus: a
Decimalsum that is exactly zero takes the addends' sign (Decimal('1.30') - Decimal('1.3')wasDecimal('-0.00'));csv.writerescapes as CPython does (withdoublequote=Falsea quote is escaped, not quoted; the escape character is escaped inside quotes too; a line-terminator character quotes the field);functools.lru_cache(typed=True)keys on the argument types andmaxsize=0caches nothing. - A module's exception class is one object for the engine's lifetime, so
type(e) is re.error(andjson.JSONDecodeError,struct.error,zlib.error,csv.Error, ...) holds,zoneinfo.ZoneInfoNotFoundError(...)can be raised, and base16/base32 decoding errors arebinascii.Error. - A module class's
__name__/__qualname__is its short name; namedtuple names and(?P<name>)groups accept Unicode identifiers;typing.NamedTuple('P', x=int)andfunctools.wraps(assigned=, updated=)use their keywords;get_type_hintstakes its parameters by name. - Arity errors use CPython's wording and name the bound actually crossed (
next(),dict.get(),deque.index(1, 2, 3, 4),filter(None),math.comb(1),[].insert(1)).'\UFFFFFFFF'is a SyntaxError;global xafter a use ofxin an f-string, a def...