Skip to content

Releases

PyLite 1.2.0

Choose a tag to compare

@sashokfestival sashokfestival released this 16 Nov 09:22

Three things changed since 1.1.0. A run now ends at its deadline whatever the script calls, and nothing of it keeps computing after Run returns. Memory a script holds is charged where it was not, so MaxAllocBytes bounds what the process keeps. And many answers, error types and texts moved to CPython 3.14's. The engine no longer limits how many runs go at once: that is the host's to decide.

dotnet add package PyLite --version 1.2.0

Every fix has its test, listed in docs/fixes.md. What still differs from CPython is in docs/deviations.md, and embedding the engine is described in the new hosting guide.

Upgrading from 1.1.0: hosts

what changed what to do
The engine no longer limits how many runs execute at once, or their memory together. EngineOptions.MaxConcurrentRuns and GlobalMemoryCeilingBytes are [Obsolete] and ignored A host that relied on them keeps its own limit around Run and sizes DefaultLimits for its parallelism; remove the properties (CS0618 under warnings-as-errors)
Run refuses a new run when Windows can no longer promise its stack, its MaxAllocBytes and 200 MB, and when its thread cannot start: HostError with LimitName "FreeMemory" or "ThreadStart" Treat both as "try later"
ScriptRunner.GetErrorLimit() and ScriptError.LimitName name a refusal ("FreeMemory", "ThreadStart", "CircuitBreaker") Tell "try later" from a failing host function by the name, not the message
Run(null) returns a HostError instead of throwing ArgumentNullException Read the result
RunCompleted fires once for every Run, also for a syntax error, a refusal and a leak A counter of runs may grow
Host failures end as HostError with a per-run copy of the exception; methods and properties built with SlotDescriptor are fenced like host functions; a host function's Thread.Abort ends the run, not the thread Do not match on EngineFault for them
Memory the process could not get while compiling or running is HostError / "FreeMemory", no longer EngineFault Treat it as "try later"; keep logging the EngineFault event
A source not in the compile cache compiles off the calling thread (1.1.0 compiled on the caller's stack and could refuse print(1) deep in it) Nothing; the answer no longer depends on the caller's stack
The compile cache keys on source and name and holds at most 2 million characters of source
JSON inputs are parsed before any prelude or provider code runs; ScalarsIn coercion is stricter (a sign and at most 4,300 ASCII digits make an int) Bad input fails earlier
The clock is validated: UtcNow must have DateTimeKind.Utc A clock returning DateTime.Now was silently wrong and is now a HostError
Host callbacks run in the invariant culture A callback that formats with the current culture sets its own
Function tables are copied and frozen on the caller's thread; HostFunctionTable.Set with a null function throws ArgumentNullException Later changes to a table do not reach a request already made
A token already cancelled when Run is called runs nothing; a run refused before it starts reports the refusal even when the source has a syntax error Read Error.Kind as before
A limit hit while a finally runs after an abort keeps the first abort's LimitName A host that maps LimitName to an action sees the cause that ended the run
A run is reported leaked only when it is stuck in host code past deadline + grace and made no progress for 5 s
Error types and texts follow CPython 3.14: re.error is re.PatternError, a missing module is ModuleNotFoundError, an unexpected indent is IndentationError, ZeroDivisionError says division by zero A host that matches Message, PythonType or Line should check them
Step and allocation charges moved, by up to about 30% for re matching, json and XML building; time.sleep charges one step per quantum Budgets sized on old runs may need another look
New: ScriptRunner.SetLimitIntBits, ScriptRunner.ConfigureSharedEngine

A run ends on its deadline

In 1.1.0 a few single calls into .NET could not be interrupted: the run overran its deadline by seconds, and past deadline + 5.25 s the engine gave up on the runner thread, which kept a core busy.

  • Big-int arithmetic runs in the engine's own kernel on the run's thread and checks the deadline every few milliseconds: Karatsuba multiply and square, Knuth division, Lehmer gcd, divide-and-conquer str(), sliding-window pow(b, e, m). There are no helper threads any more. It is faster too: at 4K bits a square ×1.9, divmod ×1.7, 3**20000 ×1.9; at a million bits it multiplies 14× faster than .NET's BigInteger.
  • Formatting: a float or Decimal formatted with a huge precision (format(x, '.1000000f')) pads zeros instead of building 10**precision, and a result past MaxStrChars is refused before it is built.
  • Regexes: the ones the engine runs on script data (csv.Sniffer, textwrap) obey the run's RegexPerOpTimeoutMs. A pattern that starts with a large count, such as re.compile('a{100000}'), compiles at once, and so does one whose nested loops .NET merges into such a count ((?:(?:a{60,}){60,}){20}): .NET used to spend seconds building a search table for them in one call the deadline could not stop.
  • Long work for one step: 28 kinds of loops that did long work for the one step they charged now end on their deadline. A method or constructor call is checked against the deadline when it returns; long string compares and searches, set and ChainMap walks, codec error handlers, case mappings of astral text, XPath and XML walks are charged as they go; keyword binding, dataclass fields, tee, deque deletion, StringIO reads, heapq.merge and long csv line terminators no longer take quadratic time.
  • No stack overflow: four scripts ended the host process at default limits (a 3,000-level [*[*[*…]]] while compiling, str() of an exception wrapped 200,000 times, repr of a deep itertools.repeat chain, <= of deeply nested Counters). They now end in a SyntaxError or a RecursionError.
  • Bounded where CPython is not: a tree that contains itself (a.append(a)) ends on MaxCollectionItems when walked; an XML tag with more than 10,000 attributes is refused; struct.unpack('0s' * n), combinations_with_replacement, product(repeat=n), slice assignment and list.insert end on MaxCollectionItems.

Memory that is now charged

  • Outputs are checked against MaxStrChars as they grow (ascii, format precisions, print, json escapes, strftime, str.format, f-strings), and lists grown from iterators are charged and capped (f(*it), random.choices, batched, XML extend). json.dumps and repr of a large list no longer keep two work-stack entries per item (850 MB and 1 GB for a 5-million-item list before).
  • Compiled regexes are charged what .NET keeps of them while they are alive: the pattern, its search table, its runner and, for a loop of groups, its backtracking stack. A script that keeps 50,000 compiled patterns in a list meets a 32 MB MaxAllocBytes; patterns a loop compiles and drops are given back once collected. A match object is charged 64 bytes and 32 a group, so a script that makes millions of many-group matches needs a larger MaxAllocBytes than before.
  • XML: an element without attributes holds no attrib dict until a script asks for it: a tree of plain elements needs a third of the memory and charges a quarter less, an attribute-heavy one charges up to half more, which is closer to what it holds. ElementPath's parent maps are charged while held and given back when a search ends; a loop that adds a row and finds row[last()] builds no map per search (6,000 rows on 2.4M steps; 18M before).
  • A method call on a namedtuple no longer keeps the run's type in the engine's script cache, and ScriptError.ScriptTraceback is capped at 262,144 characters.

Scripts that may notice

Answers that now equal CPython 3.14's, so a script that relied on the old one sees a difference:

  • Floats read and print as in CPython: a decimal text, an int and a Decimal become the nearest double, and repr gives the shortest digits that read back. repr was wrong for about 2% of everyday values (9.37631e-113 printed as 9.376309999999999e-113), so some floats, random() among them, print a different last digit.
  • str(), repr(), f-strings, %d and format() of an int of more than 4,300 decimal digits raise ValueError, and so does int() of such a string. The dialect has no sys, so the limit cannot be raised; hex(), bin(), to_bytes() and Decimal are not limited.
  • math.hypot, dist, sumprod, expm1, sin, cos, tan and sum of floats: last digits. Decimal ** n, //, % and divmod: the last digit of a power, whole digits of a 29-digit dividend.
  • re: IGNORECASE folds as sre does (the Kelvin sign matches k, ı and İ match i, ſ matches s; under re.A only ASCII letters fold); \w, \W, \b, \B follow Unicode 16; \123 after a group is an octal character and \10 with ten groups is group 10; in re.sub templates \0 is a character, not group 0, and \g<name> of a missing group is an IndexError; \UHHHHHHHH names its character; a bytes pattern is a TypeError naming the dialect.
  • Arguments that were dropped silently are a TypeError: an unknown keyword to dozens of functions and methods, a parameter given by position and by name, an extra positional (json.dumps(1, 2)), a keyword to an exception class. Many functions take their first pa...
Read more

PyLite 1.1.0

Choose a tag to compare

@sashokfestival sashokfestival released this 29 Oct 20:33

PyLite 1.1.0 adds the parts of Python that integration scripts most often needed (with, zipfile, enum, class constants, super(), io.BytesIO), makes sorting and several hot paths faster, closes every resource-limit gap that an audit with hostile scripts found, and puts the whole test suite into this repository.

dotnet add package PyLite --version 1.1.0

Upgrading from 1.0.0

Everything below is additive or a fix, but these can change what an existing host or script sees:

  • Host API: the module implementation classes (ReModule, JsonModule, CsvModule, ..., FloatRepr, NumericHash) are internal; they were public by accident and never documented.
  • Limits that now hold: SleepTotalMs (a sleep past it fails the run; each sleep used to be cut at 5 s silently), a per-run RegexPerOpTimeoutMs (patterns used a fixed 250 ms), MaxCollectionItems for dict and set, and the hard caps MaxDataDepth 10,000 and MaxIntBits 4,000,000, which EngineOptions.DefaultLimits is clamped to as well. int() of an over-long string ends as MaxIntBits (it was MaxSteps).
  • Scripts: json.dumps of too-deep data raises RecursionError (was ValueError); type(e) of an exception is its class; a @dataclass subclass runs its generated __init__ even when a base wrote one; builtins and methods refuse an extra argument or an unknown keyword they used to ignore; a Decimal literal too long for the backend is rounded instead of raising Overflow; def/class/import of a nonlocal name bind that name, not a global.

New in the dialect

  • with: one or several managers, the parenthesized form, classes with __enter__/__exit__; io.StringIO/io.BytesIO are managers. A resource limit still ends the run whatever __exit__ answers.
  • io.BytesIO: an in-memory binary file with CPython's surface (read, readline(s), iteration, write, writelines, seek, tell, truncate, getvalue); json.load, ET.parse and ElementTree.write work with it. io.StringIO gained truncate, writelines, the readable/writable/seekable flags, next(f), and zero padding past the end; both buffers are capped at MaxStrChars.
  • zipfile: read, write and append ZIP archives in an io.BytesIO (STORED and DEFLATED, ZIP64 records, open(name, 'w'), comments, testzip, printdir); archives are interchangeable with .NET's ZipArchive. Zip bombs end the run on the budget: inflation is charged and stops at the recorded size, entries are capped, overlapped entries are refused.
  • Class bodies take constants (NEW = 'new', read as Status.NEW and self.NEW, shadowed by subclasses, settable on the class) and docstrings; __match_args__, __slots__ and __hash__ = None work. Exception subclasses may carry constants too (class ApiError(Exception): CODE = 500, e.CODE).
  • super() and super(Class, self) in methods, __class__, self.__class__; type(obj) of a class instance is its class, and type(e) is ValueError holds for an exception. super().__init__(...) reaches a base's __init__, the one @dataclass generated, or object.__init__.
  • enum: Enum, IntEnum, StrEnum, auto(), @unique, Enum('Name', 'A B C'), and the class Status(str, Enum) / (int, Enum) form. Members work in match, as dict keys and in sets; IntEnum/StrEnum members compute and compare as ints/strs and json.dumps writes their values.
  • contextlib.suppress, nullcontext, closing; decimal.localcontext() (also prec=/rounding=), decimal.setcontext and decimal.Context(prec, rounding): a rounding block restores the previous context however it ends. suppress cannot keep a run alive past a resource limit.
  • string.Template (substitute, safe_substitute, $name/${name}/$$, get_identifiers, is_valid, subclasses with their own delimiter, idpattern, braceidpattern, flags or pattern) and the email.utils date helpers (formatdate, format_datetime, parsedate, parsedate_tz, parsedate_to_datetime, mktime_tz): CPython's own code, running as script code.
  • Decimal('1_000.5') and Fraction('1_000/3') accept PEP 515 underscores, as CPython does. New builtin EOFError; obj.attr += x works on engine objects that allow the attribute.

Faster

  • Sorting (sorted, list.sort and everything built on them) is 1.3–3× faster: presorted and reversed data take n−1 comparisons, int/str/float keys are compared without boxing. A comparison or key that raises no longer leaves the list empty.
  • Dict and set comprehensions over a range, a set or a dict are about twice as fast and allocate 45–60% less for large results; every dict/set insert probes once instead of twice.
  • re: match() is anchored at pos (a failing match on a long string no longer scans it: ×150 on 10 KB); group reads are 10–30% faster.
  • A function whose body is one return <expr> runs on the path lambdas use: about 20% faster per call, tracebacks unchanged.
  • Against IronPython 3.4.2 on the same scripts (docs/performance.md, samples in benchmarks/): money in decimal ×65–73 faster, JSON ×13–14 faster, a new engine in a warm process ×187–226 faster; sorted with a key ×1.08–1.36 slower (it was ×2.6–3.3 in 1.0.0).

Resource limits

From an audit with hostile scripts:

  • A host that raises MaxDataDepth can no longer have the process killed by a stack overflow (repr of ~200k nested deques); the run ends as StackDepth, and MaxDataDepth has a hard cap of 10,000. EngineOptions.DefaultLimits obeys the engine hard caps also for a run that sets no limits of its own.
  • SleepTotalMs works: a sleep past it fails the run at once, real sleep matches the request, and RunStatistics.SleepMs is reported. RegexPerOpTimeoutMs set for a run is honoured (a run's value can only lower the engine's DefaultLimits). A dict or set stops on MaxCollectionItems like a list.
  • Nested functools.partial is charged (100k levels ran 34 s and took 1.3 GB). int() of more digits than MaxIntBits holds ends as MaxIntBits, and 7**350000 or 2**999999, which fit, are no longer refused. MaxIntBits has a hard cap of 4,000,000 bits.
  • int.bit_length()/bit_count() of a huge int, Decimal and Fraction literals of a million digits, pow(b, e, m) and pow(b, -1, m) with huge numbers, math.isqrt (now CPython's algorithm), Fraction arithmetic with huge parts and int.from_bytes/to_bytes (now linear: to_bytes of a 786k-bit int took 12 s, now 0.8 s) no longer run past the deadline. A single long big-int operation no longer delays the end of an aborted run by its own length: it runs on a helper thread while the run polls its deadline.
  • Outputs are capped as they grow, not built first: str.replace(''), %-formatting, json.dumps, re.sub with a function, ET.tostring, textwrap.indent, bytes.join, itertools.tee, calendar.month. Grouping digits of a huge int, draining a set/dict with pop()/popitem() and del a[::2] are linear; str.expandtabs no longer hits the memory limit spuriously; format(x, '.100000f') no longer grows memory shared by every later run.

Fixes

  • A wrong argument is a catchable error, never an internal engine error: random.choice(), ET.fromstring(b'\xff'), uuid.UUID(version='4'), b'..'.replace(a, b, None) and others. Builtins, container methods and math refuse an extra argument or a keyword they do not take (math.sqrt(4, 5), 'x'.strip(chars='x') were silently accepted); pow(x, y, mod=), bytes(s, encoding=), replace(count=), base64.b64encode(altchars=), b64decode(validate=), binascii.hexlify(sep=) and hmac.new(msg=) now use the keyword. isinstance/issubclass no longer fault when a class is checked against a builtin type.
  • def/class/import/except as of a nonlocal name binds that name (it bound a global), and del of a captured or nonlocal variable works.
  • As CPython: 1 // 0.1 == 9.0, -6.0 % 3 == 0.0, float('-0.0') keeps its sign, bytes find/count/replace with an empty needle, type(p) is P for a namedtuple, lru_cache(partial(...)), calendar.IllegalMonthError(13), statistics.harmonic_mean/linear_regression of iterators, asdict of a cyclic dataclass is a RecursionError.
  • As CPython: json.dumps of data nested past MaxDataDepth raises RecursionError (it was ValueError); a default= hook that returns the object it was given is ValueError: Circular reference detected, one that returns a new object every time is that RecursionError.
  • As CPython, found by the differential corpus: a Decimal sum that is exactly zero takes the addends' sign (Decimal('1.30') - Decimal('1.3') was Decimal('-0.00')); csv.writer escapes as CPython does (with doublequote=False a quote is escaped, not quoted; the escape character is escaped inside quotes too; a line-terminator character quotes the field); functools.lru_cache(typed=True) keys on the argument types and maxsize=0 caches nothing.
  • A module's exception class is one object for the engine's lifetime, so type(e) is re.error (and json.JSONDecodeError, struct.error, zlib.error, csv.Error, ...) holds, zoneinfo.ZoneInfoNotFoundError(...) can be raised, and base16/base32 decoding errors are binascii.Error.
  • A module class's __name__/__qualname__ is its short name; namedtuple names and (?P<name>) groups accept Unicode identifiers; typing.NamedTuple('P', x=int) and functools.wraps(assigned=, updated=) use their keywords; get_type_hints takes its parameters by name.
  • Arity errors use CPython's wording and name the bound actually crossed (next(), dict.get(), deque.index(1, 2, 3, 4), filter(None), math.comb(1), [].insert(1)). '\UFFFFFFFF' is a SyntaxError; global x after a use of x in an f-string, a def...
Read more