Skip to content

PyLite 1.1.0

Choose a tag to compare

@sashokfestival sashokfestival released this 29 Oct 20:33
· 5 commits to main since this release

PyLite 1.1.0 adds the parts of Python that integration scripts most often needed (with, zipfile, enum, class constants, super(), io.BytesIO), makes sorting and several hot paths faster, closes every resource-limit gap that an audit with hostile scripts found, and puts the whole test suite into this repository.

dotnet add package PyLite --version 1.1.0

Upgrading from 1.0.0

Everything below is additive or a fix, but these can change what an existing host or script sees:

  • Host API: the module implementation classes (ReModule, JsonModule, CsvModule, ..., FloatRepr, NumericHash) are internal; they were public by accident and never documented.
  • Limits that now hold: SleepTotalMs (a sleep past it fails the run; each sleep used to be cut at 5 s silently), a per-run RegexPerOpTimeoutMs (patterns used a fixed 250 ms), MaxCollectionItems for dict and set, and the hard caps MaxDataDepth 10,000 and MaxIntBits 4,000,000, which EngineOptions.DefaultLimits is clamped to as well. int() of an over-long string ends as MaxIntBits (it was MaxSteps).
  • Scripts: json.dumps of too-deep data raises RecursionError (was ValueError); type(e) of an exception is its class; a @dataclass subclass runs its generated __init__ even when a base wrote one; builtins and methods refuse an extra argument or an unknown keyword they used to ignore; a Decimal literal too long for the backend is rounded instead of raising Overflow; def/class/import of a nonlocal name bind that name, not a global.

New in the dialect

  • with: one or several managers, the parenthesized form, classes with __enter__/__exit__; io.StringIO/io.BytesIO are managers. A resource limit still ends the run whatever __exit__ answers.
  • io.BytesIO: an in-memory binary file with CPython's surface (read, readline(s), iteration, write, writelines, seek, tell, truncate, getvalue); json.load, ET.parse and ElementTree.write work with it. io.StringIO gained truncate, writelines, the readable/writable/seekable flags, next(f), and zero padding past the end; both buffers are capped at MaxStrChars.
  • zipfile: read, write and append ZIP archives in an io.BytesIO (STORED and DEFLATED, ZIP64 records, open(name, 'w'), comments, testzip, printdir); archives are interchangeable with .NET's ZipArchive. Zip bombs end the run on the budget: inflation is charged and stops at the recorded size, entries are capped, overlapped entries are refused.
  • Class bodies take constants (NEW = 'new', read as Status.NEW and self.NEW, shadowed by subclasses, settable on the class) and docstrings; __match_args__, __slots__ and __hash__ = None work. Exception subclasses may carry constants too (class ApiError(Exception): CODE = 500, e.CODE).
  • super() and super(Class, self) in methods, __class__, self.__class__; type(obj) of a class instance is its class, and type(e) is ValueError holds for an exception. super().__init__(...) reaches a base's __init__, the one @dataclass generated, or object.__init__.
  • enum: Enum, IntEnum, StrEnum, auto(), @unique, Enum('Name', 'A B C'), and the class Status(str, Enum) / (int, Enum) form. Members work in match, as dict keys and in sets; IntEnum/StrEnum members compute and compare as ints/strs and json.dumps writes their values.
  • contextlib.suppress, nullcontext, closing; decimal.localcontext() (also prec=/rounding=), decimal.setcontext and decimal.Context(prec, rounding): a rounding block restores the previous context however it ends. suppress cannot keep a run alive past a resource limit.
  • string.Template (substitute, safe_substitute, $name/${name}/$$, get_identifiers, is_valid, subclasses with their own delimiter, idpattern, braceidpattern, flags or pattern) and the email.utils date helpers (formatdate, format_datetime, parsedate, parsedate_tz, parsedate_to_datetime, mktime_tz): CPython's own code, running as script code.
  • Decimal('1_000.5') and Fraction('1_000/3') accept PEP 515 underscores, as CPython does. New builtin EOFError; obj.attr += x works on engine objects that allow the attribute.

Faster

  • Sorting (sorted, list.sort and everything built on them) is 1.3–3× faster: presorted and reversed data take n−1 comparisons, int/str/float keys are compared without boxing. A comparison or key that raises no longer leaves the list empty.
  • Dict and set comprehensions over a range, a set or a dict are about twice as fast and allocate 45–60% less for large results; every dict/set insert probes once instead of twice.
  • re: match() is anchored at pos (a failing match on a long string no longer scans it: ×150 on 10 KB); group reads are 10–30% faster.
  • A function whose body is one return <expr> runs on the path lambdas use: about 20% faster per call, tracebacks unchanged.
  • Against IronPython 3.4.2 on the same scripts (docs/performance.md, samples in benchmarks/): money in decimal ×65–73 faster, JSON ×13–14 faster, a new engine in a warm process ×187–226 faster; sorted with a key ×1.08–1.36 slower (it was ×2.6–3.3 in 1.0.0).

Resource limits

From an audit with hostile scripts:

  • A host that raises MaxDataDepth can no longer have the process killed by a stack overflow (repr of ~200k nested deques); the run ends as StackDepth, and MaxDataDepth has a hard cap of 10,000. EngineOptions.DefaultLimits obeys the engine hard caps also for a run that sets no limits of its own.
  • SleepTotalMs works: a sleep past it fails the run at once, real sleep matches the request, and RunStatistics.SleepMs is reported. RegexPerOpTimeoutMs set for a run is honoured (a run's value can only lower the engine's DefaultLimits). A dict or set stops on MaxCollectionItems like a list.
  • Nested functools.partial is charged (100k levels ran 34 s and took 1.3 GB). int() of more digits than MaxIntBits holds ends as MaxIntBits, and 7**350000 or 2**999999, which fit, are no longer refused. MaxIntBits has a hard cap of 4,000,000 bits.
  • int.bit_length()/bit_count() of a huge int, Decimal and Fraction literals of a million digits, pow(b, e, m) and pow(b, -1, m) with huge numbers, math.isqrt (now CPython's algorithm), Fraction arithmetic with huge parts and int.from_bytes/to_bytes (now linear: to_bytes of a 786k-bit int took 12 s, now 0.8 s) no longer run past the deadline. A single long big-int operation no longer delays the end of an aborted run by its own length: it runs on a helper thread while the run polls its deadline.
  • Outputs are capped as they grow, not built first: str.replace(''), %-formatting, json.dumps, re.sub with a function, ET.tostring, textwrap.indent, bytes.join, itertools.tee, calendar.month. Grouping digits of a huge int, draining a set/dict with pop()/popitem() and del a[::2] are linear; str.expandtabs no longer hits the memory limit spuriously; format(x, '.100000f') no longer grows memory shared by every later run.

Fixes

  • A wrong argument is a catchable error, never an internal engine error: random.choice(), ET.fromstring(b'\xff'), uuid.UUID(version='4'), b'..'.replace(a, b, None) and others. Builtins, container methods and math refuse an extra argument or a keyword they do not take (math.sqrt(4, 5), 'x'.strip(chars='x') were silently accepted); pow(x, y, mod=), bytes(s, encoding=), replace(count=), base64.b64encode(altchars=), b64decode(validate=), binascii.hexlify(sep=) and hmac.new(msg=) now use the keyword. isinstance/issubclass no longer fault when a class is checked against a builtin type.
  • def/class/import/except as of a nonlocal name binds that name (it bound a global), and del of a captured or nonlocal variable works.
  • As CPython: 1 // 0.1 == 9.0, -6.0 % 3 == 0.0, float('-0.0') keeps its sign, bytes find/count/replace with an empty needle, type(p) is P for a namedtuple, lru_cache(partial(...)), calendar.IllegalMonthError(13), statistics.harmonic_mean/linear_regression of iterators, asdict of a cyclic dataclass is a RecursionError.
  • As CPython: json.dumps of data nested past MaxDataDepth raises RecursionError (it was ValueError); a default= hook that returns the object it was given is ValueError: Circular reference detected, one that returns a new object every time is that RecursionError.
  • As CPython, found by the differential corpus: a Decimal sum that is exactly zero takes the addends' sign (Decimal('1.30') - Decimal('1.3') was Decimal('-0.00')); csv.writer escapes as CPython does (with doublequote=False a quote is escaped, not quoted; the escape character is escaped inside quotes too; a line-terminator character quotes the field); functools.lru_cache(typed=True) keys on the argument types and maxsize=0 caches nothing.
  • A module's exception class is one object for the engine's lifetime, so type(e) is re.error (and json.JSONDecodeError, struct.error, zlib.error, csv.Error, ...) holds, zoneinfo.ZoneInfoNotFoundError(...) can be raised, and base16/base32 decoding errors are binascii.Error.
  • A module class's __name__/__qualname__ is its short name; namedtuple names and (?P<name>) groups accept Unicode identifiers; typing.NamedTuple('P', x=int) and functools.wraps(assigned=, updated=) use their keywords; get_type_hints takes its parameters by name.
  • Arity errors use CPython's wording and name the bound actually crossed (next(), dict.get(), deque.index(1, 2, 3, 4), filter(None), math.comb(1), [].insert(1)). '\UFFFFFFFF' is a SyntaxError; global x after a use of x in an f-string, a default or a comprehension's first iterable is a SyntaxError; for self.x in ... in __init__ sets an attribute; statistics.median_grouped takes Decimal data; re.match(p, s, 0, 1) refuses an extra positional argument; hmac objects have digest_size and block_size.
  • RunStatistics.RegexTime and RegexOps are reported (they were always zero), and a run that ends in an engine fault keeps its host-call time and elapsed time in the statistics. A callback from a Python prelude module into the main script runs on the main script's evaluator.

Every fixed defect is listed with the test that holds it in docs/fixes.md.

Tests and documentation

The tests are now in this repository: about 7,000 xUnit tests in five categories, 174 conformance cases adapted from CPython's Lib/test, 87 scripts compared with IronPython 3.4.2, and 130 adversarial probes. TESTING.md says how to run them; tools/run-gates.ps1 runs all of them and wrote docs/test-report.md for this release. SECURITY.md maps every sandbox guarantee to the tests that hold it, and examples/limits has one script per resource limit. docs/modules.md is now generated from the engine and docs/deviations.md is updated for 1.1.0.