Repository navigation
PyLite 1.1.0
PyLite 1.1.0 adds the parts of Python that integration scripts most often needed (with, zipfile, enum, class constants, super(), io.BytesIO), makes sorting and several hot paths faster, closes every resource-limit gap that an audit with hostile scripts found, and puts the whole test suite into this repository.
dotnet add package PyLite --version 1.1.0
Upgrading from 1.0.0
Everything below is additive or a fix, but these can change what an existing host or script sees:
- Host API: the module implementation classes (
ReModule,JsonModule,CsvModule, ...,FloatRepr,NumericHash) areinternal; they were public by accident and never documented. - Limits that now hold:
SleepTotalMs(a sleep past it fails the run; each sleep used to be cut at 5 s silently), a per-runRegexPerOpTimeoutMs(patterns used a fixed 250 ms),MaxCollectionItemsfordictandset, and the hard capsMaxDataDepth10,000 andMaxIntBits4,000,000, whichEngineOptions.DefaultLimitsis clamped to as well.int()of an over-long string ends asMaxIntBits(it wasMaxSteps). - Scripts:
json.dumpsof too-deep data raisesRecursionError(wasValueError);type(e)of an exception is its class; a@dataclasssubclass runs its generated__init__even when a base wrote one; builtins and methods refuse an extra argument or an unknown keyword they used to ignore; aDecimalliteral too long for the backend is rounded instead of raisingOverflow;def/class/importof anonlocalname bind that name, not a global.
New in the dialect
with: one or several managers, the parenthesized form, classes with__enter__/__exit__;io.StringIO/io.BytesIOare managers. A resource limit still ends the run whatever__exit__answers.io.BytesIO: an in-memory binary file with CPython's surface (read, readline(s), iteration, write, writelines, seek, tell, truncate, getvalue);json.load,ET.parseandElementTree.writework with it.io.StringIOgainedtruncate,writelines, thereadable/writable/seekableflags,next(f), and zero padding past the end; both buffers are capped atMaxStrChars.zipfile: read, write and append ZIP archives in anio.BytesIO(STORED and DEFLATED, ZIP64 records,open(name, 'w'), comments,testzip,printdir); archives are interchangeable with .NET'sZipArchive. Zip bombs end the run on the budget: inflation is charged and stops at the recorded size, entries are capped, overlapped entries are refused.- Class bodies take constants (
NEW = 'new', read asStatus.NEWandself.NEW, shadowed by subclasses, settable on the class) and docstrings;__match_args__,__slots__and__hash__ = Nonework. Exception subclasses may carry constants too (class ApiError(Exception): CODE = 500,e.CODE). super()andsuper(Class, self)in methods,__class__,self.__class__;type(obj)of a class instance is its class, andtype(e) is ValueErrorholds for an exception.super().__init__(...)reaches a base's__init__, the one@dataclassgenerated, orobject.__init__.enum:Enum,IntEnum,StrEnum,auto(),@unique,Enum('Name', 'A B C'), and theclass Status(str, Enum)/(int, Enum)form. Members work inmatch, as dict keys and in sets;IntEnum/StrEnummembers compute and compare as ints/strs andjson.dumpswrites their values.contextlib.suppress,nullcontext,closing;decimal.localcontext()(alsoprec=/rounding=),decimal.setcontextanddecimal.Context(prec, rounding): a rounding block restores the previous context however it ends.suppresscannot keep a run alive past a resource limit.string.Template(substitute,safe_substitute,$name/${name}/$$,get_identifiers,is_valid, subclasses with their owndelimiter,idpattern,braceidpattern,flagsorpattern) and theemail.utilsdate helpers (formatdate,format_datetime,parsedate,parsedate_tz,parsedate_to_datetime,mktime_tz): CPython's own code, running as script code.Decimal('1_000.5')andFraction('1_000/3')accept PEP 515 underscores, as CPython does. New builtinEOFError;obj.attr += xworks on engine objects that allow the attribute.
Faster
- Sorting (
sorted,list.sortand everything built on them) is 1.3–3× faster: presorted and reversed data take n−1 comparisons, int/str/float keys are compared without boxing. A comparison or key that raises no longer leaves the list empty. - Dict and set comprehensions over a range, a set or a dict are about twice as fast and allocate 45–60% less for large results; every dict/set insert probes once instead of twice.
re:match()is anchored atpos(a failing match on a long string no longer scans it: ×150 on 10 KB); group reads are 10–30% faster.- A function whose body is one
return <expr>runs on the path lambdas use: about 20% faster per call, tracebacks unchanged. - Against IronPython 3.4.2 on the same scripts (docs/performance.md, samples in benchmarks/): money in
decimal×65–73 faster, JSON ×13–14 faster, a new engine in a warm process ×187–226 faster;sortedwith akey×1.08–1.36 slower (it was ×2.6–3.3 in 1.0.0).
Resource limits
From an audit with hostile scripts:
- A host that raises
MaxDataDepthcan no longer have the process killed by a stack overflow (reprof ~200k nested deques); the run ends asStackDepth, andMaxDataDepthhas a hard cap of 10,000.EngineOptions.DefaultLimitsobeys the engine hard caps also for a run that sets no limits of its own. SleepTotalMsworks: a sleep past it fails the run at once, real sleep matches the request, andRunStatistics.SleepMsis reported.RegexPerOpTimeoutMsset for a run is honoured (a run's value can only lower the engine'sDefaultLimits). Adictorsetstops onMaxCollectionItemslike a list.- Nested
functools.partialis charged (100k levels ran 34 s and took 1.3 GB).int()of more digits thanMaxIntBitsholds ends asMaxIntBits, and7**350000or2**999999, which fit, are no longer refused.MaxIntBitshas a hard cap of 4,000,000 bits. int.bit_length()/bit_count()of a huge int,DecimalandFractionliterals of a million digits,pow(b, e, m)andpow(b, -1, m)with huge numbers,math.isqrt(now CPython's algorithm), Fraction arithmetic with huge parts andint.from_bytes/to_bytes(now linear:to_bytesof a 786k-bit int took 12 s, now 0.8 s) no longer run past the deadline. A single long big-int operation no longer delays the end of an aborted run by its own length: it runs on a helper thread while the run polls its deadline.- Outputs are capped as they grow, not built first:
str.replace(''),%-formatting,json.dumps,re.subwith a function,ET.tostring,textwrap.indent,bytes.join,itertools.tee,calendar.month. Grouping digits of a huge int, draining a set/dict withpop()/popitem()anddel a[::2]are linear;str.expandtabsno longer hits the memory limit spuriously;format(x, '.100000f')no longer grows memory shared by every later run.
Fixes
- A wrong argument is a catchable error, never an internal engine error:
random.choice(),ET.fromstring(b'\xff'),uuid.UUID(version='4'),b'..'.replace(a, b, None)and others. Builtins, container methods andmathrefuse an extra argument or a keyword they do not take (math.sqrt(4, 5),'x'.strip(chars='x')were silently accepted);pow(x, y, mod=),bytes(s, encoding=),replace(count=),base64.b64encode(altchars=),b64decode(validate=),binascii.hexlify(sep=)andhmac.new(msg=)now use the keyword.isinstance/issubclassno longer fault when a class is checked against a builtin type. def/class/import/except asof anonlocalname binds that name (it bound a global), anddelof a captured ornonlocalvariable works.- As CPython:
1 // 0.1 == 9.0,-6.0 % 3 == 0.0,float('-0.0')keeps its sign,bytesfind/count/replace with an empty needle,type(p) is Pfor a namedtuple,lru_cache(partial(...)),calendar.IllegalMonthError(13),statistics.harmonic_mean/linear_regressionof iterators,asdictof a cyclic dataclass is aRecursionError. - As CPython:
json.dumpsof data nested pastMaxDataDepthraisesRecursionError(it wasValueError); adefault=hook that returns the object it was given isValueError: Circular reference detected, one that returns a new object every time is thatRecursionError. - As CPython, found by the differential corpus: a
Decimalsum that is exactly zero takes the addends' sign (Decimal('1.30') - Decimal('1.3')wasDecimal('-0.00'));csv.writerescapes as CPython does (withdoublequote=Falsea quote is escaped, not quoted; the escape character is escaped inside quotes too; a line-terminator character quotes the field);functools.lru_cache(typed=True)keys on the argument types andmaxsize=0caches nothing. - A module's exception class is one object for the engine's lifetime, so
type(e) is re.error(andjson.JSONDecodeError,struct.error,zlib.error,csv.Error, ...) holds,zoneinfo.ZoneInfoNotFoundError(...)can be raised, and base16/base32 decoding errors arebinascii.Error. - A module class's
__name__/__qualname__is its short name; namedtuple names and(?P<name>)groups accept Unicode identifiers;typing.NamedTuple('P', x=int)andfunctools.wraps(assigned=, updated=)use their keywords;get_type_hintstakes its parameters by name. - Arity errors use CPython's wording and name the bound actually crossed (
next(),dict.get(),deque.index(1, 2, 3, 4),filter(None),math.comb(1),[].insert(1)).'\UFFFFFFFF'is a SyntaxError;global xafter a use ofxin an f-string, a default or a comprehension's first iterable is a SyntaxError;for self.x in ...in__init__sets an attribute;statistics.median_groupedtakes Decimal data;re.match(p, s, 0, 1)refuses an extra positional argument; hmac objects havedigest_sizeandblock_size. RunStatistics.RegexTimeandRegexOpsare reported (they were always zero), and a run that ends in an engine fault keeps its host-call time and elapsed time in the statistics. A callback from a Python prelude module into the main script runs on the main script's evaluator.
Every fixed defect is listed with the test that holds it in docs/fixes.md.
Tests and documentation
The tests are now in this repository: about 7,000 xUnit tests in five categories, 174 conformance cases adapted from CPython's Lib/test, 87 scripts compared with IronPython 3.4.2, and 130 adversarial probes. TESTING.md says how to run them; tools/run-gates.ps1 runs all of them and wrote docs/test-report.md for this release. SECURITY.md maps every sandbox guarantee to the tests that hold it, and examples/limits has one script per resource limit. docs/modules.md is now generated from the engine and docs/deviations.md is updated for 1.1.0.