v0.8.19

Built against Mojo 1.0.0 (ed45d567).

A patch release with thirty six entries in it, most of them about what a pattern and a statement are allowed to say, and a few about what the pipeline does with a chunk. On the regular expression side nine spellings that used to be refused now read, a pattern is read with whichever grammar is going to run it, and the Unicode name table is checked against RE2 rather than against a list somebody wrote out by hand, which is what turned up four names it had never heard of. The SQL front end reads IS UNKNOWN, ARRAY over a subquery, WITH ORDINALITY, TABLESAMPLE, COLUMNS(), an E string with backslash escapes, USING KEY, OVERLAY and the colon spelling of an alias.

In the pipeline the filter operator is the whole story. It reads its mask in blocks whatever the mask looks like, skips a block of rows it is going to drop whole, hands the chunk straight back when its comparison kept everything, and writes a selection rather than a copy when the thing that reads it is another filter. The last two are worth between 1.07x and 4.70x of CPU on the seven ClickBench date range queries, which lower to six filters in a row over a chunk holding URL.

The rest is CI. A pull request finishes in about six minutes rather than about fifty five, the compiler's cache is kept between runs, and the regular expression comparisons run on the merge.

Nothing in the API breaks, and every query that had an answer before gives the same answer now.

Added: IS UNKNOWN runs, as the IS NULL it is

x IS UNKNOWN is the standard's way of writing x IS NULL, and firepanda used to turn it down and say to write the other one. It runs now, at 4 statements of the corpus.

It folds in the transformer rather than becoming a node of its own. The standard reads it only over a boolean, where it means the third truth value, but DuckDB takes it over any type and answers exactly what IS NULL answers, down to naming the column (1 IS NULL) when the query wrote 1 IS UNKNOWN. So there is one test with two spellings, the AST keeps the one it already had, and the printer writes IS NULL back. Lowering and execution were already there and needed nothing.

That makes it the first of these to go all the way to an answer rather than as far as the printer. The last few read and print and stop at lowering, because the thing they mean has no node yet. This one means something firepanda already runs.

Changed: a filter with another filter above it writes a selection whatever it keeps

A filter that keeps more than SELECTION_KEEP_LIMIT of a chunk copies the rows that survived instead of writing a selection, because past that share whatever reads the chunk pays more to gather through scattered positions than the copy costs here. That trade was measured with a projection above the filter, and it does not survive the thing above being another filter: the copy is one the next filter makes again over nearly the same rows, and the one above that makes it a third time.

A conjunction lowers to one filter per conjunct, so this is the ordinary shape of a WHERE clause with several conditions in it. Seven of the ClickBench statements ask for a month of traffic from one counter and read URL above it, and CounterID = 62 keeps 41.4 percent of the 1M partition, just past the threshold, so it copied every column the group by reads. Every condition after it then met a flat chunk and copied the same columns again.

mark_chained_filters runs once when the pipeline has the whole line, which is the first point at which a node's successor is known, and it marks every filter that has another filter directly above it. The last of a run is left alone, because what reads that one is not a filter and the threshold is about the reader. So six filters in a line become five that compose, four bytes a surviving row and no column touched, and one that decides the way it always did.

Measured over ClickBench at 1M on a quiet machine, CPU per run in milliseconds: q38 59.4 to 12.6, q41 21.6 to 6.3, q40 18.4 to 11.8, q42 27.2 to 20.1, q37 80.0 to 62.8, q36 149.1 to 121.9, q39 312.1 to 290.6, and 3083.0 to 2938.4 over all 43. Every query returns the rows, sums and hashes it returned before. Six queries read slower in that pass and none of them stayed slower when they were measured again three rounds at a time against the same two binaries, q31 the worst of them at 0.72x the first time and 1.08x the second.

Added: the colon spelling of a table alias is read as the alias it is

FROM x: t is FROM t AS x with the name written first. DuckDB takes one spelling or the other and not both, FROM x: t AS y is a syntax error there, so the two can never arrive together and they mean the same reference. firepanda used to read the colon on a select item and turn it down on a table, which left 12 statements of the corpus refused for a spelling it already understood everywhere else.

The alias reader now takes both nodes and at most one of them is there, so every reference that takes an alias takes this one. That is six shapes and not just the table: a plain name, a qualified name, a table function, a subquery, a VALUES list and a parenthesised join. They all come out of the printer in the AS form, which is what the round trip needs, since there is one node for the alias and it is written the one way.

The colon spelling carries no column names, so there is nothing to read beside it and nothing new in the AST. The refusal goes away rather than moving to lowering, because by the time lowering sees the reference there is no colon left in it.

Changed: CI keeps the compiler's cache between runs

The compiler writes down what it has already built, under $MODULAR_HOME, which pixi puts inside .pixi. That directory was already being cached, but keyed on pixi.lock, and actions/cache only writes an entry when its key missed. The lockfile almost never changes, so the key almost always hit, and every run had been restoring whatever happened to be in there the last time the lockfile moved.

It is now its own cache entry with a key that is different on every run and a prefix to restore the newest by. On the workstation tests/test_sql_run.mojo takes 195 seconds against a cold cache, 22 against a warm one, and 61 when the change under test reaches run itself, with all 422 tests passing in each.

The middle number is not the cache answering from yesterday. Putting a raise at the top of run turned the warm run into 422 failures in 61 seconds, and taking it out again went back to 422 passes in 22. What the cache keys on is what a change actually reaches, not the file it was written in.

Every job that compiles Mojo gets it, not just the tests, because with the tests down the next longest jobs in the pipeline are the two differential ones and the extension build. Each of them keeps its own entry rather than sharing one, and the test job keeps one per shard. One entry shared by all of them would be whichever job finished last, and every job would carry the other jobs' work around for nothing.

Restored on every run and saved only on the merge. A pull request gets the cache the merge built and uploads nothing, which keeps a couple of hundred megabytes a shard out of the repository's cache budget and off the clock of whoever is waiting.

Fixed: two notes about test compile time that said the wrong thing

tools/run_tests.sh said a split of tests/test_sql_run.mojo failed because each part still imports firepanda.sql.run. Importing that module and calling nothing costs 15 seconds of the 195. What each part still does is call run, and one call is the whole price: a generated file that calls it once costs 124 seconds, four times costs 123, and sixty four times costs 120. The first query instantiates the engine and every query after it is free.

.github/workflows/ci.yml said there was nothing to cache between test files. That was true of handing them a precompiled package, which is still measured and still does not help, and not true of the compiler's cache, which is a different thing.

Added: ARRAY over a subquery reads and prints

ARRAY(SELECT ...) collects a whole column into one list value, and firepanda used to turn the query down at the parse. It reads and prints now, at 25 statements of the corpus, and lowering is where it stops.

The grammar puts all three spellings under one rule. [a, b] and ARRAY[a, b] write their elements out and are the same thing said two ways, so the printer picks the bracket. ARRAY(SELECT ...) names a query rather than writing elements, so it keeps the word the query wrote and it is a kind of its own. It is not EXPR_LIST, because the elements live in the statement arena rather than in the expression one and a kind holding an index that means one of two things is how a wrong arena read gets written. It is not EXPR_SUBQUERY either, for the other reason: a scalar subquery has to give back one row and this one takes however many there are.

Running it means a node that reads a column and hands back one value, which is the same thing the other subquery shapes are waiting on, so the refusal says what it always said and now says it from lowering.

Added: the regular expression comparisons run on the merge

There are five differential programs that compare firepanda's pattern handling against pandas, Python's re and the RE2 that Arrow is built with, and until now nothing ran them. They were written alongside documents 76 and 79 through 81, every pull request paid to compile them because the build script built everything in one group, and then no workflow invoked one.

That gap has a cost on the record. A change three weeks ago taught the pattern parser to read a superset of Python's grammar without telling the file that decides which engine answers a call, and firepanda spent those weeks sending 3402 patterns of a 30070 pattern corpus to the engine pandas would not have used. Any of the five would have caught it on the commit that introduced it. It was found instead by an unrelated pull request going red.

They run on the merge rather than on pull requests, which is the same gate the microbenchmarks and the other two platforms use. These compare against somebody else's release, so a failure is as likely to be a pandas or RE2 change as a change here, and that is not a thing to hold a merge on. A few minutes after the merge is soon enough to name the commit.

Changed: a filter whose comparison keeps every row hands the chunk straight back

A filter that does its own comparison already knows how many rows it kept by the time it has to choose between writing a selection and copying the survivors. When that number is the whole chunk there is nothing to choose, because both routes end up with the values already in front of them, and the copy route in particular writes every column out again to get them.

The case is more ordinary than it sounds. A conjunction lowers to one filter per conjunct and a conjunct that is true of everything in front of it is a normal thing to write: a date range drawn around a partition that lies inside it, a flag only ever set on rows some other condition has already dropped. Seven ClickBench statements ask for July 2013 over a partition that is all July 2013, so two of their six conditions kept every row, and each of those was rewriting every column the group by above them reads.

Measured as a pair, two drivers built from the same source with nothing else different, run back to back at 1M through the SQL front end on a busy machine, so CPU time per run rather than wall clock, in milliseconds: q42 50.1 against 25.3, q38 147.9 against 77.7, q37 174.8 against 116.8, q40 34.8 against 25.6, q36 289.7 against 214.8, q39 579.5 against 467.5. q41 came out level in that pass and ahead in a separate one, which is what a query of thirty milliseconds looks like at a load average of eighty. Across all 43 statements the total is 5447.3 against 5079.1 and every answer is the same. The five group by queries that looked slower in that pass were remeasured over three more rounds and sit either side of where they started.

The check is on the comparing route rather than on the one that reads a mask column, because the comparison has already counted and a mask has to be walked to be counted. Paying for that walk on every filter to find the ones that kept everything is the wrong trade. See #682 and #521.

Added: WITH ORDINALITY reads and prints

WITH ORDINALITY after a table function asks for an extra column holding the position of each row, and firepanda used to turn the query down at the parse. It reads and prints now, at 23 statements of the corpus, and lowering is where it stops.

It needed no new field. A table function reference already carried one field holding whether it was written LATERAL, and the two words are the same kind of thing, so that field is a pair of bits now and neither word costs anything beyond having been written.

Running it means the table function and the column being built together, since the position is the position the function produced the row at and nothing downstream can work it out afterwards. That is the same shape the other table function gaps are waiting on, so the refusal says what it always said and now says it from lowering.

Changed: the test shards go from ten to eight and the cost table is remeasured

A shard cannot finish before its longest file does, and on a runner tests/test_sql_run.mojo is 596 seconds of the 12086 the suite costs. Ten shards left seven of them finishing in around 310 seconds and then waiting on that one file, so the last two were buying no wall clock at all while still queueing behind every other job whenever several pull requests are open at once. Eight comes out at the same 596 seconds with a fifth fewer machines.

Not six, which the arithmetic also puts at 596. Runners differ from each other by about a third, and eight leaves the busiest unfloored shard at 384 seconds against the 596 second floor where six leaves it at 542, so a slow machine would become the new wall clock.

tools/test_costs.txt is regenerated from a green run and now covers all 162 test files. Two measurements that did not work out are written down beside it: the precompiled package makes the largest file slower rather than faster, 206 seconds against 193, and splitting that file into three costs 201, 175 and 185 against 204 for the whole, because every part still imports firepanda.sql.run.

Added: an E'...' string reads its backslash escapes

A string written with an E in front of it takes backslash escapes the way C does, and firepanda used to turn the whole spelling down rather than decode half of it. It decodes all of it now, at 25 statements of the corpus.

The set is Postgres's, which DuckDB inherits. \b, \f, \n, \r and \t are the five letters that mean a control character, \xHH is one or two hex digits, \NNN is one to three octal digits taken modulo 256, and \uXXXX and \UXXXXXXXX are a code point written out. A backslash in front of anything else is that byte with the backslash dropped, so \\ is one backslash, \' is one quote and \z is the letter. A doubled quote still means one quote, because the prefix adds a second way of writing one rather than taking the first away.

Two of Postgres's escapes are not in DuckDB, and this follows DuckDB rather than the documentation, so E'\v' and E'\a' are the letters. The two code point spellings go the other way: DuckDB 1.5.1 answers a parser error for both of them, which looks like a gap on its side rather than a decision, and no corpus statement holds one either way, so firepanda reads them.

Two results are turned down rather than kept. A NUL byte cannot be held in a text value and bytes that are not UTF-8 are not text at all, and both are a fact about the query rather than a gap in firepanda, so both answer a parser error the way a quote in the wrong place does.

Added: OVERLAY, TRY and UNPACK read as the calls they are

The grammar gives a handful of functions a rule of their own, because SQL spells them with keywords inside the parentheses where the commas would go. COALESCE, NULLIF, SUBSTRING, EXTRACT, TRIM and POSITION already read as the calls they are, and these were the last three, at 93 statements of the corpus between them.

OVERLAY(s PLACING t FROM 2 FOR 3) and overlay(s, t, 2, 3) are the same call said the standard's way and the plain way, exactly as SUBSTRING is, and both become one. It needs no refusal of its own, because DuckDB has no overlay function either and stops on the name rather than on the syntax, which is what firepanda now says about it too.

TRY(x) and UNPACK(x) are one argument each and are written the way a call is written, so they read as calls and stop at lowering. Neither one is a function. TRY answers NULL where the expression would have raised and UNPACK spreads a list across the arguments of the call around it, so both are shapes the plan would have to grow, and the refusal that used to cover all three now covers the two that are really about meaning rather than syntax.

Added: TABLESAMPLE and USING SAMPLE read and print

A sample takes a share of the rows instead of all of them, and DuckDB spells it two ways that mean the same thing. TABLESAMPLE is the spelling the standard uses and USING SAMPLE is the one DuckDB added, and the grammar takes either one in either place. It was the second largest thing firepanda still turned down at the parse, at 132 statements of the corpus, 106 of them written after a table and 26 written on the block.

The grammar hangs a sample off six rules and only three of them are reachable, since two are the Nearest spellings the error recovery uses and one is the table alias form that is refused before a sample can be looked at. Of the three, the one on a SELECT block goes in a clause slot the way every other clause of a block does, and the two on a table reference go in a field those reference kinds were not using, so nothing grew to hold it.

Three things about how a sample was written are kept rather than normalized, because the grammar takes all of them in both places and none of them changes what the sample means. Which keyword introduced it, whether the count carries %, PERCENT, ROWS or nothing at all, and whether the method name stands in front of the parentheses or inside them next to the seed. TABLESAMPLE reservoir(10%) REPEATABLE (377) and USING SAMPLE 10% (reservoir, 377) are the same sample said two ways and each prints back the way it came in.

None of it lowers. Taking a share of the rows is a row source of its own and the plan has no node for one, so the two refusals moved out of the transformer and into lowering and say the sentence they always said, reworded to name that the sample now reads and prints first.

Added: USING KEY on a WITH entry reads and prints

USING KEY is how DuckDB says that a WITH entry keeps one row per key and replaces that row as the query runs, rather than keeping every row it ever produced. It was the largest single thing left that firepanda turned down at the parse, and it is the last of the WITH modifiers to read.

The list it takes is the list a SELECT takes, so an entry may be any expression and may carry an alias, and the corpus writes both. USING KEY (k) is the common shape and USING KEY (k, min(v)) is the next one. All of it now reads into the AST and prints back out, between the column alias list and the AS, which is where the query wrote it. None of it lowers. Keeping one row per key is a node the plan does not have, and the query that asks for it first is the recursive WITH, which firepanda does not run either, so the refusal moved out of the transformer and into lowering and says the sentence it always said, reworded to name what the feature does rather than what it does to a UNION.

A WITH entry has five things to keep and four fields to keep them in, so the key list rides above the materialize tag in the same word. The tag is one of three values and takes two bits. Packing it that way means no statement kind grows a field that every other kind would carry for nothing, and cte_tags, cte_materialize and cte_keys are the three functions that put it together and take it apart.

Worth writing down while it is fresh: DuckDB takes USING KEY on a plain WITH as well as on a recursive one, so the refusal sits on the entry rather than on the RECURSIVE keyword.

Fixed: three Python tests that still expected the exception class the RE2 grammar replaced

Reading RE2's grammar changed what a pattern neither grammar can read comes back as. It used to be a NotImplementedError, on the reasoning that this library had no way to tell a broken pattern from one merely beyond it, and the whole point of the grammar is that it now can, so it is a ValueError and matches what pandas gives for the same pattern. Three tests in the Python suite still named the old class and went red on main rather than on the pull request, because the extension job runs on one platform there and on three on the merge.

The three are (a)\2 and its siblings in test_str_backreference.py, (?L:a) in test_str_scoped_flags.py, and [\z] unflagged in test_str_z_escape.py. Each of them has a paragraph about which error came from where, so each of those paragraphs is rewritten rather than the assertion being quietly swapped.

Changed: a filter reads its mask in blocks whatever the mask looks like

Reading the mask a cache line at a time was switched on only for a mask selective enough to meet blocks that keep nothing, on the reasoning that the block read is waste otherwise. The reasoning was right and the conclusion was wrong. Reading in blocks is faster even on a mask with no empty block in it anywhere, where every block test comes back true and is therefore pure overhead, which means the gain is in the shape of the loop rather than in the skipping.

The loop it replaces has its exit governed by the mask, and that is data the branch predictor cannot learn. The blocked one has an inner loop with a trip count of sixty four. On a sixty four million row int64 column on a 13900K, a mask that keeps every row goes from 21032 to 18924 microseconds, which is ten per cent for a filter that skips nothing at all. A contiguous band of fourteen per cent of the rows goes from 5108 to 4746 microseconds at the front of the column, 5301 to 5018 in the middle and 5425 to 5214 at the back. There is no shape measured where the blocked loop loses, so the threshold is gone rather than moved.

Added: COLUMNS() reads and prints

COLUMNS(...) is how DuckDB writes a set of columns where one expression goes, and it was the biggest single thing the corpus asked for that firepanda turned down at the parse. Four things may stand in the parentheses. A string holding a regular expression, matched against every column name. A list of names. A lambda taking a name and answering whether to keep it. And a star, with the three modifier lists an ordinary star carries. There is a fifth spelling, *COLUMNS(...), which hands the set over as several arguments rather than as one, and the star there stands outside the parentheses and is not the same star as the one that may stand inside them.

All of them now read into the AST and print back out. None of them lowers. Which columns a COLUMNS() stands for is a question about the bindings, and the bindings are not resolved until after lowering has started, so the refusal moved out of the transformer and into lowering and says the sentence it always said. What that buys is the round trip, which is the property the whole SQL front end is tested against: the statements go through parse, print and reparse and come back as the same text, and the shapes inside the parentheses were never what was in the way.

The expression holds what is in the parentheses as one ordinary node and a single bit saying whether the leading star was written. Which of the four forms it is decides nothing here, so nothing here tries to tell them apart, and the list, the lambda and the star each take the path they already took. That is also why the star inside keeps its EXCLUDE, REPLACE and RENAME lists without anything new reading them.

Changed: the pull request pipeline finishes in about six minutes instead of about fifty five

A pull request waited fifty one minutes for the Linux build and test job and sixty one for the macOS one, and everything else in the workflow finished in seconds. Thirty nine of those fifty one minutes were one step, the unit tests, and the rest of the wait was three jobs doing the same work on three platforms and one job measuring a benchmark that nothing compared it against.

The unit tests do not get faster on one machine. Mojo 1.0 has no mojo test, so each of the hundred and sixty one test files is a separate program that compiles the library again, and almost all of its wall clock is the compiler rather than the assertions. There is nothing to cache between files either, and that was measured rather than assumed: pointing a test run at the precompiled build/firepanda.mojoc instead of at the sources moved the largest file from 14.3 seconds to 13.8 and the next one from 21.6 to 21.8, because the generics are instantiated into whichever program uses them. So the only lever is more machines, and tools/run_tests.sh now takes FIREPANDA_TEST_SHARDS and FIREPANDA_TEST_SHARD and the workflow runs ten of them. The split is by measured time, out of a table in tools/test_costs.txt, largest first into whichever shard has least so far. Getting that table right took two tries and both failures are worth writing down. The first balanced on file size, on the reasoning that compile time tracks how much code there is, and the shards came out between 249 and 662 seconds, because a file's cost is how much of the library it imports and instantiates rather than how long it is. The second used real times measured on a sixteen core workstation, on the reasoning that only the ratios matter, and the ratios do not hold: the same assignment that sat within half a per cent there spanned 423 to 622 seconds on a four core runner. The table now comes off a runner, which tools/test_costs_from_ci.sh reads back out of a run's logs for free, because every file's time is printed next to its name.

macOS and arm64 now run on the merge rather than on the pull request, which is the same trade the workflow already made for the Mojo nightly toolchain and is made for the same reason. Three platforms is three times the machines and none of them is three times the information. No macOS or arm64 failure in this repository has ever arrived without the x86 job failing alongside it, and the price was being paid by whoever was waiting to merge. Paying it on the merge costs a revert in the case where they do diverge, and that case has not happened yet.

The differential fuzzers have moved to a new nightly workflow. They were a step in the build job bounded to five minutes a fuzzer, and that bound is the whole problem: five minutes is not where a fuzzer finds anything, it is where it has finished warming up, so the number was not a fuzzing budget but a decision about how long to make somebody wait. Every seed a fuzzer finds becomes a checked in test that runs on every pull request afterwards, so the fuzzing itself belongs where nothing is waiting for it. It now gets half an hour a fuzzer every night, which is six times the coverage, and opens a tracking issue when it finds something.

The microbenchmark job no longer runs on pull requests. Its baseline step already stood itself down there, and the comparison step is conditional on the baseline step, so what was left was nine and a half minutes of measuring followed by an artifact upload that nobody opens. The performance gate that means something on a pull request is the reference machine run recorded in it by hand, and that has not changed.

Two smaller things in the same pass. The differential job was compiling eleven programs to run six of them, because the five regular expression comparisons are run by hand out of the documents that introduce them and no workflow has ever run one; they are a separate build group now and the six that CI runs are about half the compile. And pixi run build has gained inputs and outputs, so pixi run test, which depends on it, no longer precompiles the package a second time straight after the workflow's own build step has just done it.

Added: a selective filter skips a block of rows it is going to drop whole

The compaction loop read one mask byte a row and wrote the row speculatively, advancing the output cursor by the mask bit rather than branching on it. That is the right shape for a mask that keeps most of its rows and the wrong one for a mask that keeps almost none, and one byte at a time is not enough to tell the two apart.

A selective mask is now read sixty four bytes at a time, which is the cache line the loop was going to touch anyway, and a block that keeps nothing becomes a cursor advance with no work at all. A block with anything in it falls back to the row at a time loop over those sixty four rows, so the fallback costs the block test and nothing more.

What this saves is the read of the source column rather than the write, because a speculative write has to load the row before it can decide not to keep it, and on a wide column the source is eight times the mask. That also says which masks benefit. A scattered mask keeps something in almost every block and gets nothing. A mask that keeps one contiguous range was already fast, because the loop stops once it has written the number of rows it counted and so never reads past the end of its own range. The win is on the shape in between, a mask with many separated clumps, which is what a range predicate looks like over a column that is clustered rather than sorted.

Measured on a 13900K, eight million rows, int64, ABBA with two readings a side. A mask that keeps a hundred and three rows and then drops five thousand goes from between 1183 and 1664 microseconds to between 332 and 341, so somewhere between three and a half and five times, with the spread all on the old side. An all true mask, an alternating mask and a scattered mask are all above the threshold, take the old path, and measure identically to it.

Whether to read the mask in blocks is decided from the kept row count that the offsets pass has already produced, so the decision is free. The threshold was swept at an eighth, a quarter, a half and always on. Always on was not worse anywhere and was faster on an all true mask by about ten percent, which is unexplained, so the constant sits at the conservative value where a mask above it runs the instructions it ran before.

Added: pixi run test-fast, which is the suite without the third of it that is three quarters of the time

The pipeline entry above is about what a pull request waits for. This one is about the loop somebody is in while they are writing the code, which is a different problem with a different answer. Waiting six minutes to find out whether a change to one kernel compiles is not a pipeline cost, it is the reason people stop running the tests.

The cost table that the ten shards are balanced on turns out to answer this too, and the distribution in it is lopsided enough that one number cuts the suite cleanly. Fifty one of the hundred and sixty two files cost a minute or more, the median is thirty six seconds and the largest is five hundred and twenty two, and those fifty one are seventy five per cent of the whole run. So --fast leaves them out and keeps two files in three for a quarter of the time. On an idle eight core machine that is a hundred and eleven files in about two hundred and sixty seconds of file time, which the lanes finish in well under a minute.

It is not the gate and it says so. The run prints how many files it left out and that this is not the thing that decides whether a branch is good, because the expensive files are the SQL planner, the plan lowering and the pipeline, which is exactly where a mistake in a kernel surfaces. It also refuses to write a cost table, for the same reason a shard refuses to: a table measured from a run that skipped the expensive files lists only the cheap ones and the next sharded run would balance on nothing.

Two smaller things came out of measuring this. An unsharded run went out in alphabetical order, which puts tests/test_sql_run.mojo near the end and leaves one lane running it alone for nine minutes after every other lane has run out of work, so the unsharded case now goes longest first the way the shards already do. And the width is now capped by memory as well as by cores. A mojo run of a test file wants around three quarters of a gigabyte while it is compiling, and eight of those on a machine that has given its memory to something else do not run eight times faster, they page: measured on a laptop with three other projects building on it, free memory at sixty megabytes and swap at seven gigabytes of eight, tests/test_sql_run.mojo took seventy six minutes against the five hundred and twenty two seconds the table has for it. A CI runner has its memory to itself and never reaches this.

One thing that was tried and does not work, recorded so that nobody tries it twice. mojo run -I . reads the library out of the source tree and it is tempting to point it at the build/firepanda.mojoc that pixi run build has already made. It buys nothing. With a warm compilation cache four files across the size range came out within five hundredths of a second either way, which agrees with the cold cache measurement in the pipeline entry above, and the reason is the one given there: the generics are instantiated into whichever program uses them, so there is nothing in the package for a test file to reuse.

Added: a flag group that belongs to a place, which is what RE2 thinks one is

(?i) is two different constructs depending on who reads it. Python says it is about the whole pattern, which is why Python added a rule in 3.11 that it may only be written at the very start, and a(?i)b is a parse failure. RE2 says it is about the place it is written and turns the letter on from there to the end of the enclosing group, so a(?i)b folds the b and leaves the a alone. Python also has no colon-less form that turns a letter off at all, so (?-i) and (?i-s) are parse failures with a different sentence, missing :, because what Python is looking for when it sees the minus is the colon that would make it a group.

Since pandas decides which engine answers by asking whether Python's re compiles the pattern, every one of these goes to Arrow, and until now firepanda held them out with a NotImplementedError. They are 352 of the 1805 corpus patterns that RE2 reads and this library did not, which makes them the largest item on that list that needs no data tables at all.

What the scope covers was measured one row at a time against the RE2 inside pyarrow rather than read off a description of it, because guessing would have got it wrong twice. It runs forward and not backward. It crosses a bar, so x(?i)x|c matches C and c|x(?i)x does not. It stops at the closing bracket of the group it is in, so ((?i))c does not fold the c. And it leaves nothing behind for a repeat: a(?i){2} counts the a, and xa(?i)*b matches xaB and not xAb, so the star repeats an a outside the new scope while the b beside it is inside it. That last one is why the wrap happens when the sequence ends rather than when the group is read.

The parser still reads Python's grammar, because the routing depends on it, so the tree gains a flag rather than the parser gaining a mode. Parsed already carried re2_refuses for Python syntax RE2 has never had; this adds the mirror, python_refuses, together with the exact sentence Python would have given. The compiler refuses such a tree on Python's engine and builds it on RE2's, so a caller who passed a flags argument and therefore landed on Python's engine sees no change whatever. The node that comes out is the OP_SCOPE the scoped form has always produced, so none of the three engines learned anything new; what is new is that a flag group splits the sequence it is in rather than wrapping it, and that _branch distributes the scope over every later alternative, which is the transformation that crosses a bar without changing which alternatives there are.

The letters are not the same set either, and that was measured too. RE2 takes i, m, s and U. Python takes a, i, L, m, s, u and x. They share three. x is the one worth knowing, because it is the one a person assumes is shared: RE2 has no verbose mode, so (?x) is refused by RE2 and read by Python, which is the opposite direction from everything else here, and nothing in this change touches it. U is RE2's own and swaps which quantifiers are greedy; it appears once in thirty thousand corpus patterns and stays refused, because a letter this parser does not know is a pattern held out rather than guessed at.

On the str.contains differential the grammar bucket falls from 1807 patterns to 1472. 321 of the 335 become compared answers and the other fourteen land in a held out bucket that names their actual construct, which is the right direction: a pattern that now parses and is then held out for a named character class has a true reason where it used to have a vague one. str.match and str.fullmatch move the same way, str.count and str.replace go to 27739 compared, and the Python engine differential does not move at all. All five stay at 10000 agreements in ten thousand with zero disagreements.

Document 102 is the write up, and document 101's reader needed no change: it already read all three spellings and refused (?x) and (?-) and a(?i, which was checked rather than assumed.

Added: a name for a set of characters, which RE2 has and Python has not

\p{Greek}, \P{L}, \p{^Nd} and the braceless \pL now read, compile and answer. CPython's re calls every one of them bad escape \p and always has, so pandas sends them all to Arrow, and the RE2 inside Arrow is the only engine that will ever run one. They were 392 of the corpus patterns RE2 reads and this library did not, which is the largest item left on that list after the flag groups.

The table under them was measured against that same RE2 rather than copied from CPython, by tools/gen_regexunicode.py, which asks pyarrow.compute.match_substring_regex one name at a time over a column holding all 1112064 code points that can appear in a row. That is not fussiness. RE2 and CPython are not built against the same Unicode release, and \p{L} and CPython's unicodedata disagree about 4302 code points, so a table copied from the wrong library would have been wrong about four thousand characters on the day it was written. What comes out is firepanda/kernel/regex/unicodedata.mojo, 196 names in 5803 ranges: 26 two letter general categories, the 6 one letter ones built as unions of their parts and then checked, 163 scripts, and Any.

Two of the measured rules would have been got wrong by anybody writing them from what Unicode says. \p{C} does not hold the unassigned code points; it is exactly Cc and Cf and Co and Cs, and \p{Cn} is not a name RE2 has at all, which is a gap 819533 characters wide. And \p{Cs} is a real name with nothing in it, because a surrogate has no UTF-8 encoding and every row reaching the engine arrived as UTF-8. Three smaller rules are asserted in the generator so a later release cannot move them quietly: a name is case sensitive, the Is prefix other engines take is not one RE2 takes, and Cn stays refused.

The tree gains one op, OP_UNICODE, carrying the name's index and whether it was negated, and records python_refuses with Python's own sentence, which is the field added one slice ago for exactly this shape. The compiler folds a name before negating it, the order [\d\D] and [^\W] already depend on, so (?i)\p{Lu} widens from 1831 code points to 3212 the way RE2 widens it. Inside a class a name is an item like any other.

The reader from document 101 stops declining. It had one construct it would not judge, which was this one, and the table settles it, so \p{Cn} and \p{Foo} and \p{latin} are refused with the sentence RE2 answers them with rather than held out. That means the unsure field, the branch that read a pattern because of it and the differential bucket that counted it are all gone: a field nothing can set is worse than nothing, because a reader believes it. regex_re2 now agrees with RE2 about every one of the 30052 corpus patterns, on whether it reads and on why not, at 8373 read and 21679 refused with matching reasons and nothing set aside, where it was 8339 and 21547 with 166 declined.

The corpus generated two \p patterns and now generates twenty, so this slice was measured twice. Against the old corpus the str.contains grammar bucket falls from 1472 patterns to 1032, 430 of the 440 becoming compared answers and the other ten landing in a bucket that names their actual construct. Against the widened corpus str.contains is 28410 compared and 1642 held out, str.match 28489 and 1563, str.fullmatch 28490 and 1562, str.count and str.replace 28396 and 1656, and the Python engine differential 29578 and 474. All five stay at 10000 agreements in ten thousand with zero disagreements.

Document 103 is the write up, and document 101 is amended where it recorded a bucket that no longer exists.

Added: a repeat on a position, which RE2 allows and Python does not

^*, $+, \b{0}, \A? and every other quantifier written directly against a position test now read, compile and answer. Python has a rule that a quantifier must follow something a quantifier can be applied to and a position test is not one of those things, so re.compile("^*") is nothing to repeat and pandas routes the pattern to Arrow. RE2 has no such rule.

What RE2 thinks it means was measured against the RE2 inside pyarrow over seven texts rather than reasoned about, because the last two slices both turned up a rule that reasoning got wrong. A position test asks a question about where the engine is standing, and asking it twice cannot fail if asking it once succeeded, so the whole quantifier collapses to one of two things: the test itself when the lower bound is at least one, and nothing at all when it is zero. ^+ is ^, \b{1000} is \b, and ^* and ^? and ^{0} and \b* are all empty. \b* and \b+ differing is the pair that shows the lower bound decides it rather than which quantifier was written.

Three neighbouring constructs are unchanged. (^)* and (?:^)* are patterns Python already reads, so the check added to the compiler asks the narrow question of whether the body is exactly one position test rather than the wide one of whether the body reads nothing, and both keep the path they had. (?=a)* is a quantifier on a lookahead and belongs to neither side of this. And ^** and ^*+ and ^*{2} stay refused, because RE2 refuses them for the second quantifier rather than the first: it reads ^* and puts a repeat on its stack, and a repeat on a repeat is bad repetition operator. That is why the parser builds the repeat node and the compiler is what collapses it, since collapsing at parse time would leave the next quantifier looking at the wrong thing.

Two edits and no new op. In _seq the branch that gave up now records python_refuses with Python's own sentence and falls through, which is the third use of that field after the flag group and the Unicode name. In _emit_repeat a repeat whose body is one OP_AT writes the body once or not at all instead of copying it a count of times, which is the one shape a program built by copying cannot express.

The reader from document 101 needed no change, having been written from RE2's grammar rather than Python's, and its differential does not move: 8389 read and 21681 refused with matching reasons over 30070 patterns, with nothing wrong and nothing set aside.

The corpus went from two patterns in this family to twenty four, so the slice was measured twice. On the same corpus as the last one the str.contains grammar bucket falls from 916 patterns to 841, of which 51 become compared answers and 24 move to a bucket that names their actual construct, 16 of those being the \B family landing in the byte boundary bucket where they belong. That leaves 75 as the honest count of this construct against the 202 the list started with, the other 127 being patterns with genuinely nothing in front of the quantifier, which RE2 refuses too and which were already right. On the widened corpus str.contains is 28479 compared and 1591 held out, str.match and str.fullmatch are both 28529 and 1541, str.count and str.replace are both 28465 and 1605, and the Python engine differential is 29596 and 474. All five stay at 10000 agreements in ten thousand with zero disagreements.

Document 104 is the write up.

Fixed: a pattern Python's grammar refuses now routes to Arrow again

Documents 102, 103 and 104 each taught this library to read a construct Python's re has never read, and each one kept Python's own refusal on the tree rather than throwing it. The router was not told. It asked whether the parse succeeded, which used to mean the same thing and stopped meaning it three slices ago, so (?P<n>\p{Lu})(?P=n) and a(?i)(?=b) and ^*(?=x) were sent to this library's copy of Python's engine while pandas hands every one of them to Arrow.

A pattern only misrouted when it held both a construct Python refuses and a lookaround or a backreference the walk can see, which is why nothing smaller caught it: there is no such pattern in any test file and the two routing checks that would have found it are not run by any workflow. The generated corpus has 454 of them.

holds_unsupported and reads_as_python now both ask whether Python's grammar read the pattern rather than whether this one did. The routing differential goes from 3402 disagreements to zero over 30070 patterns, and two tests that had been asserting the right thing against the wrong implementation pass again.

Added: a run of characters taken literally

\Q opens a run of characters RE2 takes literally and \E closes it, so \Qa+b\E matches three characters rather than a repeated a and a b. Python's re has neither escape and calls both bad escape, which is what sends every pattern holding one to Arrow, and they were the largest family of patterns RE2 reads and this library did not, at 181 of the generated corpus.

Almost nothing a reader would assume about the run is true, and every rule was measured against the RE2 inside pyarrow before it was written down. An unterminated run is not a refusal and reaches the end of the pattern. An \E with no run open is a refusal, so the close is not a harmless no op. Nothing inside a run is an escape, so \Qa\\E is the letter a and one backslash, because the close is found at the second backslash rather than at the first, and there is no way to write a literal \E inside a run. The run is not a group either: a quantifier after it takes the last character, so \Qab\E* is ab*, and an empty run leaves nothing behind, so \Q\E* is refused and x\Q\E* is x*. And the run crosses syntax it would otherwise stop at, so (\Qa)b\E) is (a\)b).

The implementation is one block in the sequence reader that appends one literal node per character. Every rule above falls out of that rather than being a line anywhere, which is why the block is short. The compiler, the engines and the router were not touched. The RE2 reader needed one field, because an empty run is the second construct in it that leaves nothing on RE2's stack and it had been reading \Q\E* as a pattern where RE2 refuses one.

One thing turned up that is not about \Q at all. Arrow's replace_substring_regex compiles the pattern twice, once with a capturing bracket round the whole of it to find the extent of a match and once as written so the group numbers in the replacement stay the caller's, and it validates the bracketed copy first. A pattern that ends inside a run swallows that bracket, so str.replace("\Qa", "-") raises missing ): (\Qa), quoting a pattern nobody wrote, while str.count("\Qa") and str.contains("\Qa") both answer. A replace is now asked about the bracketed text before anything else, which reproduces both the refusal and its wording.

The str.contains grammar bucket falls from 841 patterns to 703 on the corpus the last slice measured, 136 of the 138 becoming compared answers. The measured list in the corpus goes from 73 patterns to 114, and on the widened corpus of 30111 all six comparisons stay at 10000 agreements in ten thousand with zero disagreements.

Document 105 is the write up.

Added: a backslash and a digit

Python has a backreference and RE2 does not, so \12 is a reference to group twelve in one grammar and the character with code ten in the other, and which of the two a caller gets depends on how many groups they happened to write earlier in the pattern. That difference, together with the ceiling on an octal escape, was the largest family of patterns RE2 reads and this library did not, at 282 of the generated corpus counted across the two sentences Python refuses them with.

Python reads one digit, reads a second if it is a digit, and reads a third only when all three are octal, in which case the three are a character with a ceiling of 0o377. Anything else is a decimal group reference of the one or two digits, checked where it is written. That is why \777 is an octal escape and out of range while \778 is a reference to group 77. RE2 takes up to three octal digits with a ceiling of 0o777 and reads them as a character rather than a byte, so \400 is U+0100, and it tells an escape from the reference it will not read by looking at the character after the first digit, so \12 is a character and \1 and \18 are both invalid escape sequence: \1, quoting one character of the two that were written.

Every line of that was measured against the RE2 inside pyarrow 24.0.0 and against CPython's re, one pattern at a time, before it was written down. The parser now records Python's sentence beside the literal RE2 reads instead of giving up, which is the fifth use of the field document 102 added, and the same change covers the ceiling inside a character class. The engines were not touched and neither was the router.

The slice also turned up a case that is not about the grammar. The walk pandas does to pick an engine never enters a repeat, so a backreference hidden inside one is handed to Arrow, and with twelve groups in front of it the reference is \12, which RE2 reads as a newline. So ((twelve groups)\12)+ comes back with a column of booleans that has nothing to do with what the caller wrote, while the same pattern without the repeat is answered by Python and answers correctly. Both were measured against pandas, and the one upstream answers is now answered rather than refused.

The str.contains grammar bucket falls from 703 patterns to 458 on the corpus the last slice measured, 234 of the 245 becoming compared answers and 11 moving to buckets that name their actual construct. The measured list in the corpus goes from 114 patterns to 159, and on the widened corpus of 30156 all six comparisons stay at 10000 agreements in ten thousand with zero disagreements.

Document 106 is the write up.

Added: a name for a group

Python asks whether a group name is an identifier and RE2 asks a wider question, and Python will not let two groups share a name while RE2 does not care. Those two together were the largest family of patterns RE2 reads and this library did not, at 189 of the generated corpus.

Python reads every character up to the terminator and then asks name.isidentifier() of what it read, which is why (?P<n\>>a) is a complaint about the name rather than about the backslash, and why the answer for a name written outside ASCII is whatever the Unicode identifier tables say rather than anything about regular expressions. The test has a front and a back, so (?P<n1>a) is a name Python takes and (?P<1n>a) is not. RE2 asks one question of each character and nothing about the name as a whole: the general category has to be a letter, a non spacing or a spacing mark, a decimal or a letter number, or connector punctuation, which is the category the underscore is in. A digit may lead, and a name may be used twice.

That rule was measured over every code point below U+11000 rather than read off a description, against the RE2 inside pyarrow 24.0.0 and against CPython. One direction runs the other way and is now recorded rather than answered: a middle dot is a character Unicode allows in an identifier and puts in a category RE2 does not look at, so (?P<a·>x) reads here and is marked as a pattern RE2 will not take.

The RE2 reader needed a fix for the first time in three slices. It said a name character was an ASCII word character or anything at all outside ASCII, which was written down from the single observation that (?P<é>a) is a name RE2 takes, and it was too wide by every punctuation mark, symbol and space outside ASCII. The corpus had never written one, so the reader's own differential had never moved.

The category table this reads is older than the one CPython carries, which is the right table for RE2's half and was confirmed to be over every code point below U+11000. On Python's half it costs 119 code points, which are characters CPython allows in an identifier and this refuses. A pattern that is only a named group is unaffected, because pandas sends it to RE2 and RE2 refuses the name for the same reason, so the cost only shows when the pattern also holds a lookaround or a backreference.

The str.contains grammar bucket falls from 458 patterns to 279 on the corpus the last slice measured, 179 of the 189 leaving it and 10 holding a second construct the same bucket still names. The measured list in the corpus goes from 159 to 199, and on the widened corpus of 30196 all seven comparisons stay at 10000 agreements in ten thousand with zero disagreements.

Document 107 is the write up.

Added: a dash after a set

Series.str.contains(r"[\d-a]") and the same class through count, replace, match, fullmatch and extract used to raise and now answer, which is the seventh slice of the patterns RE2 reads and this library did not and the largest family left after document 107.

A range needs one character on the left of it, and the two grammars find that out at different times. Python commits to a range as soon as it sees a dash that is not the last thing in the class, and only afterwards looks at what the two ends are, so one branch refuses [\d-a], [a-\d], [\d-\w] and [b-a] with bad character range. RE2 asks first: a dash opens a range only when what was last read inside the class was one character, and otherwise the dash is a member. So [\d-a] is the digits, a dash and a letter.

The item after the dash is read as a fresh item rather than as the top of a range, which is visible: [\d-a-z] is the digits, a dash and the range a to z, not the digits and a dash and two letters. On the other side RE2 is stricter rather than looser, and [a-\d] is invalid escape sequence: \d rather than a complaint about the range, so that shape stays refused by both.

This is one direction and only one. No class in the family is one Python reads and RE2 does not, and a dash at the end of a class or after a completed range is a member to both grammars and always was.

The RE2 grammar reader needed nothing, for the fourth time in seven slices. It has carried the rule since it was written and the corpus had been confirming it on every run, which is the opposite of the last slice and the argument for keeping the two statements of the grammar separate.

The str.contains grammar bucket falls from 282 patterns to 139 on the corpus the last slice measured, with the bad character range line going to nothing rather than to a smaller number. The measured list in the corpus goes from 199 to 239, and on the widened corpus of 30236 all seven comparisons stay at zero disagreements.

Document 108 is the write up.

Added: a count with no lower bound

Series.str.contains("{,3}") and the same pattern through count, replace, match, fullmatch and extract used to raise and now answer, which is the eighth slice of the patterns RE2 reads and this library did not and the last one of any size.

Python has read {,n} as {0,n} since 3.11 and RE2 has never read it as a count at all, because a count to RE2 opens with a digit and a brace that is not followed by one is an ordinary character. So a{,3} is a repeat to one grammar and the text a{,3} to the other, and {,3} with nothing in front of it is a count with nothing to repeat to Python and four characters to RE2.

Those are two different questions. a{,3} has two readings and no refusal in it, and this library holds one tree rather than one per engine, so that pattern stays marked as one the two grammars do not agree about and stays held out. Everywhere else they part there is a refusal on Python's side, which leaves no Python column to lose, so the tree takes RE2's reading and records Python's own sentence beside it.

The sentence depends on what is in front of the brace rather than on the brace. With nothing there at all, or with a position test such as ^ or \b, Python says there is nothing to repeat, and with a repeat there, as in 9*{,3}, Python says it is a multiple repeat.

The characters go in one node each, which is the shape the quoted run already built, and that buys the stack rule for free: {,3}{2,} is three characters and a repeat of the fourth, because what the second count finds in front of it is the closing brace.

The RE2 grammar reader needed nothing, for the fifth time in eight slices.

The str.contains grammar bucket falls from 139 patterns to 6 on the corpus the last slice measured, which leaves 3 patterns about the other spelling of a named group and 3 about a braceless count written after one that had already been read as Python's. The measured list in the corpus goes from 239 to 279, and on the widened corpus of 30276 all seven comparisons stay at zero disagreements.

Document 109 is the write up.

Added: the other spelling of a named group

Series.str.contains("(?<n>a)") and the same pattern through count, replace, match, fullmatch and extract used to raise and now answer, which is the ninth and last slice of the patterns RE2 reads and this library did not.

Python reads (?< and then looks at one character, and it knows only = and ! there, so anything else is unknown extension ?< and that character. RE2 reads the same three characters and refuses = and !, because it has no lookbehind, and reads anything else as the name of a capturing group. The two grammars use the same opening for two constructs that have nothing in common and each one refuses the other's.

Python has no reading here to lose, which is why the slice can be taken. It does not get as far as the name, so there is no Python tree for the pattern, RE2's reading is the only reading, and the tree holds it with Python's own sentence recorded beside it for a caller who reached Python's engine by passing a flags argument.

Worth saying because it was measured rather than assumed: pandas hands these patterns to Arrow, so Series.str.contains("(?<n>a)") answers a column today rather than raising, and a library that refused it would be answering a question pandas does not ask.

The name rule is document 107's category test, asked without Python's identifier test beside it, so (?<1n>a) reads and (?<n.m>a) is a pattern nobody reads. A named group is a numbered group once it is read, so (?<n>a) compiles to exactly the program (?P<n>a) compiles to and the compiler, the engines and the router were not touched.

The RE2 grammar reader needed one line, which is the first thing any of the nine slices has asked of it that it did not already have. (?< with nothing after it was the one pattern in thirty thousand the reader and RE2 refused for different reasons, RE2 complaining about the bracket and the reader about a name it had gone looking for.

The count of patterns RE2 reads and this library does not falls from 7 to 4 on a corpus widened to 30316, and the four that are left are one shape that is held out for another reason anyway. On this corpus every pattern RE2 reads is now a pattern this library reads. All seven comparisons stay at zero disagreements.

Document 110 is the write up.

Added: a pattern read with whichever grammar is going to run it

[[:alpha:]] is a set of letters to RE2 and a set of brackets and colons and letters to Python, and a{,2} is the five characters a{,2} to RE2 and a count to Python. Both are patterns pandas hands straight to Arrow, so the answer a caller gets is RE2's, and both were held out here because the tree this library built was Python's.

They are the only two constructs of their kind. Every slice from document 102 to document 110 rested on Python having no reading at all, so RE2's was the only one and a tree holding it threw nothing away. These two have a whole reading on each side, one tree cannot hold both, and the way out is a second parse rather than a cleverer tree.

parse_pattern takes a for_re2 argument now, defaulting to Python's grammar so that no existing caller changes. The router is one of those callers and stays where it was, because the tree pandas had is Python's and a lookaround written beside a POSIX class still sends the pattern to Python, where Python's reading of the class is the right one. The second reading is asked for by the one caller that has already picked an engine, which is the RE2 compile in method.mojo.

RE2's rule for finding a POSIX class is a scan forward for the next :] anywhere, without stopping at the ] that would close the set, so [[:a]b:]] is a class named a]b and a refusal while [x[:y]] and [[:]] are ordinary members. The fourteen names went in as the ranges they stand for, measured against pyarrow, and a name opening with ^ is complemented over every code point rather than over ASCII.

There is a refusal shape here that could not happen before, which is a pattern Python reads and RE2 does not. [[:bogus:]] is one, and it comes back as the ValueError pandas raises with the sentence the grammar reader gives rather than as a gap this library does not have.

The reader needed the same correction the parser needed, which is the second time in eleven slices the corpus has caught it out. It read a POSIX name as letters and wanted the :] straight after them, which is right for all fourteen names and wrong for a name with a bracket in it.

RE2 reads this syntax differently was a held out reason on six of the seven comparisons and is on none of them now. The corpus is widened to 30397 and all seven stay at zero disagreements.

Document 111 is the write up.

Added: a corpus of texts for the regular expression differentials

The six regular expression differentials compared thirty thousand patterns against sixteen pieces of text, and the sixteen were handpicked one at a time against differences somebody already knew about. That made them a regression test rather than a search, and it is why every disagreement the differentials have ever reported was found on the pattern side.

There are ninety six texts now. The original sixteen are kept verbatim and kept first, so that every answer agreed on so far is still being asked in the same place, and eighty more are generated from forty five fragments. Nineteen of the fragments are outside ASCII, covering nine scripts, the byte lengths two, three and four, a combining mark and five folding corners, which is what a differential between an engine that runs on bytes and an engine that runs on code points needs and did not have.

The list lives in tools/regex_texts.py rather than inside one of the four oracles, and the four Mojo differentials read that module directly rather than each reaching the list by its own path through a different oracle.

Widening the texts six fold found zero disagreements on all six comparisons. That is the measurement rather than a disappointment: the agreement the differentials report can now be said to be agreement about the engines rather than agreement about sixteen strings. The held out tallies are unchanged, since a pattern is held out for what the pattern is.

Document 112.

Fixed: a byte non boundary held out further than it had to be

RE2 runs on bytes and asks the word boundary question between them, so \B matches at every byte position inside a character written with more than one byte and this engine, which walks characters, has no such position to stand on. Every pattern holding \B was refused for the RE2 engine on that basis, which was 124 corpus patterns held out of six comparisons.

Every one of those extra positions is strictly inside a character, so nothing whole can be read there, no position test but the non boundary can hold there, and the only match RE2 can have there is the empty string. They are also all above zero, which means a program that can only match at position zero never attempts one. That is the whole of str.match and str.fullmatch, and both are now answered rather than held out.

For the other four, a program with no path to a match through non boundaries alone has nothing to find at an interior position either, so \Ba is answered while \B on its own is not.

The refusal moved from while the program is being emitted to after it is finished, which is where the compiler already works out whether a program is anchored.

296 held out patterns are answered, with all seven comparisons still at zero disagreements. 106 remain and they want the engine walked over bytes, which is the change the original comment named and is not this one.

Document 113.

Fixed: a named character held out on an engine that has never had one

\N{NAME} was the largest regular expression gap left, at 374 patterns held out of str.contains and much the same of three others, because resolving a name means carrying the Unicode name table.

RE2 has no \N at all. It stops at the N, never looks at the brace, and says invalid escape sequence to every spelling of one, whether the name exists or not and whether or not it is inside a character class. So a caller who writes one and passes no flag argument has written a pattern pandas hands to Arrow and Arrow rejects, and no table would change that.

The parser now records it on the re2_refuses flag it already carried for the comment group, the \u escape, the non trailing \Z and three others, which the compiler reads on the RE2 side only and answers as a refusal rather than a gap.

The table is narrowed rather than cancelled. 54 patterns stay held out of str.contains and 319 out of str.findall, and those are the ones that route to Python's engine, where a named character is real syntax and a placeholder would be a column of wrong answers.

1276 held out patterns are answered across four comparisons, with all seven still at zero disagreements.

Document 114.

Fixed: four names RE2 has that the table had never heard of

Series.str.contains(r"\p{Zs}") raised RE2 has no such character class where pandas answers a column. Zs and Zl and Zp and the Z they make up are names RE2 takes, and the generated table that \p{...} is read against had none of them, because the generator asks RE2 about each name on a list and the list of general categories was written out by hand with the six anybody thinks of on it. The grammar reader that answers whether RE2 would read a pattern consults the same table and was wrong in the same way, and both were fixed by the four rows.

The list is gone. The generator now asks RE2 about every name of one letter and every name of a letter followed by a lower case letter, and refuses to write anything if RE2 takes a name it was never told about. Thirty eight patterns naming a category or a script are added to the measured corpus, which is the pattern side of the script question document 103 asked for, and every differential tally is unchanged with all seven still at zero disagreements. Document 115.

Added: a differential that checks the Unicode name table against RE2

The table \p{...} is read against was measured once, against one release of pyarrow, and carried in the repository ever since with nothing checking it. Regenerating it is the honest check and it is minutes of compute, and the corpus cannot do the job, since a pattern that names a category proves the name is found and says nothing about whether its ranges are RE2's.

pixi run differential-regex-property asks RE2 about the four code points around each of the 5820 ranges, which is where a range table is wrong if it is wrong at all. 200 names, 18834 code points, half a second against a warm build, and zero disagreements against pyarrow 24.0.0. It was checked by moving one high by one and watching it name the character and exit. Document 116.