v0.8.11

Built against Mojo 1.0.0 (ed45d567).

A patch release with three threads in it: the regular expression engine reading fewer positions per row, more of the SQL surface answered, and one more TPC-H query.

The regular expression machine now knows which characters a match can begin with and steps over a position holding anything else, which is worth about twice the run on an unanchored pattern and is the general form of the anchored rule 0.8.10 shipped. Beside it, scoped flag groups are read on either engine, so (?i:a)bc is answered rather than refused and that is the end of what the flags work on the string accessor was waiting for. And \B on an empty row follows the interpreter the call arrived in, since CPython up to 3.13 fails it there and 3.14 does not, which was a wrong answer under one supported Python and a right one under the next.

TPC-H q20 answers, which takes pixi run tpch to twenty one of twenty two. A join on two key pairs used to be refused by the streaming operator because packing a key tuple into a number needs both frames at once and a streaming join never has them. The tuple is packed into bytes instead, which needs nothing but the row, and the query that wanted it correlates a scalar subquery through a part and a supplier together. The one query left is q21.

Two pieces of SQL that were refused are answered. LIKE ... ESCAPE reads the character a pattern names to make the byte after it literal, and seven more EXTRACT fields are arithmetic over a field that already had a kernel, so a decade is the year over ten and a century is the year one less over a hundred one more.

Fixed: \B on an empty row now follows the interpreter the call arrived in

Series.str.contains("\\B", flags=re.MULTILINE) answered False for a row with nothing in it whatever Python was running. CPython up to 3.13 fails a \B on an empty subject and 3.14 took the case out and made \B the plain negation of \b, which is what RE2 has always had, so the old answer was right under one interpreter and wrong under the next. This library now reads sys.version_info.minor at its own door and compiles for the Python it is beside. Issue #8 M6.

The rule was found by a differential sweep that came back with 106 disagreements at the end of an unrelated slice. Every pattern it named held a \B, none of the new code in that slice ran for any of them, and the two test suites turned out to be asking two different interpreters: the accessor tests run under 3.13 and the sweep runs under the pixi environment, which is 3.14.

The empty row case is now an instruction of its own rather than a third value of the position code. (?a) narrows which characters count as word characters and says nothing at all about a row that holds none of any kind, so AT_NON_BOUNDARY_ASCII is deleted and the narrow reading of \B is RE2's code again. \b has two codes, one per alphabet, and \B now has the same two.

Nothing moved for RE2, which has no version of Python and no \B either, since this library refuses that one on that engine on the separate ground that RE2 asks the word boundary question between bytes rather than between characters.

280 of the 30052 held-out corpus patterns answer differently on 3.13 and on 3.14, and 130 of them are this one. The other 150 are \z, which became a legal escape in the same release and is a slice of its own. Document 90.

Changed: a pattern says which characters a match can begin with, and the scans step over the rest

The anchored shortcut in 0.8.10 knew that every position but zero was hopeless for a pattern opening with ^. This is the general form of it: the compiler walks the program from its first instruction through the splits, the jumps, the saves and the assertions, collects every character a first step could read, and writes that set on the end of the range table the program already carries. A scan with nothing running is about to start a fresh attempt, and an attempt that begins by reading a character no first step accepts dies on its first step, so the scan steps over that position rather than walking the program at it.

It is worth about twice the run on an unanchored pattern. A str.contains scan over two hundred thousand rows of URLs, three passes, one thread, measured as user CPU because the machine it was measured on carries other work: a pattern that matches nothing in any row went from 0.29 s to 0.12 s, and one that matches in most of them from 0.27 s to 0.14 s. The second is the smaller win for the obvious reason, that a row which matches has to be walked from wherever the match begins.

Four kinds of pattern are left without a set. One that can match nothing begins a match at every position, so there is nothing to step over. One opening with (?s). accepts every character. One whose set holds more than three quarters of ASCII is left out on purpose, because the test costs a search per position and only pays where it rejects. And an anchored pattern is left out because 0.8.10's rule already ends its row, which keeps ClickBench q28 the scan it was rather than the scan that asks a question per row and never uses the answer.

The set is what the first step could read and not what a match must begin with, which are different for a pattern whose first step is an assertion. \bfoo is walked through the boundary to the f, so the set is one letter and the assertion still decides. An alternation contributes both arms, a star contributes the letter it repeats and the letter after it, and anything the walk does not recognise ends it with no set at all. Issue #863.

Added: scoped flag groups, which are a letter with a reach

Series.str.contains("(?i:a)bc") used to be refused with a scoped flag group is not carried yet, and with it every pattern writing a flag letter inside a bracket rather than in front of the whole pattern. All of them are answered now, on either engine, and the regex differential compares 434 more corpus patterns than it did, 8205 of 30052. Every one of the seven letters is read in both forms now but for locale, which Python will not take on a pattern made of text in either form. Issue #8 M6.

A scoped group is the first place where a and b in the same pattern can be read under different rules, and a field with one value in it cannot hold that. The letters ride on a node of their own, and the compiler sets the flags, emits the children and puts the flags back, which is what makes (?i:a)b fold the a and leave the b alone. Putting them back rather than clearing them is what makes (?i:(?-i:a)b) fold the b and not the a.

The node is this library's shape rather than CPython's. Python hangs the two flag sets on a subpattern whose group number is None, and a subpattern here carries a capture number that is a payload rather than an option, so borrowing that shape would teach every reader of it that zero might mean two things.

Verbose mode is handled by the parser and not by the compiler, because it decides which characters there are rather than what a character means, and there is nothing left of it by the time a program is built. (?x:a b)c d matches abc d and not abcd, since the space after the bracket is outside the group.

The router was taught the node. pandas walks the parsed pattern looking for a lookaround or a backreference and recurses into a subpattern, and a scoped group is a subpattern upstream, so (?i:(?=a)) goes to Python's engine there and had to go to Python's engine here rather than to Arrow, which has never heard of (?=.

One cell parts company with pandas and it is a CPython defect rather than a difference of opinion. re.search on a pattern opening with a scoped group that widens the alphabet skips positions the pattern matches, because the first character set is computed from the pattern's own flags rather than from the ones in force inside the group, and re.fullmatch on the same pattern and the same row answers correctly. So Series.str.contains("(?u:\\w)", flags=re.ASCII) on an e-acute is True here and False upstream while str.fullmatch is True in both. It is a registered divergence with a test asserting both sides, because reproducing it would mean building an optimisation this engine does not have in order to put a defect in it.

RE2 keeps the three letters it has and keeps refusing the four it never had, so (?x:a b) and (?a:\w) with no flag beside them are an error here exactly as they are upstream. Document 89.

Added: LIKE ... ESCAPE

A pattern can now name a character that makes the byte after it a literal one, so x LIKE 'a!%b' ESCAPE '!' looks for a real percent sign rather than for anything at all. NOT LIKE takes one the same way. The escape is read before the wildcards are, which is what makes ESCAPE '%' legal and leaves a pattern with no wildcards left in it at all.

The pattern reader sends anything that carries an escape to the general search whatever shape it looks like. The four faster readings find their shape by splitting the pattern on %, and they would split on an escaped one too, so they are not asked to. Nothing about a pattern with no escape in it changed.

An ESCAPE that is not one character, or that is an expression rather than something written out, is refused at plan time, and so is a pattern whose last byte is a loose escape with nothing after it to make literal. DuckDB refuses that last one too, but it does it per row while reading, so 'ab' LIKE 'ab!' ESCAPE '!' is false there and only a row that gets far enough into the pattern raises. Refusing the whole query is the same answer for every row.

ILIKE ... ESCAPE stays refused with the message ILIKE already had, since the missing piece is the case fold and not the escape. SIMILAR TO ... ESCAPE stays refused, which is what DuckDB does with it as well.

Added: the EXTRACT fields that are arithmetic over a field firepanda already has

DECADE, CENTURY, MILLENNIUM, YEARWEEK, DAYOFMONTH, WEEKDAY and ERA, with DuckDB's plural and abbreviated spellings of each. None of them needed a kernel. A decade is the year over ten, a century is the year one less over a hundred one more, because 1900 is in the nineteenth century and 1901 is in the twentieth, a year week is the ISO year times a hundred plus the ISO week, a day of the month is the day, a weekday is the day of the week that was already there, and an era is one for every row that holds a date. Writing them as arithmetic over the fields that do have kernels is also what makes it impossible for EXTRACT(DECADE FROM d) to disagree with EXTRACT(YEAR FROM d) on the same row.

An era is written as a conditional rather than as the number one. A row with no date in it has no era, and a literal would have answered for it anyway.

Four fields DuckDB has stay refused, and each now says which piece is missing rather than saying nothing is called that. EPOCH wants a cast from a date or a timestamp to the number it is stored as. MILLISECOND and MICROSECOND are the whole of the seconds and the fraction together in DuckDB, and the fraction here lives under a name that means a different thing in SQL, so there is nothing to add the two from yet. The TIMEZONE three read an offset off a timestamp that has none to read, and answering zero would be answering a question the column was never asked.

Added: a streaming join on more than one key column, and TPC-H q20 runs

A join whose condition is two equalities rather than one was refused by the operator a plan lowers to, and the refusal was about where the numbers come from. Matching rows means comparing key tuples, every table here compares one column, so something has to turn a tuple into one value. align_keys does that by taking each key's range over both frames at once and laying the keys out in positional notation, which is the better answer where it applies and needs both frames to apply at all, because two sides that pack differently have packed nothing. A streaming join has the build side whole and the probe side arriving a chunk at a time, so it never has both.

The tuple is packed into bytes now rather than into a number. Each key writes what it is stored as, a fixed width value as its own bytes and a string as its length in four bytes followed by its bytes, and the row's key is the concatenation. That is injective, which is the only property the join needs: a reader could tell where one key ends and the next begins, so two rows write the same bytes exactly when they hold the same tuple. Nothing in it reads a value, so nothing in it needs a range, a minimum or a second frame. firepanda/join/packed.mojo is the packing and the argument for it.

What happens after that is the text key route the operator already had. Build the table over the right frame's packed column, probe with each chunk's packed column as it arrives, compare the bytes on a hash match. That route exists because a hash is not an exact answer for a string, and a byte string is a string.

An inner join on two keys used to lower to a table built from the first key pair with a comparison and a filter per remaining pair stacked above the join. It answered correctly and it got there by making pairs it then threw away, and it was inner joins only, because a left join has to emit the rows that matched nothing and a filter above the pairing cannot tell those from the rows it is dropping. Both are gone. The join pairs on the whole key.

TPC-H q20 is the query that wanted this. It correlates a scalar subquery on lineitem through l_partkey and l_suppkey together, and decorrelating that gives a left join on two key pairs. q20 answers what DuckDB answers now, and with q11 and q13 closed in the two releases either side of this, pixi run tpch is twenty one of twenty two. The one left is q21, which correlates an EXISTS through an inequality as well as an equality.

The whole frame join_on is untouched and still concatenates both sides for a compound key. The same packing would give it the build on the smaller side and probe the larger shape it has for a single key, and that is written down in docs/specs/planner/07-operator-selection.md rather than done here.