v0.8.10

Built against Mojo 1.0.0 (ed45d567).

A patch release with three threads in it: the end of the flags work on the string accessor, one more TPC-H query, and the regular expression machine doing less work per row.

Verbose mode and the ascii flag are the last two of the seven flag letters, so every letter re names that means anything on a pattern made of text is read now. Verbose mode turned out to be a rule about where whitespace is skipped rather than about which characters are skipped, and the ascii flag needed a third alphabet rather than a narrowing of the two that were there, since Python's ASCII \s holds a vertical tab and RE2's never did.

TPC-H q13 answers, which takes pixi run tpch to twenty of twenty two. An outer join with a condition about its right side used to be refused because a residual above the join would test the padding on a left row that matched nothing. A part of the condition that reads only the right side goes under the right input instead, before the pairing is built, and the left rows are padded or dropped exactly as they were.

ClickBench q28 at 1M is 1.26 s where the 0.8.9 notes had it at 1.95 s, on 3.39 cores where it was on 3.41, machine idle for both and q27 run beside it as the control. DuckDB 1.5.5 answered the same query in 0.154 s in the same minute, so this library is about eight times slower on it rather than about twelve. Almost all of that is the anchored pattern change below; the byte table beside it is about four percent. Issue #830 stays open on the third of its three costs, which is a lazy DFA and is milestone sized.

Added: verbose mode and the ascii flag, which are the last two of the seven letters

Series.str.contains("a # comment\nb", flags=re.VERBOSE) and Series.str.contains(r"\w", flags=re.ASCII) used to be refused, and with them every call on the accessor carrying either letter or a written (?x) or (?a). All of them are answered now, out of Python's engine, which is where upstream sends a call carrying any flag. Six of the seven letters are read and the seventh is one Python will not take on a pattern made of text either. Issue #8 M6.

Verbose mode is a rule about where the skip happens and not about which characters are skipped. The skip is at the top of the item loop and nowhere the item loop calls into, which is why [a b] still matches a space, why a{1, 2} is not a repeat, and why the ? in a * ? is a second quantifier rather than a lazy marker. The whitespace set is the six characters re names and not Unicode's, so a file separator in a pattern still means something. A # runs to the end of the line and \ and \# and a character class put both characters back.

The ascii flag narrows six classes, the ignorecase fold and the word boundary, and it could not reuse RE2's tables to do it. Python's ASCII \s holds a vertical tab and RE2's never did, so there are three alphabets here where there were two. Under the letter the Kelvin sign stops folding onto k and an e-acute stops folding onto its own capital, and (?a)(?u) is refused in Python's own words.

The word boundary moves too and its two halves do not move to the same place. \b takes RE2's reading, since the ASCII word class is one set both engines have. \B takes a reading of its own, because Python fails a \B on an empty row whichever alphabet was asked for and RE2 matches one, so RE2's reading would have brought its answer for the empty row along with its alphabet.

fullmatch and match on Python's engine are answered by writing \A(pattern)\z around the pattern text, and verbose mode is the first flag that reaches into that rewrite. A verbose pattern may end in the middle of a comment, and gluing a bracket onto a # c puts the bracket inside the comment where the grammar never sees it. A newline goes in ahead of the closing bracket under verbose mode and nowhere else, because a newline is the only thing that ends a comment and verbose mode throws one away.

A scoped flag group is still refused. (?x:a b) now reads as a scoped group rather than as an unread letter, so the refusal moved from one sentence to another and the thing missing is a place to hang a flag on a node. Document 88.

Fixed: an outer join takes a condition about its right side, so TPC-H q13 answers

FROM customer LEFT JOIN orders ON c_custkey = o_custkey AND o_comment NOT LIKE '%special%requests%' was refused. It runs now, and TPC-H q13 with it. Issue #816.

The join node carries key pairs rather than a predicate, so a written condition is split on AND and each part is asked which sides it reads. A part that is an equality with one side on the left and the other on the right is a key pair. Anything else is a residual, and a residual over an inner join is a filter above the pairing, which answers the same query because every pairing the join produced is a pairing the condition asked about. Over an outer join it is not the same, since a left row that matched nothing is padded and kept and a filter above the join would then test the padding and throw the row away, so a residual there was refused.

What that missed is that some residuals do not have to go above the join at all. A part that reads the right side and nothing else is saying which right rows are worth matching against, and a left, a semi and an anti join each ask their right side one question, which is whether a row there matches. A right row the condition throws away was never going to be the answer to that question, so throwing it away first is the same answer. The part goes under the right input, before the pairing is built, and the left rows are padded or dropped exactly as they were.

A part that reads the left side, or both sides, is still refused, and with the sentence it came back with before. There is nowhere for those to go: a left row that fails one has to come out padded rather than dropped, and neither above the join nor under one of its inputs is a place that can say so.

pixi run tpch is twenty of twenty two. The two left stop in two different places, q20 on a left join with two key pairs and q21 on a correlated EXISTS that matches through an inequality as well as an equality.

Changed: the table that turns a character position into a byte position is only built when it is needed

A row of one byte characters has the same number for both, so the table is left empty for one and byte_of reads an empty table as saying the two agree. text_extract_regex also builds it after the search rather than before, since a row that matched nothing has no piece to hand back and the table is only ever read to cut one out.

It is worth about four percent and not more. The replacing scan over 200000 URL rows with q28's pattern went from about 221 ms to about 212 ms, because the machine step that reads a character costs much more than the Int that was being written beside it. Issue #830 listed this as one of three costs and this is the measurement that says which of the three it is.

Changed: a pattern anchored at the start is tried once rather than once a character

The search is unanchored, so the machine starts a fresh attempt at every position of every row. For a pattern beginning with ^ outside multiline mode, or with \A, every one of those after the first walks from the first instruction to the anchor, asks whether the position is zero, and stops. The compiler records that shape on the program now and the two scans skip those attempts, so a row that does not match ends at the second position rather than at the last one.

It is worth having because anchored patterns are not a corner. ClickBench q28 reads a column of URLs with one, str.match is str.contains with the pattern wrapped in ^(...) by pandas itself, and match and fullmatch on Python's engine are answered by writing \A(pattern)\z around what the caller wrote, so all three have been paying this. q28 at 1M is 1.26 s where the 0.8.9 notes had it at 1.95 s, on 3.39 cores where it was on 3.41, with the machine idle for both and q27 run beside it as the control. The byte table entry above is in that number as well and is about four percent of it. DuckDB 1.5.5 answered the same query in 0.154 s in the same minute, so this library is about eight times slower on it rather than about twelve. Issue #830.

A pattern that begins with an alternation is left alone even when both of its arms are anchored, since it compiles to a split first and the flag is a promise about the program rather than about one path through it.