v0.8.9

Built against Mojo 1.0.0 (ed45d567).

A patch release with three threads in it: the last of the flags work on the string accessor, one more TPC-H query, and the wall clock number 0.8.8 said it owed.

str.count, str.replace and str.extract all read a flags argument now. That is the last three of the six pattern methods and the end of what the two case and flags entries in 0.8.7 were waiting for. A count or a replace carrying a flag goes to Python's engine, which counts and replaces by different rules than Arrow does, so str.count("^") on a row of three letters is four and the same call under re.MULTILINE is one, and both numbers are pandas'. A case=False replacement is a compiled pattern and a scan now rather than a byte search, because that is what upstream does with it. And a flag bit re never named was being dropped rather than refused on every one of the six, which lasted two releases because the values anybody would write a test with were the ones it already got right.

TPC-H q11 answers and agrees with DuckDB, which takes pixi run tpch to nineteen of twenty two. A subquery that answers one value was refused in a HAVING and allowed in the WHERE of the same query, and what differed was where the cross join carrying it was put rather than anything about the subquery. It goes above the aggregate when the clause that wrote it reads it from up there.

Every text kernel of the builder shape runs on every core now. The literal text_replace and the folded one beside it followed the regular expression replace from 0.8.8, and text_extract_regex went last with the join run once per capturing group. The number 0.8.8 could not take, because the machine it would have been measured on was carrying a load average above ninety, is in this one: ClickBench q28 at 1M is 1.95 s where it was 5.48 s, on 3.41 cores where it was 1.00, with q27 run alongside as the control and unchanged either way. That leaves this library about twelve times slower than DuckDB on that query rather than about twenty, and issue #830 stays open on the rest of it.

Changed: the replace differential compares the sweeps it used to set aside

tests/differential/regex_replace.mojo held 186 sweeps out of its comparison because pandas answered them out of an engine this library did not have. It has had one since the counting and replacing scans landed, so those sweeps are compared now and the set aside count for them is gone from the report.

An empty pattern is the one shape pandas sends to Python's engine with no flag in sight, because pyarrow used not to terminate on one, and there it runs a different scan and reads its replacement by a different grammar. The harness reads that route the way the pandas layer reads it and compiles the pattern for the engine that answered it. All 186 agree. The run is still 10000 in ten thousand with 0 disagreements over 7771 compared patterns. Document 87 section 9.

Added: str.count and str.replace under a flag, and four more calls that were refused

Series.str.count("^", flags=re.MULTILINE) and Series.str.replace("a*", "#", flags=re.IGNORECASE) used to be refused. Both are answered now, out of Python's engine, which is where upstream sends a call carrying any flag. That finishes the scan the two case and flags entries in 0.8.7 were waiting for. Issue #8 M6.

The loop is the whole of the work. Arrow counts by cutting the row down after every match, stepping one byte past a match of no width, and moving to where a match ended rather than past it. Python does none of those three. So str.count("^") on a row of three letters is four and the same call with re.MULTILINE is one, and str.count("") on a row holding a sharp s is three without a flag and two with. Both numbers are pandas' and both are now this library's.

str.replace(pat, repl, case=False) no longer takes the one to one fold the other four pattern methods take. Upstream turns that argument into re.IGNORECASE and compiles the pattern with it, escaping the pattern first when regex is False rather than taking a different path, so the fold is the engine's for both settings of regex. The four Turkish I code points are where that shows: replace("i", "#", case=False) swaps a dotted capital I and contains("i", case=False) does not find one. A case=False literal replacement is a compiled pattern and a scan now where it used to be a byte search, which is correct and is slower.

Three refusals came off that were never about flags. A replacement holding \g< beside regex=True is answered, because pandas reads that out of re and the template grammar is written now. An empty pattern with a backslash in the replacement is answered, for the same reason one step removed. And a count beside either of those is answered, with n=0 reading as unlimited, which is what re.sub has read it as since long before pandas existed and is the opposite of what the Arrow path reads it as.

A bad replacement template raises InvalidArgumentError, which is a ValueError. Upstream raises re.PatternError, which is not one, and an unknown group name comes back from re as an IndexError. That is a divergence rather than a bug and document 86 says why matching it would be worse.

A count beside a real pattern with no flag and no case still lands on Arrow's bounded loop and is still refused, because that loop replaces nothing after the first match and raises on a pattern of no width.

Added: str.extract under a flag, which is the last of the six

Series.str.extract(r"([a-z])(\d)", flags=re.IGNORECASE) used to be refused. It is answered now, and with it every pattern method on the accessor reads a flags argument. Issue #8 M6.

This one is the odd member of the six and the reason is worth a sentence. Everywhere else a flags argument says two things at once, what the letters mean and that the call has left Arrow, and the second is what costs a word at the door the call crosses by. extract is one of the three names pandas never sends to Arrow at all, so there is no route for the argument to carry and the letters cross on their own beside the pattern. It is also the only pattern method with no case argument, so it is the only one where the two spellings of a fold cannot disagree with each other.

The letters are read before the groups are counted rather than after. A pattern that opens no group is refused with pandas' own sentence and a flag this library cannot carry is refused with its own, and a pattern that is both should be refused over the flag, because upstream compiles before it counts.

The type check on the pattern moved to the entrance. It lived on the path a pattern took to reach the byte search, which was correct while every call took that path, and extract under a flag is a second path. That is the same shape as the check that went wrong one slice ago and it is why this one was moved rather than copied.

re.LOCALE is a ValueError here because Python turns it down on text and pandas hands text to re. Verbose mode and the ascii flag are refused, which is this library's gap rather than anything upstream does, and they are the last two of the seven letters.

Fixed: a flag bit re never named was dropped instead of refused

Series.str.contains("a", flags=1024) answered as though no flag had been passed. Every pattern method had it, because they share the one helper that reads the argument.

The helper builds a mask by oring the seven letters together and then asks whether the caller's value has a bit outside it. The letters are re.RegexFlag members and re.RegexFlag is an IntFlag, whose complement is bounded by the bits the enumeration defines rather than by the integer, so every bit re does not name read as already known. re.DEBUG and the other named ones were refused correctly the whole time, which is why this lasted two releases: the values anybody would write the test with were the values it got right. Document 87 section 6.

Fixed: a subquery that answers one value may be written in a HAVING

SELECT ps_partkey, sum(ps_supplycost * ps_availqty) FROM ... GROUP BY ps_partkey HAVING sum(ps_supplycost * ps_availqty) > (SELECT sum(ps_supplycost * ps_availqty) * 0.0001 FROM ...) was refused, and the same subquery written in the WHERE of the same query was not. TPC-H q11 is that query. Issue #816.

A subquery written as a value is one row holding one value, so it lowers to a cross join onto that row and the expression that held it reads the column back by a generated name. The join went above the FROM in every case, which is under the aggregate, and an aggregate hands up its keys and its folds rather than everything it read, so the column was gone by the time the HAVING above it asked for it. The lowering said so in the refusal and that was the whole of the gap: the join was in the wrong place, not missing.

Where the join goes is now decided by the clause that wrote it. One in a WHERE goes where it went, above the FROM and under the filter that reads it. One in a HAVING, or in the select list of a query that folds, goes above the aggregate instead. Both are built at the same point, before either of those clauses is lowered, because building one is what settles the name they read it back by, and only the node the join is put over differs.

A correlated one above an aggregate is still refused, with a message about the correlation rather than about the aggregate. It lowers to a fold and a left join under the FROM, and what that left join pairs against is the outer rows, which up there have been folded away.

TPC-H q11 answers now and agrees with DuckDB. pixi run tpch is nineteen of twenty two, and the three left are refused in two places.

Changed: the other two replaces run on every core as well

str.replace with regex=False is the default in pandas 3 and is the path most calls to that name take, and it landed on text_replace in pattern.mojo, which ran on one thread. text_replace_folded, which is the same call with case=False on it, ran on one thread beside it. Both of them said so in their own docstrings and both gave the same reason the regular expression kernel used to: the answer is text, how long a row comes out is not known until the search has run over it, and a StringBuilder is one buffer with one cursor that four threads cannot share.

That reason stopped being one when stack_payloads landed in 0.8.8. Both kernels take the route it opened. The views are 16 bytes each and there is one per row, so that buffer is sized before anything starts and each thread writes only its own rows of it, the bytes that do not fit in a view go into a List that belongs to the morsel, and the join lays those end to end afterwards and moves each long view onto where its morsel landed. Nothing about what either kernel answers moves: the searches are the same searches, the three readings of the count are the same three, an empty needle still counts characters where str.count counts bytes, and a row keeps whatever case it was written in everywhere a folded pattern did not reach.

The test is the one the regular expression kernel got, asked twice over one column. The same rows are replaced once short enough to fit a single morsel and once doubled past the row count where every core takes a share, and row i of the tall answer has to be what row i of the short one was, for the exact search and then for the folded one. Three of the sample's rows come out too long to sit inside a view and go into a payload, the rest stay inside one, and two are null. Taking the offset shift out of the join on purpose fails it.

text_extract_regex was the last kernel of this shape still running on one thread when this landed. It wants the same join done once per capturing group, which is what the extract entry in this release does.

Changed: pulling capture groups out of a column runs on every core

text_extract_regex is the fourth and last kernel to take the route stack_payloads opened in 0.8.8, and it is the one that answers more than one column. str.extract gives back a column per capturing group, each as tall as the column it read, so the work is the same work as the three replaces with one join per group at the end of it rather than one. Each group gets a view buffer sized before anything starts, since there is one 16 byte view per row whatever the pattern finds, and a payload per morsel for the groups too long to sit inside a view. The payloads are laid out so a group's morsels sit next to each other, which is the order the join wants to read them in.

The one thing here that the three replaces did not have to do is build a validity bitmap. Everywhere else in that file a row of the answer is missing exactly when the input row was, so the input's bitmap is copied and the work is done. A row of this answer is missing for three reasons: the input was null, the row matched nothing, or the group took no part in the match its row was in, which is what (a)(x)? says about a row holding a. So each group's bitmap starts empty and a thread sets the bit for the rows it owns. Two threads never touch the same byte of one, because a morsel is 131072 rows and that is a whole number of bytes, which is the same argument repair_range has been making about its own writes since it was written.

Nothing about the answer moves. The match is still the leftmost one anywhere in the row, a row with no match is still missing in every column so that the columns of a row agree about whether there was a match at all, and a group that matched the empty string is still empty text rather than missing.

The test is the one the other three got. The same five rows are extracted once inside a single morsel and once tiled past the row count where every core takes a share, and row i of each tall column has to say what row i of the short one said. The tile is chosen so the question is a real one: one row is long enough that both of its groups go into a payload, one sits inside its views, one leaves the second group out, one matches nothing, and one is null. Laying the payloads out per morsel per group and then reading them back in the other order fails it.

There is no ClickBench query on this path, so this one has no number from the suite and is not claimed to have one. What it has is the same shape as its three neighbours, and the reason for doing it is that it was the last kernel in the package whose docstring still explained why it ran on one thread.