v0.8.30

Built against Mojo 1.0.0 (ed45d567).

A patch release. The Parquet read can now keep a string column that repeats as int32 codes into its distinct values, behind Session.run(encode_strings=True), and it codes each group of rows as it arrives, so on TPC-H sf1 lineitem the frame is 813 MB instead of 1150 and the peak of the read is 2.39 GB instead of 2.80. The dtype still says string. The four str_ tests on a series read such a column once per distinct value, and with those in, all 22 TPC-H queries in firepanda-bench give the same answers encoded as flat. The option stays off by default until the kernels outside the bench learn the encoding too. On the pandas side, a series answers value_counts, mode, idxmax, idxmin, argmax, argmin and between, and &, | and ^ work on boolean series and frames. SQL reads ANY and ALL over a written list, JOIN BY (TYPE t), and keeps a FILTER it cannot rewrite instead of refusing it while parsing. The 0.8.29 shortcut that answered a sum of a column and a constant from one sum and one count is taken out again.

Changed: an encoding Parquet read turns each group into codes as it arrives

With encode_strings on, each string column now goes through a RepeatEncoder (in kernel/dictionary.mojo) one group of rows at a time. Before, it was encoded after the whole read had been stacked. The first group decides, on a 64K row sample and then the whole group, and every later group is coded against one growing list of distinct values, so a code handed out early still means the same value at the end. If a column breaks the sixteen rows a value rule part way through, the groups encoded so far are decoded back and the rest of the column is read flat. The answer is the same column either way. On TPC-H sf1 lineitem the peak of the read went from 2.80 GB flat, and 2.58 GB with the old encoding at the end, to 2.39 GB on a six core Linux machine, with the frame at 813 MB against 1150 as before (#979).

Added

• A FILTER on a call it cannot be rewritten into, such as list(i ORDER BY i) FILTER (WHERE i > 1), first(x) FILTER (...) or a macro like my_sum(i) FILTER (...) OVER (), now parses and prints back as it was written instead of being refused by the transformer. The clause stays on the call in the AST, behind the call's own ORDER BY, and the plan refuses it by name there. A filter on an ordinary fold is still a CASE around its argument and runs as before.

• x = ANY ([1, 2]) and x > ALL ([1, 2]), a quantified comparison over a list written out, which DuckDB unnests. ANY becomes one comparison per item joined by OR and ALL joins them with AND, so a null in the list gives the same three-valued answer DuckDB gives, and an empty list answers false for ANY and true for ALL. The form now also parses and prints back when the right side is not written out, such as a list column, and only the plan refuses it there.

• JOIN BY (TYPE t), the spelling DuckDB gives its internal join types, for the six that are ordinary joins: inner, left, right, full (or outer), semi and anti, with an optional _join on the end and in any case. Each one builds and prints as the regular join it names. The mark, single, right_semi and right_anti types are still refused by name, and a name that is no join type at all is a parser error that says so.

Added: a Parquet read can hold the string columns that repeat as codes

Session.run takes encode_strings, and with it on, each string column of the answer whose values repeat sixteen times on average comes back dictionary encoded, with dtype still string. encode_repetitive in kernel/dictionary.mojo makes the call: it hashes the first 64K rows and gives up there if they do not repeat enough, so a comment column costs one hash of a sample, and otherwise hashes the whole column and keeps the distinct values in the order they first appear. A column of only nulls is left flat. On TPC-H sf1 lineitem, l_returnflag, l_linestatus, l_shipinstruct and l_shipmode go from 96, 96, 146 and 96 MB to 24 MB each and l_comment stays flat, which takes the frame from 1150 MB to 813 and the peak of the read from 2.80 GB to 2.58 on a six core Linux machine. The option is off by default until every kernel the TPC-H queries reach can read the encoding (#979).

Added: value_counts and mode on a series

value_counts is a group by on the column and a count, run in the kernel, and then a stable sort on the count, so a tie keeps the order the values first appear in, which is the order pandas' hash table gives. normalize, sort, ascending and dropna all work, the answer is named count or proportion and its index is named for the column. mode answers every value tied for the most rows, sorted, with a missing value last when dropna=False counts it, and keeps the column's type. bins and a category column are refused for now.

Added: idxmax, idxmin, argmax, argmin and between on a series

The four position methods find the extreme with the reduction and the first row equal to it with a filter, so both passes run in the kernel and a tie answers the first row, as numpy's does. idxmax and idxmin answer the label and argmax and argmin the position. A gap under skipna=False, a column of nothing but gaps and an empty column each raise pandas' ValueError in pandas' words, checked in pandas' order. between is two comparisons and an and, the way pandas writes it, with inclusive taking both, neither, left and right, a missing value answering False and the name kept.

Added: &, | and ^ on a boolean series and frame

Series and DataFrame take the three logical operators and their reflected forms, between two boolean operands or a boolean operand and True or False, so a mask built from two comparisons, (s > 1) & (s < 4), now works. Two labelled operands are lined up first and a gap follows pandas' rule, which is not symmetric: a row the right side is missing reads it as False and a row the left side is missing answers False, so a | b and b | a differ there, and a column only one of two frames has is all NaN. A frame against a series lines the series up with the columns. The kernel answers each one with the arithmetic that means the same thing on two bools, *, + and !=. A constant that is not a bool is the TypeError pandas raises, and two integer columns, which pandas answers bitwise, are refused for now.

Removed

• The shortcut from 0.8.29 that answered a whole input SUM(x + c), SUM(x - c), SUM(c - x) or SUM(x * c) from the sum and the count of x. The lowering of an aggregation already says not to do this: ClickBench q29 exists to measure whether an engine fuses an expression into a reduction, and answering it with algebra reports a number the engine did not earn. The reduction builds and sums every column again, one at a time, as it did in 0.8.28.

Changed: the text tests on a series read an encoded column once per distinct value

str_contains, str_contains_in_order, str_starts_with and str_ends_with on a series whose column is dictionary encoded run over the distinct values and spread the answer over the rows by code, so a test on six million rows of seven values is seven tests and a gather. str_slice decodes first, because its answer is text and keeping it encoded would mean deduplicating the cut values again. These are the text calls the TPC-H bench makes on columns the Parquet read can now encode (#979).