v0.8.33

Built against Mojo 1.0.0 (ed45d567).

A patch release. On the pandas side, a group by gains head, tail, nth, idxmax and idxmin, agg and aggregate take functions by name and NamedAgg, the DataFrame and Series constructors read records, rows, numpy arrays, index= and columns= as pandas does, and temporal columns hand out and take in moments, spans and dates. A boolean mask now lines up on the row labels and an operator against a moment or a span keeps the column's name. On the engine side, a semi or anti join marks the keys the other side has instead of bucketing its rows, a small text filter or gather writes its column directly, and the regex kernels cut a column finer and walk a greedy repeat in a loop, which takes ClickBench q28 from about 520 ms to about 390 ms on 1M rows.

Added: head, tail, nth, idxmax and idxmin on a group by

A group by now keeps the first or last n rows of every group with head and tail, a negative n dropping that many from the other end, and picks rows by their place in the group with nth[...] or nth(n), one place or a list of them, counting from the end when negative. All three answer rows of the frame in the frame's order under its own labels, and a row whose key is missing is in no group, as in pandas. idxmax and idxmin answer the label of each group's largest or smallest value, the first one on a tie, for one column or for every column that is not a key, in the key order sort asks for, and a group with only missing values raises pandas' error. nth with dropna and idxmax with as_index=False are refused by name. python/tests/test_group_filters.py checks 57 cases against pandas.

Fixed: a boolean series used as a mask lines up on the row labels

df.loc[mask] read a boolean series by position, so a mask whose labels were in another order picked the wrong rows. It is now lined up on the labels the way pandas does, and a mask missing one of the frame's labels raises firepanda.errors.IndexingError, which is new and carries pandas' name and message. df[mask] with a boolean series, list or array, and df[function], now work as they do in pandas, and a list of the wrong length raises pandas' message.

Added: groupby(...).agg and aggregate by name, and NamedAgg

A group by now takes agg in the four shapes pandas reads when every function is a name: one name answers what the method of that name answers, a list of names over one column answers a column a name, a mapping of column to name answers a column a column, and keywords of name=(column, function) or name=NamedAgg(column, function) answer a column a keyword. Each column is that method run on its own, so the numbers and types are the method's, and the keys come back as labels or as columns the way as_index says. firepanda.NamedAgg has pandas 3's fields and printed form, and a pandas one is read too. A mapping handed to one column's agg raises firepanda.errors.SpecificationError, which is new and carries pandas' name. A Python function, and a list over a frame, which pandas answers with two levels of column labels, are refused by name. python/tests/test_group_agg.py checks 47 cases against pandas.

Fixed: an operator against a moment or a span keeps the column's name

s - s.min(), s > datetime(2024, 1, 1) and the other operators with a Timestamp, datetime, Timedelta or timedelta on one side answered a series with no name, because the scalar was lined up as an unnamed column first. It is now lined up under the column's own name, so the answer keeps it the way pandas does and the way an operator against a number already did.

Fixed: a temporal column hands out moments, spans and dates, and takes them in

tolist, iterating a series or an index, items, itertuples, reading one cell with iloc, loc, iat or at, and the reductions that answer a value of the column's own kind now hand out a Timestamp for a moment, with its unit and zone, a Timedelta for a span and a datetime.date for a day, where they used to hand out the stored count. A missing value stays None. The other way round, a list of datetime or Timestamp values given to Series or DataFrame is a column of moments at the finest unit any value is quoted at, with a zone every value shares kept, and a list of timedelta values is a column of spans, which is what pandas infers. A datetime or timedelta on the other side of an operator is lined up as a column, so s - s.min() and s > datetime(2024, 1, 1) answer what pandas answers. This closes #348. python/tests/test_temporal_values.py checks 48 cases against pandas.

Added: DataFrame(...) and Series(...) read every shape of data pandas does

DataFrame now takes a list of records, a list of rows with columns=, a two dimensional numpy array with columns= and another frame, and index= and columns= beside any of them. A single value in a mapping is repeated down the rows, series in a mapping are lined up on the union of their labels as pandas lines them up, and a category or an instant series keeps its type. Series takes index=, one value repeated along it, and a series with an index, which pandas reads as a reindex. A numpy array keeps its own type, int32, uint8 and datetime64[s] included, and numpy scalars in a list are read as the values they hold. A shape that would need a column named 0, a column of objects or a numpy array of spans is refused by name, and the wrong lengths raise pandas' own messages. python/tests/test_construct_shapes.py checks 56 cases against pandas.

Changed: a semi or anti join marks the keys the other side has

A semi, anti or mark join only asks whether the other side has a key, never which of its rows hold it, but the table it probed was the one an inner join uses, built by counting, prefix summing and scattering every row of that side into buckets. It now makes one pass that marks each key the side has, on every core once the side is tall enough, and the streaming join does the same for these three kinds. At SF1 on a laptop, TPC-H q4, which keeps orders that have a late lineitem, goes from about 28 ms to 8.5 ms, and q22's anti join against 1.5 million orders goes from 6.5 ms to 1.3 ms, which takes the query from about 21 ms to 14.5 ms.

Changed: a small filter or gather of text writes its column directly

A filter or gather of a text column below the size where it splits across cores went through StringBuilder, which grows two lists a row and then copies both into the finished column. It now runs the same count and copy the split route does, as one worker on the calling thread. Filtering 42,000 rows of TPC-H customers takes 175 us rather than 485 us for the two byte country code and 715 us rather than 1,180 us for the address, and q22 at SF1 goes from about 26 ms to 24 ms.

Changed: a regex over a column cuts it finer and walks a greedy repeat in a loop

A regex kernel over a column (match, count, extract and replace) used to hand the cores one piece per 128K rows, which is one piece per streaming chunk, so a chunk was walked on one core. It now cuts a column into about 256 pieces of at least 4096 rows each. The backtracker walks a greedy repeat of one character, such as [^/]+ or .*, in a loop that pushes only the arms leaving the repeat, where it used to push and pop the arm going round once per character, and it keeps its stack height in a counter rather than shortening its lists. A capture rewrite of 4000 URLs with ^https?://(?:www\.)?([^/]+)/.*$ goes from about 1000 ns a row to about 700.