v0.8.32

Built against Mojo 1.0.0 (ed45d567).

A patch release. On the pandas side, concat stacks frames and series down the rows and side by side, DataFrame.join and merge join on the row labels, merge(indicator=) says which side each row came from, and DataFrame.assign, attribute access to a column, rank, round, the grouped transforms and grouped diff and transform by name are new. Series built from a mapping now holds its values under its keys. On the engine side, a join probed by many more rows than it builds indexes by value, a hashed join table that most probes miss keeps a bit per key value in front of it, a join in a pipeline hands its probe side on as positions, a sum of an integer column and a constant builds no column, which takes ClickBench q29 from 146 ms to 37 ms, and a streaming group on one integer key writes its map on every core.

Added: DataFrame.assign and a column read as an attribute

df.assign(**columns) adds or replaces columns and hands back a new frame. The keywords are taken in order and each one sees the frame the earlier ones made, so assign(c=lambda f: f.a * 2, d=lambda f: f.c + 1) works as it does in pandas. A value can be a function of the frame, a series lined up on the row labels, a frame of one column, a mapping, a list, a tuple, a range or a numpy array as long as the frame, or one value for every row. A replaced column keeps its place and a new one goes on the end. A wrong length, a frame of several columns and a set raise pandas' errors, and None is refused, because pandas makes it an object column. df.name now reads a column as an attribute when no method has that name, and dir(df) lists the columns that are identifiers.

Added: rank on a series, a frame and a group by

Series.rank, DataFrame.rank and groupby(...).rank() are written, with every method pandas has (average, min, max, first and dense), the three placements for a missing value, both directions and pct. All three are one kernel in the core, a stable sort and a pass over the ties that reads equality off the factorize codes, so numbers, text, flags and times all rank the same way. An ordered category ranks by its place in the categories and an unordered one by its value, and a group by refuses the unordered one, all as pandas does. The answer is float64, as pandas' is, a row whose group key is missing ranks NaN under dropna, and transform("rank") now answers rather than being refused. A rank across a row, axis=1 on a frame, is still refused.

Fixed: Series built from a mapping

Series({"a": 1, "b": 2}) used to read the mapping as a sequence and hold its keys as the values. It now holds the values under the keys as row labels, in the mapping's order, as pandas does, and index= beside a mapping picks those keys out in its own order with a missing value for a key the mapping does not have. name= and dtype= work with it as they do with a list.

Added: round on Series and DataFrame

Series.round(decimals) and DataFrame.round(decimals) round every number the way numpy does, which is to scale by a power of ten, round half to even and scale back, all in the column's own floating point type, so 2.5 rounds to 2.0, 2.675 to two places is 2.67 and a float32 column stays float32. An integer column is unchanged unless decimals is negative, and then it rounds to the nearest ten, hundred and so on and keeps its width. A column of text, bools, categories or times comes back as it was, as in pandas. The frame's method also takes a mapping or a series of places by column name and leaves the columns it does not name alone. Python's own round works on both, numpy's out is accepted empty and refused otherwise, and every mistake raises pandas' class with pandas' message.

Added: grouped diff and transform by name

diff on DataFrameGroupBy and SeriesGroupBy answers each row less the row that many before it in its group, taking whole numbers to float32 for the two narrow signed widths and to float64 otherwise, as pandas does. transform takes the name of a reduction and puts each group's value on every one of its rows, through a new core group_broadcast that reduces the groups once and gathers by ordinal, and it casts the answer back to float32 or to the column's own integer width where pandas does. It also takes the name of a scan. A Python function, the names pandas takes that are not written yet, and a bool column's diff, which pandas answers with an object column, are refused by name, and an unknown name gets pandas' ValueError.

Added: grouped transforms

cumsum, cumprod, cummax, cummin, shift, cumcount and ngroup on DataFrameGroupBy and SeriesGroupBy answer one value a row, in the frame's order and with its labels, as pandas does. The folds run down the rows once with a carry per group and nothing is sorted, shift places each group's rows with one counting pass, and ngroup numbers the groups in key order or in order of appearance. A row whose key is missing belongs to no group and answers missing, so an integer answer with such a row in it is float64. Counting from the end, a fill value, a frequency, a suffix and a list of periods are refused by name.

Added: merge(indicator=)

merge(indicator=) answers pandas' category column that says which side each row came from, left_only, right_only or both, with all three categories whichever occur. True names it _merge and a string names it that, on key columns and on row labels alike, and pandas' ValueError sentences are given for a name that is taken or an indicator that is neither a flag nor a string.

Added: DataFrame.join and merge on the row labels

merge(left_index=True, right_index=True), merge(left_on=..., right_index=True) and the mirror of it now work, and so does DataFrame.join, with or without on. A side keyed on its row labels has them put in a column, and the merge on columns does the rest, so suffixes, key types, a missing key matching a missing key and the outer sort all behave as they already did. With both sides on their labels the answer is labelled by the key, under the right side's label name in a right join and the left side's otherwise. With one side on its labels the answer keeps the other side's row labels, and a row that side does not have loses its label, so an integer label becomes float64 with NaN and the name goes, as in pandas. join takes a named series or a list of one frame, and a cross join, a longer list, and text labels that go missing (pandas' object index) are refused by name.

Added: concat down the rows and side by side

firepanda.concat stacks frames and series the way pandas.concat does, through two new core bindings, DataFrame.stack_rows and DataFrame.stack_columns. Down the rows the columns are the union of the parts' columns in the order they first appear, or the shared ones with join="inner", and a part without a column is missing on its rows. Each column takes the type pandas gives it, which is numpy's rule for mixed numbers, float64 for an integer column with a gap and text for category columns whose categories differ. The row labels come along with their shared name, or are numbered again with ignore_index. Series on series give a series, and a named series next to a frame is a column. With axis=1 the parts are lined up on their row labels, the union of them or with join="inner" the shared ones, and sort sorts them. verify_integrity and the mistakes raise pandas' class and message. keys, levels, names, a dict of parts, and anything pandas answers with an object column are refused by name.

Changed: a join in a pipeline hands its probe side on as positions

The streaming Join node now takes a chunk under a selection and gives one back. It gathers only its key columns, and the probe side of its output is a list of positions into the chunk that arrived, composed with the selection that chunk was under, while the build side is gathered as before. Before, a filtered chunk was flattened on its way in, which copied every column, and then each paired row was copied again on the way out. Now a filter under a join, a join and a filter over it copy each column once, when something finally reads it, and a column nothing reads is never copied. A mark join passes its chunk's columns and selection through untouched. On a filter keeping half of ten million rows, a join against a hundred thousand keys and a sum of two columns, the pipeline goes from 46 to 49 ms down to 42 ms. With no filter and five columns summed, it goes from 62 to 66 ms down to 50 to 54 ms, so the copy that moved to the end is also cheaper than the one it replaced (#79).

Changed: a join probed by many more rows than it builds indexes by value

A join whose build side is small but whose integer keys spread over a wide range used to hash every probe row, because the table indexed by the key value is sized by the build side alone. build_side now also takes the probe height, and a whole frame join accepts a value indexed table up to half that height and at most four million slots. TPC-H q17 is the shape: two hundred parts out of two hundred thousand, probed by six million lines. At SF1, over three interleaved rounds, q17 went from 25.3 to 27.0 ms to 9.9 to 12.2 ms, q8 from 31.3 to 33.6 ms to 16.4 to 18.3 ms, q9 from 93.1 to 102.7 ms to 76.7 to 77.6 ms and q22 from 32.0 to 34.7 ms to 28.3 to 29.1 ms, and the other queries did not move outside their spread. A streaming join does not know its probe height and keeps the old rule.

Changed: a streaming group on one integer key writes its map on the cores

A GROUP BY over one integer key whose values spread wider than the direct table used to insert every chunk into one hash table on one thread, the join's build called a thousand rows at a time. On ClickBench q15, which groups on UserID, that thread was most of the query. The map now uses the parts table the text keys got in 0.8.31: the rows are hashed and bucketed by part on the cores, each part is walked on a core, and the new keys are numbered in row order by a prefix sum, so the ordinals come out as before. An integer key needs no comparison, because its hash is a bijection on its bits, so the walk is only the probe. On a loaded 10 core Mac, a million int64 keys in eight chunks go from 35 ms to 21 ms with 400 thousand groups and from 16 ms to 13 ms with 100 thousand, ten thousand groups is unchanged, and q15's best time over 1M rows goes from about 22 ms to 16 ms. All 43 ClickBench answers are unchanged.

Changed: a sum of an integer column and a constant builds no column

A whole input SUM(x + c), SUM(x - c), SUM(c - x) or SUM(x * c) over an integer column used to build x + c for the chunk and then add it up, which on ClickBench q29 is ninety columns built and read back per chunk, each one a cast of the source to int64 first. reduce_value_any in kernel/fold.mojo runs the operation inside the sum instead: each row is read, widened, given the operation and added, and nothing is written in between. This is not the algebra 0.8.29 had and 0.8.30 took out, and each of the ninety sums still does its own additions over every row. Integer sums wrap and so do not depend on the order of the additions, which is why the answer is the same bit for bit, and why float columns, divisions and the other reductions stay on the old route. On a loaded 10 core Mac, q29 over 1M rows goes from 146 ms at best to 37 ms, and user CPU time drops by three quarters. All 43 ClickBench answers are unchanged.

Changed: a hashed join table keeps a bit per key value when most probes miss

A join build side whose integer keys are too spread out for a table indexed by value now also keeps one bit per value over its range, when the keys fill at most a quarter of that range and the bits are no larger than the hash table. A probe row whose bit is clear is a miss without hashing it into the table, and each morsel stops asking the bits after the first chunk that lets more than half its rows through, so a join where nearly every row hits pays for one chunk of bit tests in sixty four. Ten million probe rows against twenty thousand keys over a million values went from 17.5 to 4.7 ms when the probes miss and stayed at 8.7 to 9.5 ms when they all hit. Ten of the twenty two TPC-H queries keep a sieve at SF1, and over four paired rounds on a loaded laptop q12 ran 1.24 times as fast, q5 and q7 1.22, q8 and q9 1.14 and q4 1.11, while the other four stayed inside the noise band of the queries the change does not touch.