v0.8.28

Built against Mojo 1.0.0 (ed45d567).

A patch release. A positional column such as #2 and a subscript or slice of text such as s[2:-2] run in SQL. On the pandas side, a comparison, a string question or arithmetic on a row with a missing value now answers what pandas answers, quantile takes the four other interpolation rules and a list, the series reductions take skipna=False and min_count, a category column reduces the way pandas does, DataFrame.from_arrow is there, and the zoned roundings take ambiguous and nonexistent. median and quantile agree with pandas to the last bit, != against a NaN answers True in the vector loops, and the ClickBench driver builds again after a compiler hang that a zone file read had set off. A column now carries an encoding field, which is groundwork for dictionary coded strings, and a pull request runs only the tests its change can reach.

Added

• A positional column, #n, which DuckDB reads as the nth column of the FROM counting every table left to right. It works in the select list, WHERE, GROUP BY, HAVING and QUALIFY, and a bare ORDER BY #n is the nth output column the way ORDER BY n is. The column keeps its table, so #5 over two tables that both have a b is the right b, and the output column is named after it, as in DuckDB. #0, a signed or fractional number and one past 32 bits are the syntax errors DuckDB gives, and one past the last column is DuckDB's out of range error. It is still refused in an ON clause, over a USING or NATURAL join, where DuckDB counts both copies of the merged column, and in a correlated subquery.

• A subscript or a slice of text, such as s[2], s[-1], s[2:3] and s[2:-2], when the bounds are whole numbers written in the query. DuckDB reads these as array_extract and array_slice, and on text both are a substring, so they lower to substring with the numbers worked out while the plan is built. A slice from the front to the back is two cuts. Every shape was checked against DuckDB for strings of up to six characters and every pair of bounds from minus eight to eight. A slice from the back to the front, a bound that is an expression, a step, and a subscript of a list are still refused.

• min and max on an ordered category column answer the category at the smallest and largest code, and nunique on any category column counts the categories in use, with dropna=False counting a missing row as one more. min answered the code itself and nunique refused the column. Every other reduction on a category column now raises pandas' TypeError rather than reducing the codes as numbers: min and max when the categories are unordered, pointing at as_ordered, and sum, mean, median, std, var, prod, sem, skew, kurt, any and all outright. quantile is refused as not supported yet, since pandas reads one off the codes by a rule of its own. (Issue #8)

• quantile takes interpolation="lower", "higher", "midpoint" and "nearest" and a list of quantiles, on a series and down the columns of a frame, all of which were refused. The four rules land on a value in the column or halfway between two, at the position numpy reads, which is q times one less than the count with no round trip through a percentage, and nearest rounds a half to the even position as numpy.around does. The column is sorted once and every position a list needs is read in one take. A list answers a series labelled by the quantiles on a series and a frame with a row per quantile on a frame, and a scalar under one of the four rules answers a series labelled by the columns on a frame. linear is still the kernel, one reduction per quantile. The eight newer rules numpy added are still refused by name, and so is a list or a rule on a groupby and a frame along its rows. The out of range errors now say what pandas 3 says, which is one sentence for a quantile and a list of them alike, and a sentence of its own on a groupby. (Issue #8)

• A string question asked of a missing row answers False, which is what pandas answers. That covers startswith and endswith with one prefix or a tuple of them, the ten is questions about case and kind, and contains, match and fullmatch down both the byte search and the regular expression engine. pandas fills that row with False even with na=None written out, and with na when it is anything else, and the accessor now does both. The kernel still answers null, and the fill happens once in the Python layer for every mask the accessor hands back. A tuple of prefixes also stopped losing track of a missing row when the tuple is empty, so startswith((), na=True) answers True there, as pandas does.

• A comparison on a row with a missing value answers what pandas answers, which is False for ==, <, <=, > and >= and True for !=, on a series and on a frame, through the operators and the named forms alike. The kernel still answers null there, which is the three valued answer, and the pandas facing operators fill it in, so nothing below them changes. This reverses the choice the compat repository's document 28 recorded and retires the compat divergence engine/comparison-null, because the library's goal is pandas' answer and a count of the False rows is exactly where a caller would notice the difference.

• skipna=False and min_count on every reduction of a series, nunique(dropna=False) on a series and a frame, and DataFrame.dropna(how="all"), all of which were refused by name. Measured against pandas 3.0.3: skipna=False answers NaN when a value is missing, on an integer column too, and the ordinary answer when none is, and min_count answers NaN when fewer values are present than it asks for, including on an integer column and an empty one. Both are applied after the reduction runs, so a column the reduction cannot read still raises the reduction's own error. nunique(dropna=False) counts every missing value as one more distinct value however many there are. dropna(how="all") drops only the rows where every column looked at is missing, with and without subset and inplace, and raises pandas' ValueError for an unknown how and its TypeError for how beside thresh. skipna=False on a temporal column with a gap is still refused, because pandas answers NaT there and firepanda has no NaT to give back, and the two flags on a whole frame's reductions are still refused as well.

• DataFrame.from_arrow, pandas' door from an Arrow table, which reads it into the types pandas reads it into. An integer column with a missing row comes back as float64 with a NaN in the gap, a float column holds its gap as a NaN and keeps its width, and a column with nothing missing keeps its type, all measured against pandas.DataFrame.from_arrow in pandas 3.0.3. firepanda.from_arrow is unchanged and still keeps Arrow's types, so there is a door with pandas' name and pandas' meaning and a door with firepanda's own. The widening is the widen_for_missing that read_csv already did, now reachable from Python as _widened_for_missing. (Issue #8)

• tz_localize, floor, ceil and round take pandas' ambiguous and nonexistent on a zoned column, on a series and on a DatetimeIndex, where every value but "raise" used to be refused. ambiguous takes "NaT", True, False and a list of flags with one per row, and nonexistent takes "NaT", "shift_forward", "shift_backward" and a timedelta. The shifts go by the whole hour of the clock rather than by where the gap ends, because that is what pandas does, so on Lord Howe, where the clock goes forward half an hour, a shift forward lands on the hour after the gap. A timedelta that leaves the reading in the same hour raises pandas' own ValueError naming the timedelta. The policy crosses the boundary once as two words and a shift, through a new temporal_placed method beside temporal_part, and a list of flags is answered by asking twice and taking each row from the side its flag names. ambiguous="infer" is still refused, since it reads the order of the rows rather than being told. (Issue #349)

Added: a column carries its encoding apart from its logical type

AnyArray has an encoding field and an is_flat method, and firepanda.array exports Encoding. Flat is the only encoding there is, so every column says flat and nothing reads the field yet. This is the first box of #979, which is about holding a low cardinality string column as dictionary codes while it still answers dtype string, and later about a filter handing back positions instead of a copy. Both of those are a different layout of the same logical type, so the layout needed a field of its own before either could land.

The field takes the eight bytes _slack used to occupy, and it is eight bytes wide for that reason. Those bytes keep AnyArray from ending short of its alignment, which Mojo miscompiles when the struct is held in an Optional (#286). The struct is the same size it was, and test_a_column_has_no_trailing_padding now counts the encoding where it counted the slack.

Changed: a pull request runs the tests its change can reach, not all 162

The test shards on a pull request now run tools/run_tests.sh --changed, which picks the test files that import something the branch touched and falls back to the whole suite whenever it cannot tell. A push to main still runs everything, so main is gated exactly as before. Over the last eight merged branches this runs nothing for the three that only touched the Python package, 30 files for a planner change, 135 for a kernel change, and the whole suite for the three that touched tools/bindings.py or the manifests. The shards still all start, since the selection is made inside each one, but a shard with nothing to run now finishes green in the time it takes to set up instead of failing.

tools/affected.py no longer treats CHANGELOG.md, docs/, or the Python package as reasons to run the whole suite. No Mojo test reads any of them, which was checked rather than assumed, and without this nearly every branch touched the changelog and selected everything. A .mojo file under python/ still selects the whole suite.

Fixed

• Arithmetic between two labelled operands answers NaN in a row only one side has, where it answered null, and widens an integer column to float64 to hold it, which is what pandas does. That covers a series with a series and a frame with a frame, through the operators and the named forms, and a frame with a series along either axis. Along the columns pandas lines the series up first, so an integer series missing one of the frame's column labels becomes a float64 series and every column of the answer is a float, including the ones with nothing missing, and that is copied too. The kernel still answers null there and the pandas facing layer turns it into NaN, the same way shift closes the gap it opens. Two operands with the same labels open no gap, so the null an integer division by zero writes is left as it is, which is the registered divergence engine/zero-divisor. A power is the one operator where pandas can answer 1 in a gap, because numpy's 1 ** nan is 1, and that is not copied yet. (Issue #8)

• median and a linear quantile now agree with pandas to the last bit, on a series, down the columns of a frame and on a groupby. The value between two neighbours was low + gap * part, which LLVM fused into one multiply add, so it could land one unit in the last place away from pandas. A median now averages its two middle values the way pandas does, a whole column quantile lands the way numpy's _lerp does, from the high side past a half, and a grouped quantile keeps the multiply apart from the add the way pandas' Cython does on x86. The SQL engine keeps its own rule. The rule a reduction lands by rides on AggKind as landing, which is not part of which reduction it is. (Issue #8)

• != against a NaN answered False in the vector loops of kernel/compare.mojo, both between two columns and against a constant, where IEEE, numpy and pandas all answer True. SIMD.ne lowers to the ordered compare, which is false on a NaN lane, while the scalar form of the same method is true, so the answer depended on whether the row landed in the vector body or the tail. The loops now take the inverse of equality.

• A program that reads a rule zone and also reaches the readers in firepanda.io builds again. Since #1010 the zone file was read with the standard library's open, and the Mojo 1.0.0 compiler stops partway through building such a program, every thread parked and nothing emitted, which is what the ClickBench driver in firepanda-bench had been doing since #1010 merged. The file is now read with fopen and fread, and the driver builds. Nothing about what a zone answers changed.