v0.8.38

What's Changed

• Answer cross and anti merges as pandas does by @tamnd in #1338
• Read a list of integers with a gap as float64 by @tamnd in #1339
• Answer str.partition and str.rpartition with expand=False by @tamnd in #1340
• Answer seven parameters that used to refuse by @tamnd in #1341
• Count days, hours, minutes and floats in to_datetime, and fill every gap in align by @tamnd in #1342
• Add pivot_table margins by @tamnd in #1343
• Refuse pivot_table margins beside column keys that are not text by @tamnd in #1344
• Add the masked text type string by @tamnd in #1346
• Reduce across rows, and sort with a key by @tamnd in #1347
• Take a positional axis on the reductions, with pandas' warning by @tamnd in #1348
• Apply with raw and result_type, and transform with lists by @tamnd in #1349
• Take skipna, min_count and numeric_only on group by reductions by @tamnd in #1350
• PIVOT with no IN list pivots on the values the column holds by @tamnd in #1351
• Widen where, mask and shift the way pandas does by @tamnd in #1352
• CHANGELOG: put the PIVOT entry at the top of Unreleased by @tamnd in #1353
• A VALUES column takes the type that holds every row by @tamnd in #1354
• Keep the row labels' frequency and shift by a frequency by @tamnd in #1355
• Take skipna=False on group first and last, and keep picked categories by @tamnd in #1356
• Hold flags beside a gap as objects, as pandas does by @tamnd in #1357
• Write flags cast to text as True and False by @tamnd in #1358
• Fuse a group by whose text key is coded by @tamnd in #1359
• Put a blank line before changelog headings by @tamnd in #1360
• Take a dropna of its own on group value counts by @tamnd in #1361
• Read a gap on a level of numbers back as NaN by @tamnd in #1362
• Take dtype on the Index constructor and read instants and spans by dtype by @tamnd in #1363
• Parse instants asked for by dtype one by one by @tamnd in #1364
• CAST of a date or a timestamp to VARCHAR writes the instant by @tamnd in #1365
• Hold an index of objects as pandas does by @tamnd in #1366
• Hold an index of a masked type as pandas does by @tamnd in #1367
• Keep a masked key's type on the answer by @tamnd in #1368
• Keep a masked column's type under group reductions by @tamnd in #1369
• Export a masked column to Arrow at its own width by @tamnd in #1370
• Parse text through astype to instants and spans, and close three smaller gaps by @tamnd in #1371
• Read each text on its own when astype parses instants by @tamnd in #1372
• Read a list of tuples as levels where pandas does by @tamnd in #1373
• Give the axes and names the setters pandas has by @tamnd in #1374
• Refuse text and number arithmetic with pandas' TypeError, let df.a write column a by @tamnd in #1375
• Sum and scan text as pandas does, refuse the other reductions in pandas' words by @tamnd in #1376
• Decay ewm by real instants with times= as pandas does by @tamnd in #1377
• Take a DatetimeIndex and read infinities as gaps in ewm times= by @tamnd in #1378
• Read infinities in rolling, expanding and ewm windows as missing values as pandas does by @tamnd in #1379
• Lay resample bins from origin= end, end_day or a timestamp and move them by offset= as pandas does by @tamnd in #1380
• Answer float64 counts and positions for text with a gap as pandas does by @tamnd in #1381
• Name the column axis as pandas does by @tamnd in #1382
• Pivot into columns of several levels as pandas does by @tamnd in #1383
• Total pivot tables beside several values as pandas does by @tamnd in #1384
• Refuse a window over a column that is not numbers with pandas' DataError by @tamnd in #1385
• Take labels as well as column names in set_index as pandas does by @tamnd in #1386
• Resample on weeks, months, quarters and years as pandas does by @tamnd in #1387
• Drop missing values with thresh, across the columns and numbering again by @tamnd in #1388
• Print boolean column names flush left to one width as pandas does by @tamnd in #1389
• Concat with keys across the columns as pandas does by @tamnd in #1390
• Describe moments and spans, and pick columns by type by @tamnd in #1391
• Take tz, dtype and copy in the temporal index constructors by @tamnd in #1392
• Drop repeated row labels, refuse a lossy transpose, keep categories handed in whole by @tamnd in #1393
• Make CategoricalIndex a class with pandas' lookups and category methods by @tamnd in #1394
• Look labels up beside a gap in a CategoricalIndex, and map a gap as pandas does by @tamnd in #1395
• Name a missing row label in drop as pandas does by @tamnd in #1396
• Answer in and drop for a label of another type as pandas does by @tamnd in #1397
• Settle a repeated or skipped hour in Timestamp.tz_localize as pandas does by @tamnd in #1398
• A zone on one level of a MultiIndex, and numpy codes from factorize by @tamnd in #1399
• A list of reductions over a window of a frame by @tamnd in #1400
• corr and cov of every pair of columns over a window by @tamnd in #1401
• rename of the row labels by @tamnd in #1402
• mean, median and the spreads over a whole frame by @tamnd in #1403
• Arithmetic on one level of a MultiIndex by @tamnd in #1404
• Keep int64 when a Series covers every MultiIndex column by @tamnd in #1405
• Fold flags beside numbers or text down the columns as pandas does by @tamnd in #1406
• Reduce rows of flags beside numbers as objects as pandas does by @tamnd in #1407
• SET, RESET, PRAGMA, PREPARE and enable_external_access by @tamnd in #1408
• Scan across the rows of a frame and down object columns as pandas does by @tamnd in #1410
• Rank, shift, diff, pct_change and fill across the rows as pandas does by @tamnd in #1411
• Count values in bins and verify set_index labels as pandas does by @tamnd in #1412
• Take a list of quantiles and the picking rules in a group by's quantile by @tamnd in #1413
• Take a slice of places in a group by's nth by @tamnd in #1414
• Sort the index with a direction per level as pandas does by @tamnd in #1415
• firepanda.sql and DataFrame.sql run a query over the frames in scope by @tamnd in #1416
• Shift by a list of periods as pandas does by @tamnd in #1417
• Take DataFrame.quantile across the rows as pandas does by @tamnd in #1418
• Unstack several levels at once by @tamnd in #1419
• Answer at on a repeated label as pandas does by @tamnd in #1420
• Series.compare with align_axis=0 by @tamnd in #1421
• DataFrame.mode along the rows by @tamnd in #1422
• from_records with several index columns by @tamnd in #1423
• Group idxmax and idxmin with as_index=False by @tamnd in #1424
• Group by the levels of a MultiIndex by @tamnd in #1425
• Groupers and levels beside other keys i...

v0.8.37

Built against Mojo 1.0.0 (ed45d567).

A patch release. On the pandas side it brings the readers and writers pandas has beyond CSV and Parquet (Excel, Stata, SAS, HDF, SQL over sqlite3, HTML, XML, LaTeX, the clipboard, Iceberg and SPSS), periods and interval columns, categoricals, object, sparse and masked columns, rows and columns labelled by a MultiIndex, DataFrame.style with to_excel, plotting with matplotlib, firepanda.testing, and pandas' byte counts, str dtype and float answers for var, std, sem, skew and kurt. SQL gains LATERAL subqueries, ASOF joins, PIVOT and UNPIVOT, recursive CTEs, named windows, window frames and CREATE, INSERT and DROP. The group by skips text keys another key already decides, packs two wide integer keys without factorizing either and uses every core on half a million rows or more, and the ClickBench driver builds again after a compiler hang that came in with the first of those.

Changed: Byte counts are pandas' byte counts

Series.nbytes, Index.nbytes and memory_usage on the frame, the column and the index now answer the number pandas answers, counted from the numpy or Arrow arrays pandas would hold the data in: a width a row for numbers, a mask byte on top for a masked type, offsets and UTF-8 bytes and a bitmap for text, codes and categories for a category, a pointer a row for objects with deep adding each object, and the Python range and its three numbers for a RangeIndex. info spells text str and objects object, and writes pandas' + when object memory was left out. A column or frame built from columns that carry the plain range from zero now keeps a RangeIndex rather than turning it into stored integers, which also fixes their repr.

Changed: a series' var, std and sem are pandas' floats

Series.var, Series.std and Series.sem of a column of numbers take the squared distances from the mean numpy sums and leave its error alone, the steps pandas' nanvar takes, where the kernel corrected for that error. On five values near two to the fifty two that is pandas' 37.25 where exact arithmetic gives 37.2, and skew agrees with pandas' twelve percent on the same column. Without numpy the kernel still answers.

Changed: skew and kurt of whole numbers are pandas' answers

Series.skew and Series.kurt cast a column of whole numbers to float64 before taking its moments and add in numpy's order, step by step as pandas' nanskew and nankurt do, so near two to the sixty two they answer pandas' float rather than the exact value a few parts in ten million away. A float column's skew now goes the same way, which was already how its kurtosis went.

Changed: a text column's dtype is pandas' str

Series.dtype, Series.dtypes, Index.dtype and DataFrame.dtypes answer pandas' StringDtype with NaN for a gap for a column of text, which prints as str the way pandas 3 prints it and still equals string, so code comparing against either word keeps working. Everything below the surface reads a dtype back through one helper that gives the core's word, so no internal decision changed. A column made with dtype="string" is also spelt str, because firepanda does not yet keep the NA variant apart.

Added: a map's key and value types, and histogram's MAP(K, UBIGINT)

A MAP type now carries its key and value types, written MAP(K, V) as DuckDB writes them, and two maps are the same type only when both agree. histogram answers MAP(K, UBIGINT) for a key of any flat type, including the bins form, as DuckDB does, so the differential semantics check now compares its full type rather than the overload alone. A map whose key or value is itself nested is still a bare MAP, and says nothing about what it holds.

Changed: Rolling.max takes pandas' extra arguments

Rolling.max is the one window reduction pandas declares with *args before its engine arguments and **kwargs after them, and its body never reads either. firepanda declared the same three parameters as the other reductions and refused anything else, which was a registered divergence. The signature now matches pandas, and the extra arguments are dropped as pandas drops them. Expanding.max declares neither in pandas and still refuses them here.

Added: Styler.to_excel

Styler.to_excel now writes a styled frame to a workbook as pandas does. Each cell's CSS goes through pandas' CSSToExcelConverter, ported without numpy along with its CSS reader, and the openpyxl, xlsxwriter and odf writers turn the result into the same fonts, fills, borders, alignments and number formats pandas writes, including across merged cells. firepanda.io.formats.excel and firepanda.io.formats.css hold the converter and the reader where pandas keeps them. A loc key on rows labelled by several levels that holds a slice or a list for fewer levels than there are, such as loc[(slice(None),)], now keeps every level as pandas does, rather than dropping the ones it named.

Added: DataFrame.style

DataFrame.style now answers pandas' Styler, ported from pandas and run over firepanda frames, when jinja2 is installed. It writes a frame as HTML, LaTeX, text or Typst with the same formats, highlights, bars, gradients, tooltips, hidden rows and columns, sticky headers, table styles and concatenated footers as pandas, and the same text for each, down to the element ids. Without jinja2 the accessor raises pandas' own AttributeError. The gradients and colormapped bars need matplotlib, as they do in pandas. firepanda.io.formats.style.Styler is importable where pandas keeps it. Styler.to_excel is not ported yet. A column of a frame that was transposed twice now hands its labels back from items as the values they are, not as the text the extension holds them as.

Added: sequence answers in apply, level keys in loc and numbers compared with text

DataFrame.apply now frames a list, a tuple or an array answered for each column the way pandas does, labelled by the rows when the answers are as long as the frame. loc reads a row key with a slice or a list for some level, such as IndexSlice["A", :], level by level through MultiIndex.get_locs, which now answers rows in the order its lists name them, and loc[:, "a"] on rows with levels keeps every level. A column of numbers or flags compared with text is False for == and True for !=, and an order between them raises pandas' TypeError.

Added: DuckDB's .test files run against firepanda, with a floor

pixi run conformance runs DuckDB's own .test files from the vendored corpus through firepanda and checks each against what the file says DuckDB answers: every statement's success or error and every query's rows, whether sorted, hashed or labelled. The files are run unmodified. A file that needs an extension, a database file or the corpus's data directory is counted as not run, and every failing file is put in the bucket of its first failure: a refusal by name, a missing function, a divergence, a wrong answer or a crash. The per directory table over the target directories is in the README with its denominators, and tests/conformance_floor.txt holds the count that passes in each, which CI checks so a change that lowers it fails. --all runs every directory and writes the failures to build/conformance/failures.txt.

Added: HDFStore, read_hdf and to_hdf

HDFStore, read_hdf, DataFrame.to_hdf and Series.to_hdf read and write HDF5 files in pandas' fixed and table formats, and firepanda.io.pytables holds the storers where pandas keeps them. The module is pandas' own pandas.io.pytables ported class for class on top of PyTables, which is imported only when a store opens. Columns are grouped into blocks and consolidated by pandas' rules, so a file firepanda writes has the same nodes, attributes and types as the one pandas writes and each library reads the other's files. Where clauses, data columns, appends, removal, chunks, coordinates, several tables at once, copies, table indexes, the printed store and the errors all match pandas.

Added: read_sas and the SAS readers

read_sas reads SAS transport files and SAS7BDAT data sets the way pandas does, and firepanda.io.sas holds SASReader, XportReader and SAS7BDATReader where pandas keeps them. Both readers are ported to plain Python with struct, including the IBM float conversion of transport files and the run length and Ross decompressors of data sets, so they need no other package. Names, types, values, dates, chunks, reader attributes and errors match pandas on pandas' own SAS test files, with and without an encoding.

Added: DataFrame.to_stata and the Stata writers

DataFrame.to_stata, StataWriter, StataWriter117, StataWriterUTF8, StataStrLWriter, StataValueLabel and StataNonCatValueLabel now write Stata dta files in versions 114, 117, 118 and 119. The writer is a pure struct port of pandas' own, so with the same time stamp the file is byte for byte the one pandas writes, including type narrowing, renamed columns, value labels, dates in every Stata format, long strings, byte order and compression, and it raises pandas' warnings and errors in pandas' words.

Added: read_stata and StataReader

read_stata reads Stata dta files of versions 102 to 119, whole or in chunks through StataReader, which also lives at firepanda.io.stata with StataMissingValue. It is a port of pandas' reader that uses struct in place of numpy, and it follows pandas for missing values, the nine date formats, long strings, value labels as ordered categoricals, preserve_dtypes, columns, index_col and compressed files, with pandas' warnings and error sentences.

Added: convert_dtypes with the pyarrow backend

convert_dtypes(dtype_backend="pyarrow") now gives Arrow types as pandas does, such as int64[pyarrow], double[pyarrow], string[pyarrow] and timestamp[us][pyarrow], keeping each column's width and following the four switches, while ...

v0.8.36

Built against Mojo 1.0.0 (ed45d567).

A patch release. On the pandas side, df[key] = value writes columns into a frame, groupby(...).sample and json_normalize arrive, an index, a frame and a column print what pandas prints, NaT and missing values read out the way pandas spells them, and to_datetime takes dayfirst, yearfirst, the separators ISO8601 allows, numbers among moments and moments mixed with text. The ClickBench group by got faster again: the text walk prefetches the row a pending slot names, and a tuple of keys is read back from the map once at the end rather than gathered from every chunk, which together take about a tenth off q34, a quarter off q39 and more than a quarter off q18 at 1M rows.

Added: Writing a column into a frame with df[key] = value

DataFrame.__setitem__ and DataFrame.__delitem__ now exist. A name puts one column, lined up on the row labels when it is a series, a list of names puts several from a frame, from rows or from one value, and a column or frame of true and false puts the value in the rows or cells it marks. The write rebinds the frame's one slot to what assign or mask would answer, as inplace does, so a copy or a column taken before the write keeps what it had, which is pandas' copy on write. None and a function are refused, since pandas holds them in an object column.

Added: Exact kurtosis of whole numbers far from zero

Series.kurt and DataFrame.kurt on a whole number column now measure the values from the column's least value as whole numbers before the cast to float, as skew already does. On values near two to the sixty two the cast alone moved each value by up to 512 and the kurtosis in its seventh digit. The answer is now within a few bits of exact arithmetic, and within the suite's tolerance of pandas.

Added: Names and text read from a DatetimeIndex with a gap

day_name, month_name and strftime on a DatetimeIndex with a gap raised since the gap handling for whole number parts, because the text went through the float path. They now answer the text with the gap shown as NaN, as pandas does. time, timetz and to_pydatetime hand out NaT for a gap where they handed out None.

Added: Numbers among moments in to_datetime, and text among spans in to_timedelta

to_datetime of a list mixing a Timestamp, datetime or date with a number now reads the number as pandas does, as nanoseconds since 1970, and holds the column in nanoseconds. A NaN beside a moment is a gap, a flag raises pandas' TypeError, and a zoned moment beside a number raises pandas' mixed timezones ValueError unless utc is set. to_timedelta of text mixed with spans now answers microseconds, or the finer unit of the spans, where it answered nanoseconds, and keeps nanoseconds when a number is mixed in.

Added: to_datetime of moments mixed with text

to_datetime of a list mixing Timestamp, datetime or date values with text now reads as pandas reads it, taking each moment as it is and reading the text with the format worked out from the first piece of text, where it refused the list as a column of mixed kinds. format, dayfirst, errors="coerce" and utc apply to the text as before, and under utc the moments are put on UTC too.

Added: Instants and spans among objects in a frame's values

When a frame's values or to_numpy() is an object array, an instant or a span in it is now a Timestamp or a Timedelta and a gap is NaT, as pandas gives them, where numpy had put a datetime, a timedelta and None.

Added: Instants and spans in different units meet as in pandas

concat of instant or span columns, or of instant row labels, in different units now keeps the finest unit, as pandas does, where it refused them as pandas' object column. A mix of zones, or of instants and spans, is still refused, since pandas answers an object column there. astype to the same kind in another unit, such as astype("datetime64[ms]"), is now as_unit on a series and on a frame, and dropping a zone that way raises pandas' TypeError.

Added: Calendar parts of a column with a gap, typed as in pandas

A whole number read through dt or a DatetimeIndex from instants or spans with a gap, such as year, hour, dayofweek or days, is now float64 with NaN in the gap, as pandas gives it, where it was int32 with a gap. The yes or no fields, such as is_month_start and is_leap_year, answer False for a gap as pandas does, where they answered missing. Without a gap nothing changes.

Added: NaT, the missing moment and span

firepanda.NaT is pandas' missing moment and span: one value, a datetime that never equals anything, sorts against nothing and turns the arithmetic it meets into itself, with its fields NaN and the methods pandas refuses refused with pandas' words. A gap in a column of instants or spans now reads out as NaT through tolist, iteration, iat, a frame cell and an index, Timestamp(None), Timedelta(None), to_datetime(None) and to_timedelta(None) answer it, and a reduction over only gaps answers it where it answered NaN. NaT goes back in wherever pandas takes it, so a column built from it, fillna, where, replace, isin, get_loc, in and adding it to a span column all match pandas. Replacing a moment or a span, which never matched a row before, now does.

Added: Read a missing value out the way pandas spells it

A gap read out of a column through tolist, iteration, iloc, iat, loc, at or items now comes back as pandas hands it out for the column's type. A float column, an integer column with a gap, a text column and a category column hand out nan, and a column of flags hands out None, as an object column does in pandas. A frame cell and an index read the same way, a category's missing row has a code of -1, and a NaN given back among text or asked for by get_indexer finds the gap. A missing instant or span still reads as None, since firepanda has no NaT yet.

Changed: a tuple of keys is read back from the map at the end

A group by on several keys with text among them writes each row's keys out as bytes and looks those up in the text map, which already stores every distinct tuple. Each chunk also gathered its new groups' keys out of its own columns, a second copy of the same values, and then found its new groups again with a serial pass over the ordinals. The text map now hands back the rows it already knew were new, and the key columns are read out of the stored tuples once, on the cores, when the operator finishes. On a six core Linux machine, alternating the old and new driver over eight rounds, ClickBench q18 went from 106 to 84 ms at best, and every answer matched.

Added: groupby sample

groupby(...).sample draws rows at random from each group, the same rows pandas draws for the same seed. pandas walks the groups in order and asks numpy's choice for each one's positions from one random state, and this walks them the same way, so n, frac, replace, weights as a list or a column name, sort=False and dropna=False all give pandas' rows in pandas' order, and a size or weights pandas refuses are refused with its words. The size and weight checks moved out of DataFrame.sample into two helpers both now share. 11 calls and 7 mistakes match pandas.

Added: json_normalize

firepanda.json_normalize flattens nested records into a frame the way pandas.json_normalize does. It is a port of pandas' own code: nested dicts become dotted columns, sep and max_level change how, record_path pulls the rows out of lists nested in each object, meta repeats outer fields on every row, the two prefixes rename columns, and errors='ignore' fills a missing field with NaN. The error messages are pandas' words. A cell left holding a dict or a list needs a column of objects, which firepanda does not have, so that case still raises. 14 calls and 6 mistakes match pandas.

Added: An index prints what pandas prints

repr and str of an Index, a DatetimeIndex and a TimedeltaIndex used to print the core's summary, so a zoned index printed the counts behind its instants. They now go through a port of pandas' format_object_summary: the labels in brackets, wrapped at display.width, cut to ten from each end past display.max_seq_items with dots between, then the type, the name, the length when cut, and freq for instants and spans. Labels that are still a range print as a RangeIndex. Dates at midnight print as dates, whole days as days, text is quoted with its tabs and newlines escaped, and a gap prints as pandas prints it for each kind. 24 shapes and 16 option combinations match pandas.

Changed: the text walk prefetches the row a pending slot names

A chunk that brings most of its own keys compares most of its rows against the row of the same chunk that first opened the key's slot, not against a stored key, and that comparison was the one the walk did not prefetch. The walk now asks for that row's view and then its bytes ahead of time, as it does for a stored key, and compares the two rows by their view words without the validity checks a key column with no nulls does not need. On a six core Linux machine that was busy with other work, alternating the old and new driver over eight rounds, ClickBench q18 went from 108 to 96 ms at best, q34 from 88 to 79 and q39 from 169 to 131, and every answer matched.

Added: A frame and a column print what pandas prints

repr and str of a DataFrame printed a summary of its columns, and a column of instants with a zone printed their counts. Both now go through the text formatter under pandas' display options, so a frame prints its rows, a long one loses its middle to dots, a wide one is cut to fit the terminal or wrapped at display.width, and a zoned instant prints as a time with its offset. to_string takes max_rows, min_rows, max_cols and line_width instead of refus...

v0.8.35

Built against Mojo 1.0.0 (ed45d567).

A patch release. On the pandas side, MultiIndex, RangeIndex, TimedeltaIndex and Interval arrive, along with merge_asof, merge_ordered, align, pivot and pivot_table, crosstab, cut and qcut, read_json and to_json, to_csv, to_numpy, to_numeric, to_timedelta, date_range with calendar steps, and map, apply, agg and transform on frames and columns, and to_datetime reads formats the way pandas guesses them. In SQL, right, full, cross, ASOF and POSITIONAL joins run. The ClickBench group by got faster: a lasting tuple is written a key at a time, a stored text key is compared through a flat head the walk prefetches, the bucket pass counts through a pointer, and a grown table is zeroed on the cores, which together take about a tenth to a quarter off q15, q18, q33, q34 and q39 at 1M rows.

Changed: a grown lasting table is zeroed a part at a time on the cores

When a lasting map grows its table for a chunk that could be all new keys, the new table can be tens of megabytes, and it was zeroed on the calling thread with its page faults before the parts were moved in on the cores. Each part now zeroes its own share as it moves. On a six core Linux machine that was heavily loaded by other work, alternating the old and new driver over six rounds, ClickBench q15 went from 72 to 58 ms at best and q34 from 186 to 174, and every answer matched.

Changed: a lasting map counts and scatters its buckets through a pointer

The pass that sorts a chunk's rows by part indexed a list of counters on every row, which checked the bound and reloaded the list's length each time. It now keeps its morsel's counters through a pointer. On ClickBench q15 the two passes went from about 15% of the query's CPU to about 8%, and the best time from 31.8 to 29.4 ms on a six core Linux machine busy with other work, with every answer the same.

Added: to_datetime guesses a format from the first row the way pandas does

to_datetime with no format used to read ISO 8601 and refuse anything else. It now guesses a format from the first value that is not missing, as pandas' guess_datetime_format does, and holds every row to it, so 01/02/2026, Jan 2, 2024, 31.12.2024 and Sat, 06 Jan 2024 10:00:00 +0000 read as they do in pandas. When no format can be guessed each row is read on its own, as pandas does. A strptime format the core does not read, such as one with %B, %I and %p, %j or %Z, is read the same way, with pandas' words for a value that does not match, for data left over, and for a day past the end of its month. Labels in read_json are still read as ISO 8601 only, which is what pandas does there.

Added: tz_convert(None) and tz_convert with a tzinfo

tz_convert(None) on a DatetimeIndex or through .dt now moves the instants to UTC and takes the clock off, as pandas does, where it used to be refused. tz_convert also reads a zoneinfo.ZoneInfo or datetime.timezone.utc as the zone it names instead of refusing anything that is not a string. Converting instants that carry no zone raises a TypeError with pandas' own sentence, without the explanation the core adds after it.

Added: as_unit names the first instant nanoseconds cannot reach

as_unit("ns") on a DatetimeIndex or through .dt now raises OutOfBoundsDatetime with pandas' own sentence, Out of bounds nanosecond timestamp: followed by the first instant outside the range on the UTC wall clock and to the second, where it used to raise a type error from the core. An except clause written against pandas now catches it. Instants just inside either end of the range still restate.

Added: to_datetime answers an index for a list and an instant for one value

to_datetime now answers what pandas answers for each kind of input: a column for a column, a DatetimeIndex for a list, tuple or index, keeping the index name, and one instant for one value, with None where pandas gives NaT. A list with nothing to read is held at seconds, as in pandas. The DatetimeIndex constructor now reads text every row with its own format, the way pandas does, so a list of dates in different shapes builds an index instead of raising. A DatetimeIndex can be put in a frame or a column directly, and a column built from an index takes the index name.

Added: to_datetime reads every row with its own format

to_datetime(format="ISO8601") and to_datetime(format="mixed") used to be refused by name. Both now read each row with its own format, as pandas does. ISO8601 holds every row to the ISO 8601 shapes pandas reads. mixed also reads the common shapes dateutil reads: slashed and dotted dates, month first unless the first number cannot be a month, month and weekday names, a clock with AM or PM, and a two digit year in the century nearest today. The unit, the zone, utc, errors="coerce" and pandas' own errors for text that will not read and for rows at more than one offset all match pandas. A column whose rows share one shape still reads in a single pass through the core.

Added: at_time, between_time and asof on a column and a frame

Series and DataFrame now have at_time, between_time and asof. The first two choose rows by the time of day of labels that are instants, through the index indexers, and raise pandas' Index must be DatetimeIndex on any other labels. asof finds the last row at or before each label that holds values, skipping missing values on a column and rows with a missing value in subset on a frame. One label answers a value or a row named by the label, several answer a column or a frame on them, and rows that found nothing come back missing with whole numbers widened to floats, as in pandas.

Changed: the paths written for a dictionary encoded column ask for it by name

Fifty one kernels and string methods that have a fast path over a dictionary encoded column's codes used to take it for any column not held flat. They now ask is_coded(), so a column in an encoding they were not written for reaches the check in front of the values and is refused with a message naming decoded(), rather than having its buffer read as codes. Nothing is held in such an encoding yet. This is the ground the next encoding of #979, a column held as positions into another, needs before anything builds one.

Changed: an exchanged inner join sorts its pairs back on every core

When an inner join's left side is much shorter than its right, the join builds on the left and walks the right, then counting sorts the pairs back into left row order so the exchange stays invisible. That sort ran on one thread. On TPC-H q2, 2,000 European suppliers against 800,000 part supplies, it was 7% of the query's cycles. A pairing of at least 131,072 entries now gets cut into one run per worker. Each run counts its own left rows, the prefix sum goes over the left rows and, within a row, over the runs in order, and each run scatters into places no other run writes, so the order is the same stable one the single thread gave. The two output lists are no longer zero filled before the scatter overwrites them. On a busy 8 core VM the sort dropped to under 2% of q2's cycles, spread across the cores, and q2's best time went from 102.6 to 84.5 ms with the two builds run back to back.

Added: The members only an index of instants has

DatetimeIndex now has shift, snap, indexer_at_time, indexer_between_time, isocalendar, mean, std, to_julian_date, to_pydatetime, time, timetz, tzinfo and resolution. shift and snap count along calendar frequencies such as MS, W-MON and B with the steps date_range already uses, and a label between two landing dates rolls the way pandas rolls it. The time of day indexers read times the way pandas does, including the evening time pandas reads a missing label as, so a range that wraps past midnight takes it in as it does in pandas. Slicing or picking labels out of an index of instants or spans now keeps its type. shift without freq raises NullFrequencyError, since firepanda keeps no frequency on an index.

Added: Index join, asof and the members around them

Index and every index built on it now have join, asof, asof_locs, get_indexer_for, get_indexer_non_unique, sortlevel, groupby, view, array, shift and a str accessor that answers indexes. join follows pandas down each of its three paths, the sorted join, the merge join for repeated labels and the lookup join, so the rows and the positions come back in pandas' order, None included where pandas answers it. A join of text with numbers is refused, since the answer would be an index of mixed values. get_indexer on an index of instants or spans now reads labels written as text, where it used to fail.

Added: MultiIndex

MultiIndex is here, with every member pandas declares on it spelled the way pandas spells it. It is built with from_arrays, from_tuples, from_product, from_frame or out of levels and codes, and keeps what pandas keeps: sorted levels, codes with -1 for a gap, and level names. Lookups (get_loc, get_locs, get_loc_level, get_indexer, isin, slice_locs), reshaping of levels (droplevel, swaplevel, reorder_levels, set_levels, set_codes, remove_unused_levels), sorting, set operations, append, insert, drop and the duplicate members answer as pandas does, arrays as numpy arrays when numpy is installed. What needs a frame or column labelled by a MultiIndex, or a column of tuples, is refused by name for now: to_frame(index=True), to_series, to_flat_index and value_counts.

Added: cov, corr and aggregate on a decay

ExponentialMovingWindow gains cov, corr, aggregate and agg. cov and corr follow pandas' own recurrence, with adjust, ignore_na, min_periods and bias, and only the rows where both sides hold a value count. They pair a column with a column, a...

v0.8.34

Built against Mojo 1.0.0 (ed45d567).

A patch release. On the pandas side, a frame and a column gain interpolate, corr, cov, autocorr, unique, factorize, describe, kurt, melt, get_dummies, query, eval and resample, a group by gains ffill, bfill, pct_change and filter, and iloc reads one row across columns and raises pandas' IndexError past the end. In SQL, a semi or anti join checks the rest of its condition, so TPC-H q21 runs. Text is hashed sixteen bytes a step in two lanes, which makes hashing a short string about four times faster and helps every text group by and join.

Fixed: a gap interpolate leaves is NaN

The gaps interpolate leaves behind, at the front of a forward fill, past a limit or outside a limit_area, and every row of a column with no value at all, are NaN now rather than null, since NaN is the only missing value a float column has in pandas. Before this they crossed to Arrow as nulls and a consumer counting nulls saw a different column from the one pandas answers.

Added: interpolate on a frame and a column

Series.interpolate and DataFrame.interpolate fill each gap on the straight line between the values either side of it, with numpy's arithmetic so the answers agree to the last bit, along row positions for linear and along the labels for index and values. limit, limit_direction and limit_area leave the gaps pandas leaves, whole numbers or flags with no gap come back untouched, whole numbers with a gap come back as float64, and inplace=True hands back the object as pandas does. Text, flags with a gap and a category are refused with pandas' errors, every bad argument with pandas' message in pandas' order, and the methods pandas hands to scipy as not supported.

Added: corr, cov and autocorr on a frame and a column

DataFrame.corr and DataFrame.cov answer the square frame pandas does, labelled by the numeric columns both ways, pairing each two columns over the rows where both hold a value. Series.corr, Series.cov and Series.autocorr answer one number, lining the two columns up by label first. Pearson and Spearman are both there, min_periods and ddof behave as in pandas, including the difference in DataFrame.cov between a frame with a gap and one without, and text is refused with numpy's words unless numeric_only is set. Kendall is refused as not supported yet.

Added: a row position past the end raises pandas' IndexError

df.iloc[9999] on a short frame raises IndexError with pandas' message, "single positional indexer is out-of-bounds", from either end. It used to pass the position to the core and raise the core's slice message.

Added: one row of a frame as a series, and names that are numbers

df.iloc[0], df.iloc[-1, [1, 0]] and df.loc[label] read one row across several columns, which used to raise. The answer is a series labelled by the column names and named by the row's label, and it holds the one type every column fits: columns of one type keep it, and numbers meet the way numpy's do, so int32 next to int8 is int32 and float32 next to int64 is float64. A number next to a flag or to text is an object column in pandas, and that is still refused with NotImplementedError, since firepanda has no object column.

A series name can be a number now. The core keeps a name as text, so the wrapper keeps the number next to it and name answers the number while the core's text still spells it. rename(5).name is 5, a row's name is its label, and DataFrame.quantile(0.5).name is 0.5, where it used to be None. quantile(1) is named 1.0, as in pandas. An operation that makes a new column carries the name as text, so rename(5) + 1 is still named "5".

Fixed: a column labelled with empty text exports to Arrow

A frame with a column labelled "" could not be read by pyarrow, which raised a SystemError from pa.table, because the column's schema went out with a null name, the form Arrow keeps for an array with no name at all, and pyarrow refuses a struct child without one. get_dummies makes that label whenever a column holds an empty string. A frame's column now always carries its name, empty or not, and a lone column or index with no name still goes out with a null one.

Added: unique and factorize

Series.unique and firepanda.unique give each value once in the order first seen, with a missing value kept once, and a categorical column keeps every one of its categories, used or not, as in pandas. The answer is a FirepandaArray, which is pandas' array answer here: values with a type and no row labels or name, with a length, positions, slices and the Arrow PyCapsule export, and Series.array gives one too. Series.factorize, Index.factorize and firepanda.factorize give an int64 array of codes and the uniques, as an index for a column or an index and as an array for an array, with sort and use_na_sentinel doing what they do in pandas, a float NaN included. The codes are the group numbers of a group by, so no Python loop runs over the rows. A list is refused with pandas' own TypeError, and factorize on a categorical column is refused, because its uniques are a categorical index, which firepanda does not hold yet.

Added: pandas' aliases and small helpers on a frame and a column

A frame and a column now have div, divide, rdiv, multiply and subtract as the names pandas gives the arithmetic already there, isnull and notnull beside isna and notna, and add_prefix, add_suffix, pipe, first_valid_index, last_valid_index, pop, equals and to_dict. A frame has insert, which puts a column at a position and refuses a label it already holds, as pandas does, and a column has item, to_list and is_unique, where two missing values count as the same value. firepanda.isna, notna, isnull and notnull answer for a scalar, a column or a frame. to_dict takes each of pandas' orients for a frame and a mapping class for into, and each signature matches pandas' parameter for parameter. UndefinedVariableError is now built from the name and whether it was the caller's, the way pandas builds it, so the compat signature check agrees.

Added: pandas.get_dummies

firepanda.get_dummies turns a column, a list, or the text and categorical columns of a frame into one column of flags per level, the way pandas does. The levels are the categories in order for a categorical column, unused ones included, and the distinct values sorted for anything else. prefix and prefix_sep take one value, a list with one per encoded column, or a mapping, and dummy_na, columns, drop_first and dtype all work, with pandas' messages for a prefix list of the wrong length, a columns that is not a list and an object dtype. sparse=True is refused, and so is a level that would label a column with something other than text, such as a number with no prefix, since a firepanda column label is text.

Added: a semi or anti join asks the rest of its condition, and TPC-H q21 runs

A semi or an anti join, and the correlated EXISTS and NOT EXISTS that become one, now take a condition with more than equalities in it, such as EXISTS (SELECT 1 FROM lineitem l2 WHERE l2.l_orderkey = l1.l_orderkey AND l2.l_suppkey <> l1.l_suppkey). The equalities pair the rows up and the rest is asked of each pairing, and a left row is kept when some pairing passes (semi) or none does (anti). The plan shows it as JOIN semi [a = k] where k > b and the JSON form writes it under "where". This is the last piece TPC-H q21 needed, so all 22 TPC-H queries now answer the same as DuckDB (#816).

Added: DataFrame.query and DataFrame.eval

query keeps the rows where an expression over the columns is True, and eval answers the value of one, or the frame with a column assigned for c = a + b. The expression is read with Python's own parser and pandas' three changes: & and | bind looser than a comparison, a name in backticks may hold spaces, and @name reads a variable of the caller, or of local_dict and global_dict. Chains such as 1 < a < 4, membership with in, not in and == against a list, the row labels as index or by their name, abs, and methods such as b.isna() and s.str.startswith('x') all work, each node worked out over whole columns. A name that is nothing the query can read raises the new firepanda.errors.UndefinedVariableError, a subclass of NameError, and the nodes pandas does not read, such as is and if else, raise NotImplementedError as pandas does. inplace works for both.

Added: DataFrame.resample, Series.resample and the resampler

resample bins a datetime index, or the column on= names, into fixed steps of time, from nanoseconds to days, with multiples and fractions such as 90min and 1.5h. It is the group by firepanda already has, keyed on the bin number of every timestamp, with every bin from the first to the last in the answer. An empty bin holds what pandas puts there: 0 for sum, count, size and nunique, 1 for prod, and NaN for everything else, which widens an integer column to float64. closed, label and the origins start_day, start and epoch place the bins the way pandas places them, and a rule in days always starts at midnight of the first day. The resampler has every reduction the group by has plus ohlc on a series, agg and apply by name or by a mapping of columns to names, transform by name, pipe, asfreq, get_group, a column by key or by attribute, and groups, indices, ngroups, ndim, ax, binner and obj. Calendar rules such as weeks and months, offset=, the end origins, zoned timestamps, functions, lists of names and upsampling are refused by name. python/tests/test_resample.py checks all of this against pandas.

Fixed: a group fill leaves NaN, not a null, in a float column

A row that groupby(...).ffill() or bfill() could not fill, bec...

v0.8.33

Built against Mojo 1.0.0 (ed45d567).

A patch release. On the pandas side, a group by gains head, tail, nth, idxmax and idxmin, agg and aggregate take functions by name and NamedAgg, the DataFrame and Series constructors read records, rows, numpy arrays, index= and columns= as pandas does, and temporal columns hand out and take in moments, spans and dates. A boolean mask now lines up on the row labels and an operator against a moment or a span keeps the column's name. On the engine side, a semi or anti join marks the keys the other side has instead of bucketing its rows, a small text filter or gather writes its column directly, and the regex kernels cut a column finer and walk a greedy repeat in a loop, which takes ClickBench q28 from about 520 ms to about 390 ms on 1M rows.

Added: head, tail, nth, idxmax and idxmin on a group by

A group by now keeps the first or last n rows of every group with head and tail, a negative n dropping that many from the other end, and picks rows by their place in the group with nth[...] or nth(n), one place or a list of them, counting from the end when negative. All three answer rows of the frame in the frame's order under its own labels, and a row whose key is missing is in no group, as in pandas. idxmax and idxmin answer the label of each group's largest or smallest value, the first one on a tie, for one column or for every column that is not a key, in the key order sort asks for, and a group with only missing values raises pandas' error. nth with dropna and idxmax with as_index=False are refused by name. python/tests/test_group_filters.py checks 57 cases against pandas.

Fixed: a boolean series used as a mask lines up on the row labels

df.loc[mask] read a boolean series by position, so a mask whose labels were in another order picked the wrong rows. It is now lined up on the labels the way pandas does, and a mask missing one of the frame's labels raises firepanda.errors.IndexingError, which is new and carries pandas' name and message. df[mask] with a boolean series, list or array, and df[function], now work as they do in pandas, and a list of the wrong length raises pandas' message.

Added: groupby(...).agg and aggregate by name, and NamedAgg

A group by now takes agg in the four shapes pandas reads when every function is a name: one name answers what the method of that name answers, a list of names over one column answers a column a name, a mapping of column to name answers a column a column, and keywords of name=(column, function) or name=NamedAgg(column, function) answer a column a keyword. Each column is that method run on its own, so the numbers and types are the method's, and the keys come back as labels or as columns the way as_index says. firepanda.NamedAgg has pandas 3's fields and printed form, and a pandas one is read too. A mapping handed to one column's agg raises firepanda.errors.SpecificationError, which is new and carries pandas' name. A Python function, and a list over a frame, which pandas answers with two levels of column labels, are refused by name. python/tests/test_group_agg.py checks 47 cases against pandas.

Fixed: an operator against a moment or a span keeps the column's name

s - s.min(), s > datetime(2024, 1, 1) and the other operators with a Timestamp, datetime, Timedelta or timedelta on one side answered a series with no name, because the scalar was lined up as an unnamed column first. It is now lined up under the column's own name, so the answer keeps it the way pandas does and the way an operator against a number already did.

Fixed: a temporal column hands out moments, spans and dates, and takes them in

tolist, iterating a series or an index, items, itertuples, reading one cell with iloc, loc, iat or at, and the reductions that answer a value of the column's own kind now hand out a Timestamp for a moment, with its unit and zone, a Timedelta for a span and a datetime.date for a day, where they used to hand out the stored count. A missing value stays None. The other way round, a list of datetime or Timestamp values given to Series or DataFrame is a column of moments at the finest unit any value is quoted at, with a zone every value shares kept, and a list of timedelta values is a column of spans, which is what pandas infers. A datetime or timedelta on the other side of an operator is lined up as a column, so s - s.min() and s > datetime(2024, 1, 1) answer what pandas answers. This closes #348. python/tests/test_temporal_values.py checks 48 cases against pandas.

Added: DataFrame(...) and Series(...) read every shape of data pandas does

DataFrame now takes a list of records, a list of rows with columns=, a two dimensional numpy array with columns= and another frame, and index= and columns= beside any of them. A single value in a mapping is repeated down the rows, series in a mapping are lined up on the union of their labels as pandas lines them up, and a category or an instant series keeps its type. Series takes index=, one value repeated along it, and a series with an index, which pandas reads as a reindex. A numpy array keeps its own type, int32, uint8 and datetime64[s] included, and numpy scalars in a list are read as the values they hold. A shape that would need a column named 0, a column of objects or a numpy array of spans is refused by name, and the wrong lengths raise pandas' own messages. python/tests/test_construct_shapes.py checks 56 cases against pandas.

Changed: a semi or anti join marks the keys the other side has

A semi, anti or mark join only asks whether the other side has a key, never which of its rows hold it, but the table it probed was the one an inner join uses, built by counting, prefix summing and scattering every row of that side into buckets. It now makes one pass that marks each key the side has, on every core once the side is tall enough, and the streaming join does the same for these three kinds. At SF1 on a laptop, TPC-H q4, which keeps orders that have a late lineitem, goes from about 28 ms to 8.5 ms, and q22's anti join against 1.5 million orders goes from 6.5 ms to 1.3 ms, which takes the query from about 21 ms to 14.5 ms.

Changed: a small filter or gather of text writes its column directly

A filter or gather of a text column below the size where it splits across cores went through StringBuilder, which grows two lists a row and then copies both into the finished column. It now runs the same count and copy the split route does, as one worker on the calling thread. Filtering 42,000 rows of TPC-H customers takes 175 us rather than 485 us for the two byte country code and 715 us rather than 1,180 us for the address, and q22 at SF1 goes from about 26 ms to 24 ms.

Changed: a regex over a column cuts it finer and walks a greedy repeat in a loop

A regex kernel over a column (match, count, extract and replace) used to hand the cores one piece per 128K rows, which is one piece per streaming chunk, so a chunk was walked on one core. It now cuts a column into about 256 pieces of at least 4096 rows each. The backtracker walks a greedy repeat of one character, such as [^/]+ or .*, in a loop that pushes only the arms leaving the repeat, where it used to push and pop the arm going round once per character, and it keeps its stack height in a counter rather than shortening its lists. A capture rewrite of 4000 URLs with ^https?://(?:www\.)?([^/]+)/.*$ goes from about 1000 ns a row to about 700.

v0.8.32

Built against Mojo 1.0.0 (ed45d567).

A patch release. On the pandas side, concat stacks frames and series down the rows and side by side, DataFrame.join and merge join on the row labels, merge(indicator=) says which side each row came from, and DataFrame.assign, attribute access to a column, rank, round, the grouped transforms and grouped diff and transform by name are new. Series built from a mapping now holds its values under its keys. On the engine side, a join probed by many more rows than it builds indexes by value, a hashed join table that most probes miss keeps a bit per key value in front of it, a join in a pipeline hands its probe side on as positions, a sum of an integer column and a constant builds no column, which takes ClickBench q29 from 146 ms to 37 ms, and a streaming group on one integer key writes its map on every core.

Added: DataFrame.assign and a column read as an attribute

df.assign(**columns) adds or replaces columns and hands back a new frame. The keywords are taken in order and each one sees the frame the earlier ones made, so assign(c=lambda f: f.a * 2, d=lambda f: f.c + 1) works as it does in pandas. A value can be a function of the frame, a series lined up on the row labels, a frame of one column, a mapping, a list, a tuple, a range or a numpy array as long as the frame, or one value for every row. A replaced column keeps its place and a new one goes on the end. A wrong length, a frame of several columns and a set raise pandas' errors, and None is refused, because pandas makes it an object column. df.name now reads a column as an attribute when no method has that name, and dir(df) lists the columns that are identifiers.

Added: rank on a series, a frame and a group by

Series.rank, DataFrame.rank and groupby(...).rank() are written, with every method pandas has (average, min, max, first and dense), the three placements for a missing value, both directions and pct. All three are one kernel in the core, a stable sort and a pass over the ties that reads equality off the factorize codes, so numbers, text, flags and times all rank the same way. An ordered category ranks by its place in the categories and an unordered one by its value, and a group by refuses the unordered one, all as pandas does. The answer is float64, as pandas' is, a row whose group key is missing ranks NaN under dropna, and transform("rank") now answers rather than being refused. A rank across a row, axis=1 on a frame, is still refused.

Fixed: Series built from a mapping

Series({"a": 1, "b": 2}) used to read the mapping as a sequence and hold its keys as the values. It now holds the values under the keys as row labels, in the mapping's order, as pandas does, and index= beside a mapping picks those keys out in its own order with a missing value for a key the mapping does not have. name= and dtype= work with it as they do with a list.

Added: round on Series and DataFrame

Series.round(decimals) and DataFrame.round(decimals) round every number the way numpy does, which is to scale by a power of ten, round half to even and scale back, all in the column's own floating point type, so 2.5 rounds to 2.0, 2.675 to two places is 2.67 and a float32 column stays float32. An integer column is unchanged unless decimals is negative, and then it rounds to the nearest ten, hundred and so on and keeps its width. A column of text, bools, categories or times comes back as it was, as in pandas. The frame's method also takes a mapping or a series of places by column name and leaves the columns it does not name alone. Python's own round works on both, numpy's out is accepted empty and refused otherwise, and every mistake raises pandas' class with pandas' message.

Added: grouped diff and transform by name

diff on DataFrameGroupBy and SeriesGroupBy answers each row less the row that many before it in its group, taking whole numbers to float32 for the two narrow signed widths and to float64 otherwise, as pandas does. transform takes the name of a reduction and puts each group's value on every one of its rows, through a new core group_broadcast that reduces the groups once and gathers by ordinal, and it casts the answer back to float32 or to the column's own integer width where pandas does. It also takes the name of a scan. A Python function, the names pandas takes that are not written yet, and a bool column's diff, which pandas answers with an object column, are refused by name, and an unknown name gets pandas' ValueError.

Added: grouped transforms

cumsum, cumprod, cummax, cummin, shift, cumcount and ngroup on DataFrameGroupBy and SeriesGroupBy answer one value a row, in the frame's order and with its labels, as pandas does. The folds run down the rows once with a carry per group and nothing is sorted, shift places each group's rows with one counting pass, and ngroup numbers the groups in key order or in order of appearance. A row whose key is missing belongs to no group and answers missing, so an integer answer with such a row in it is float64. Counting from the end, a fill value, a frequency, a suffix and a list of periods are refused by name.

Added: merge(indicator=)

merge(indicator=) answers pandas' category column that says which side each row came from, left_only, right_only or both, with all three categories whichever occur. True names it _merge and a string names it that, on key columns and on row labels alike, and pandas' ValueError sentences are given for a name that is taken or an indicator that is neither a flag nor a string.

Added: DataFrame.join and merge on the row labels

merge(left_index=True, right_index=True), merge(left_on=..., right_index=True) and the mirror of it now work, and so does DataFrame.join, with or without on. A side keyed on its row labels has them put in a column, and the merge on columns does the rest, so suffixes, key types, a missing key matching a missing key and the outer sort all behave as they already did. With both sides on their labels the answer is labelled by the key, under the right side's label name in a right join and the left side's otherwise. With one side on its labels the answer keeps the other side's row labels, and a row that side does not have loses its label, so an integer label becomes float64 with NaN and the name goes, as in pandas. join takes a named series or a list of one frame, and a cross join, a longer list, and text labels that go missing (pandas' object index) are refused by name.

Added: concat down the rows and side by side

firepanda.concat stacks frames and series the way pandas.concat does, through two new core bindings, DataFrame.stack_rows and DataFrame.stack_columns. Down the rows the columns are the union of the parts' columns in the order they first appear, or the shared ones with join="inner", and a part without a column is missing on its rows. Each column takes the type pandas gives it, which is numpy's rule for mixed numbers, float64 for an integer column with a gap and text for category columns whose categories differ. The row labels come along with their shared name, or are numbered again with ignore_index. Series on series give a series, and a named series next to a frame is a column. With axis=1 the parts are lined up on their row labels, the union of them or with join="inner" the shared ones, and sort sorts them. verify_integrity and the mistakes raise pandas' class and message. keys, levels, names, a dict of parts, and anything pandas answers with an object column are refused by name.

Changed: a join in a pipeline hands its probe side on as positions

The streaming Join node now takes a chunk under a selection and gives one back. It gathers only its key columns, and the probe side of its output is a list of positions into the chunk that arrived, composed with the selection that chunk was under, while the build side is gathered as before. Before, a filtered chunk was flattened on its way in, which copied every column, and then each paired row was copied again on the way out. Now a filter under a join, a join and a filter over it copy each column once, when something finally reads it, and a column nothing reads is never copied. A mark join passes its chunk's columns and selection through untouched. On a filter keeping half of ten million rows, a join against a hundred thousand keys and a sum of two columns, the pipeline goes from 46 to 49 ms down to 42 ms. With no filter and five columns summed, it goes from 62 to 66 ms down to 50 to 54 ms, so the copy that moved to the end is also cheaper than the one it replaced (#79).

Changed: a join probed by many more rows than it builds indexes by value

A join whose build side is small but whose integer keys spread over a wide range used to hash every probe row, because the table indexed by the key value is sized by the build side alone. build_side now also takes the probe height, and a whole frame join accepts a value indexed table up to half that height and at most four million slots. TPC-H q17 is the shape: two hundred parts out of two hundred thousand, probed by six million lines. At SF1, over three interleaved rounds, q17 went from 25.3 to 27.0 ms to 9.9 to 12.2 ms, q8 from 31.3 to 33.6 ms to 16.4 to 18.3 ms, q9 from 93.1 to 102.7 ms to 76.7 to 77.6 ms and q22 from 32.0 to 34.7 ms to 28.3 to 29.1 ms, and the other queries did not move outside their spread. A streaming join does not know its probe height and keeps the old rule.

Changed: a streaming group on one integer key writes its map on the cores

A GROUP BY over one integer key whose values spread wider than the direct table used to insert every chunk into one hash table on one thread, the join's build called a thousand rows at a time. On ClickBench q15, which groups on UserID, that thread was most of ...

v0.8.31

Built against Mojo 1.0.0 (ed45d567).

A patch release. read_parquet now holds a string column that repeats as int32 codes into its distinct values by default, with dtype still string, so TPC-H sf1 lineitem reads into 813 MB instead of 1150 in the same time, which is less than DuckDB's 1138 MB for the same table. To get there, sorts, Arrow export, the IPC writer, every per row Series.chars_* method and the rest of the string kernels learned the encoding: the text methods run once per distinct value, and concat, cast, the fills, pick, the text reductions, joins, grouped aggregation and the per cell readers decode first. encode_strings=False on Session.run gives the old flat columns. A group by on a text column now inserts its new keys on every core. On the pandas side, pd.merge and DataFrame.merge answer a merge on key columns the way pandas does. SQL reads a NEAREST join and a slice whose end is written as a minus.

Changed: read_parquet holds a string column that repeats as codes by default

Session.run(encode_strings=...) now defaults to on, so read_parquet and everything built on it hand back a string column where each value repeats at least sixteen times on average as int32 codes into its distinct values (#979). The dtype still says string, and every frame and Series method reads the encoding or decodes first, so nothing a caller does changes except memory and speed. On TPC-H sf1 lineitem the frame is 813 MB rather than 1150 MB, and the read takes the same time. encode_strings=False gives every column flat, for a caller that reads the buffers directly.

Added: merge and DataFrame.merge on key columns

The core pairs the rows through a new join_on binding and the Python layer does the rest the way pandas does it. on, left_on and right_on work, as does a merge on the shared columns when no key is named. A suffix goes on both sides of an overlapping column, an int key against a float key becomes a float64 key, and a missing key matches a missing key. Inner and left joins keep the left order, a right join keeps the right order, and an outer join is sorted on the key, all as pandas 3 does. sort, suffixes, validate and a named series on either side work, and the mistakes raise the pandas class with the pandas message, with MergeError added to firepanda.errors. A cross or asof join, left_index, right_index and indicator are refused by name for now.

Added

• A NEAREST join, such as JOIN products t APPROX NEAREST 2 BY SIMILARITY f(q.e, t.e), now parses and prints back, with or without an alias on the right side and with the join type, APPROX or EXACT and the count all optional. The plan still refuses it as a kind of join firepanda does not run.

• A slice whose end is written as a minus, such as a[1:-:2] or a[:-:-1], which is how DuckDB says the end of the list when a step follows. It now parses and prints back as written, and the plan refuses it the way it refuses any other step. a[1:-] with no second colon is a syntax error, as it is in DuckDB.

Added: the text methods and the rest of the string kernels read a column held as codes

Every Series.chars_* method that answers a row from its value alone (case, strip, pad, slice, find, replace, the regex and folded matches, the is_* tests, translate and the rest) now runs once per distinct string of a column held as codes (#979), and the answer is spread over the rows with through_codes. On six million rows of seven values that is seven calls rather than six million. through_codes takes a string answer too, and hands it back flat, because two categories can map to one answer ("a" and "A" upper cased) and codes into a list with a repeat would break a group by on the codes. The methods that are not a row at a time (chars_extract_regex, chars_partition, chars_dummies, chars_join, to_datetime) decode first.

Concat, cast, coalesce, forward and backward fill, pick, a text reduction (min, max, any, all), a grouped aggregation of an encoded value column, a join on an encoded key, a column against column comparison and the lasting hash now decode such a column rather than raising. A reduction does not work over the categories, because a filter keeps them all and the smallest may be a value no row still holds. distinct_count_any counts the codes the rows still use and never compares a string. AnyArray.text_at(i) reads one row however it is held, and the CSV writer, the display, the index label compare, python_list and the Parquet DESCRIBE read use it.

Added: a sort and an Arrow export read a string column held as codes

A sort on a column held as codes (#979) sorts its few distinct strings once and then radix sorts the rows on each one's rank, an int32, rather than comparing strings row by row. argsort_any, argsort_multi, is_sorted_any, DataFrame.argsort, sort_values and the bounded argsort_limit all go through it, and the order is the one the decoded column gives, ties and nulls included. export_array, export_array_borrowed and the IPC writers hand such a column out as the strings it stands for, since a consumer handed an Arrow dictionary would call it a category. DataFrame.is_flat() and DataFrame.decoded() are new, for a writer that reads buffers as they lie.

Changed: grouping by a text column inserts the new keys on every core

LastingText, the table that gives each distinct string of a text key its group number across the chunks of a scan, did all its inserts on one thread, and on ClickBench q33 that serial insert was most of the main thread's time. It now splits the table into 32 parts by the top bits of the hash. Each chunk hashes its rows in parallel, sorts the row numbers into their parts, and walks the parts on all cores, so a row either finds its string or claims a slot. A chunk that a filter left small takes one worker per 8K rows rather than one per part. Rows that claim one are then numbered in row order, so new groups still get consecutive numbers in the order their first row appears and nothing downstream sees a difference. The distinct strings are kept as one piece per chunk instead of being appended to a single column, and are stacked once at the end. On the URL column of the first million ClickBench rows, split into 128K row chunks, the best of 30 runs goes from 73.6 ms to 58.6 ms. On the planner route over the million rows, q33 goes from 109 ms to 74 ms at best and q18 from 127 ms to 96 ms, and q13, q14, q16, q17 and q39 do not get slower. All 43 queries give the answers they gave before (#997).

v0.8.30

Built against Mojo 1.0.0 (ed45d567).

A patch release. The Parquet read can now keep a string column that repeats as int32 codes into its distinct values, behind Session.run(encode_strings=True), and it codes each group of rows as it arrives, so on TPC-H sf1 lineitem the frame is 813 MB instead of 1150 and the peak of the read is 2.39 GB instead of 2.80. The dtype still says string. The four str_ tests on a series read such a column once per distinct value, and with those in, all 22 TPC-H queries in firepanda-bench give the same answers encoded as flat. The option stays off by default until the kernels outside the bench learn the encoding too. On the pandas side, a series answers value_counts, mode, idxmax, idxmin, argmax, argmin and between, and &, | and ^ work on boolean series and frames. SQL reads ANY and ALL over a written list, JOIN BY (TYPE t), and keeps a FILTER it cannot rewrite instead of refusing it while parsing. The 0.8.29 shortcut that answered a sum of a column and a constant from one sum and one count is taken out again.

Changed: an encoding Parquet read turns each group into codes as it arrives

With encode_strings on, each string column now goes through a RepeatEncoder (in kernel/dictionary.mojo) one group of rows at a time. Before, it was encoded after the whole read had been stacked. The first group decides, on a 64K row sample and then the whole group, and every later group is coded against one growing list of distinct values, so a code handed out early still means the same value at the end. If a column breaks the sixteen rows a value rule part way through, the groups encoded so far are decoded back and the rest of the column is read flat. The answer is the same column either way. On TPC-H sf1 lineitem the peak of the read went from 2.80 GB flat, and 2.58 GB with the old encoding at the end, to 2.39 GB on a six core Linux machine, with the frame at 813 MB against 1150 as before (#979).

Added

• A FILTER on a call it cannot be rewritten into, such as list(i ORDER BY i) FILTER (WHERE i > 1), first(x) FILTER (...) or a macro like my_sum(i) FILTER (...) OVER (), now parses and prints back as it was written instead of being refused by the transformer. The clause stays on the call in the AST, behind the call's own ORDER BY, and the plan refuses it by name there. A filter on an ordinary fold is still a CASE around its argument and runs as before.

• x = ANY ([1, 2]) and x > ALL ([1, 2]), a quantified comparison over a list written out, which DuckDB unnests. ANY becomes one comparison per item joined by OR and ALL joins them with AND, so a null in the list gives the same three-valued answer DuckDB gives, and an empty list answers false for ANY and true for ALL. The form now also parses and prints back when the right side is not written out, such as a list column, and only the plan refuses it there.

• JOIN BY (TYPE t), the spelling DuckDB gives its internal join types, for the six that are ordinary joins: inner, left, right, full (or outer), semi and anti, with an optional _join on the end and in any case. Each one builds and prints as the regular join it names. The mark, single, right_semi and right_anti types are still refused by name, and a name that is no join type at all is a parser error that says so.

Added: a Parquet read can hold the string columns that repeat as codes

Session.run takes encode_strings, and with it on, each string column of the answer whose values repeat sixteen times on average comes back dictionary encoded, with dtype still string. encode_repetitive in kernel/dictionary.mojo makes the call: it hashes the first 64K rows and gives up there if they do not repeat enough, so a comment column costs one hash of a sample, and otherwise hashes the whole column and keeps the distinct values in the order they first appear. A column of only nulls is left flat. On TPC-H sf1 lineitem, l_returnflag, l_linestatus, l_shipinstruct and l_shipmode go from 96, 96, 146 and 96 MB to 24 MB each and l_comment stays flat, which takes the frame from 1150 MB to 813 and the peak of the read from 2.80 GB to 2.58 on a six core Linux machine. The option is off by default until every kernel the TPC-H queries reach can read the encoding (#979).

Added: value_counts and mode on a series

value_counts is a group by on the column and a count, run in the kernel, and then a stable sort on the count, so a tie keeps the order the values first appear in, which is the order pandas' hash table gives. normalize, sort, ascending and dropna all work, the answer is named count or proportion and its index is named for the column. mode answers every value tied for the most rows, sorted, with a missing value last when dropna=False counts it, and keeps the column's type. bins and a category column are refused for now.

Added: idxmax, idxmin, argmax, argmin and between on a series

The four position methods find the extreme with the reduction and the first row equal to it with a filter, so both passes run in the kernel and a tie answers the first row, as numpy's does. idxmax and idxmin answer the label and argmax and argmin the position. A gap under skipna=False, a column of nothing but gaps and an empty column each raise pandas' ValueError in pandas' words, checked in pandas' order. between is two comparisons and an and, the way pandas writes it, with inclusive taking both, neither, left and right, a missing value answering False and the name kept.

Added: &, | and ^ on a boolean series and frame

Series and DataFrame take the three logical operators and their reflected forms, between two boolean operands or a boolean operand and True or False, so a mask built from two comparisons, (s > 1) & (s < 4), now works. Two labelled operands are lined up first and a gap follows pandas' rule, which is not symmetric: a row the right side is missing reads it as False and a row the left side is missing answers False, so a | b and b | a differ there, and a column only one of two frames has is all NaN. A frame against a series lines the series up with the columns. The kernel answers each one with the arithmetic that means the same thing on two bools, *, + and !=. A constant that is not a bool is the TypeError pandas raises, and two integer columns, which pandas answers bitwise, are refused for now.

Removed

• The shortcut from 0.8.29 that answered a whole input SUM(x + c), SUM(x - c), SUM(c - x) or SUM(x * c) from the sum and the count of x. The lowering of an aggregation already says not to do this: ClickBench q29 exists to measure whether an engine fuses an expression into a reduction, and answering it with algebra reports a number the engine did not earn. The reduction builds and sums every column again, one at a time, as it did in 0.8.28.

Changed: the text tests on a series read an encoded column once per distinct value

str_contains, str_contains_in_order, str_starts_with and str_ends_with on a series whose column is dictionary encoded run over the distinct values and spread the answer over the rows by code, so a test on six million rows of seven values is seven tests and a gather. str_slice decodes first, because its answer is text and keeping it encoded would mean deduplicating the cut values again. These are the text calls the TPC-H bench makes on columns the Parquet read can now encode (#979).

v0.8.29

Built against Mojo 1.0.0 (ed45d567).

A patch release. SQL reads GROUPING SETS, CUBE, ROLLUP and GROUPING(), and a MAP literal parses. A string column can now be held as codes into its distinct values, and take, filter, a comparison, is_in and a group by read it without decoding it, while the scan decodes one morsel at a time for everything else. Two ClickBench shapes got faster through the planner: a group by on several keys with text among them keeps a map across chunks, which took q18 from 80 to 48 ms and q39 from 112 to 67 ms at 1M, and a sum of a column and a constant is answered from one sum and one count, which took q29 from 4.7 times the hand written port to faster than it. On the pandas side, a power across two labelled operands and the whole frame sum and prod answer what pandas answers where there is a gap or a NaN, and a frame reduction takes skipna=False and min_count.

Changed: take, filter, compare, is_in and group by read an encoded string column without decoding it

take_any, gather_any and filter_any move the codes of a dictionary encoded string column and hand back a column that is still encoded, so moving its rows costs four bytes a row instead of sixteen and the string bytes are never touched. A comparison with a constant and is_in work out the answer once per distinct value and spread it over the rows by code, factorize_any groups on the codes, and DataFrame.group_by does both and decodes only the keys it returns, one row a group. On six million rows over seven distinct values on a six core Linux machine the column is 24 MB against 96 MB flat, a filter took 1.0 to 1.3 ms against 12, is_in 7 ms against 25 and a group by with a sum 24 ms against 65, while an equality with a constant was within noise of flat at 7 ms. This is the third box of #979.

Changed: a query decodes an encoded column one morsel at a time

The pipeline's scan now decodes a column held in any encoding but flat as it hands each morsel out, instead of every operator refusing it. The flat copy only ever exists a morsel at a time, so a query over a frame of dictionary encoded strings pays for a few megabytes of views per morsel and not for the whole column. An operator taught to read the codes will skip this, one operator at a time (#979).

Added

• A MAP literal such as MAP {'a': 1} now reads and prints back instead of being refused while the query is read. Its keys and values are expressions like any other. The plan still refuses it by name, because no firepanda column holds a map yet.

• GROUPING SETS, CUBE and ROLLUP in SQL, alone or beside plain keys, and the GROUPING() function (also spelled grouping_id) that says which keys a row left out. Each grouping set is its own aggregate over a copy of the input, the keys a set leaves out come back as null, and the sets are stacked with a union that keeps every row, so a set written twice gives its rows twice the way DuckDB does. GROUPING(a, b) is a whole number per set with the first argument as the highest bit, and on a plain GROUP BY it is 0. A GROUPING with no groups, or over a column that is not a key, fails with DuckDB's binder message.

Added: a string column can be held as codes into its distinct values

AnyArray.dictionary_encoded(codes, categories) builds a string column whose rows are int32 codes into a list of distinct strings, with Encoding.DICTIONARY on it and dtype still string. That last part is what separates it from the category type: nothing the user can see changes. decoded() turns it back into a flat column by copying one sixteen byte view per row and sharing the categories' payload, so no string bytes are copied. slice and window keep it encoded and nbytes counts the codes plus the categories. Ten million rows over seventeen strings of about thirty bytes weigh 41 MB this way against 485 MB flat, and decoding them took 41 ms on a six core Linux machine. Nothing builds one yet outside the tests; the Arrow and Parquet readers are the next step (#979).

Added: every read of a column's values asks whether the column is flat first

AnyArray.require_flat() raises unless the column is held flat, and AnyArray.decoded() returns a flat column whatever it is held as. check_dtype, strings and into_strings call the first, so as_typed, as_typed_view and into_typed do too, and so does dispatch, whose kernels read through unsafe_ptr and check nothing themselves. The two Arrow exporters read buffers straight out of the column and ask the same question before they do. This is the second box of #979. When a dictionary encoded string column lands, a kernel nobody taught about it raises and names decoded() instead of reading codes as though they were values, and it does not have to remember to ask.

Every column is flat today, so none of these can raise yet, and decoded() on a flat column shares its buffers rather than copying them. The check is one compare a column per kernel call, never one a row.

Changed

• A whole input sum of a signed integer column and a constant, SUM(x + c), SUM(x - c), SUM(c - x) or SUM(x * c), is answered from the sum and the count of x, which are read once a chunk however many sums ask, rather than by building a column for every sum. A count under any of those is the count of x. The operation runs in its own type and can wrap there, so in a type narrower than 64 bits the chunk's smallest and largest values are checked against that type first and the column is built as before when they do not fit, which keeps the answer the same to the bit. ClickBench q29, ninety sums of ResolutionWidth plus ninety constants, now runs faster through the planner than the hand written port that does the ninety additions, where it was 4.7 times slower.
• A streaming group by on two keys or more, when one of them is text, keeps its groups in a map that lasts across chunks rather than stacking the running table with every chunk's table and grouping the two again. Each row's tuple is written out as bytes, a presence byte per key and a length in front of text, on the cores, and looked up in the text map the single key route already used, so a null in a key no longer matters to the route either. ClickBench q18 at 1M went from 80.2 to 48.2 ms and q39 from 112.2 to 67.4 ms through the SQL planner, and the planner's total over the 43 queries against the hand written port went from 1.42 to 1.28, with all 43 still agreeing. A table that arrives in one chunk and a tuple of integers alone stay on the old route, because the map inserts its groups one at a time and loses to one parallel pass there. A float key does too.

Fixed

• A power between two labelled operands answers what pandas answers in a row only one side has, which is 1 where numpy's 1 ** nan or nan ** 0 is 1 and NaN everywhere else. The kernel answers a gap with a null before it looks at either value, so the pandas facing layer lines the two sides up by label with NaN in each gap and takes the power again. That covers a series with a series and a frame with a frame, through **, its reflection and pow and rpow without a fill_value. A side with a repeated label cannot be lined up that way and keeps the NaN it had. (Issue #8)

• sum and prod over a whole frame keep a NaN one column's own arithmetic made, such as inf plus -inf, where they skipped it and answered the total of the other columns. The whole frame form is the per column pass run again over its answers, and the second pass read that NaN as a missing value. A missing cell is already skipped by the first pass, so for a total or a product the second pass no longer skips anything. min and max still skip there, because their NaN is a column with nothing in it. (Issue #8)

• A frame reduction takes skipna=False and min_count, which it refused by name. Each is a rule about the answer, the same as on a series, so the kernel reduces every column as before and a column the rule catches is answered NaN after: a column with a missing value under skipna=False, or one with fewer values than min_count. The answer widens to float64 only when a column is caught, which is what pandas does, and with axis=None the floor counts every cell in the frame rather than each column. A caught column whose answer is not a number is refused, because pandas answers NaT or an object NaN there. (Issue #8)

Note added after the release: the q29 change (#1041) is taken out again by #1044, because the planner's own rule is that q29 must fuse the expression into the reduction rather than answer it with algebra. It will be gone in the next release.