v0.8.37

Built against Mojo 1.0.0 (ed45d567).

A patch release. On the pandas side it brings the readers and writers pandas has beyond CSV and Parquet (Excel, Stata, SAS, HDF, SQL over sqlite3, HTML, XML, LaTeX, the clipboard, Iceberg and SPSS), periods and interval columns, categoricals, object, sparse and masked columns, rows and columns labelled by a MultiIndex, DataFrame.style with to_excel, plotting with matplotlib, firepanda.testing, and pandas' byte counts, str dtype and float answers for var, std, sem, skew and kurt. SQL gains LATERAL subqueries, ASOF joins, PIVOT and UNPIVOT, recursive CTEs, named windows, window frames and CREATE, INSERT and DROP. The group by skips text keys another key already decides, packs two wide integer keys without factorizing either and uses every core on half a million rows or more, and the ClickBench driver builds again after a compiler hang that came in with the first of those.

Changed: Byte counts are pandas' byte counts

Series.nbytes, Index.nbytes and memory_usage on the frame, the column and the index now answer the number pandas answers, counted from the numpy or Arrow arrays pandas would hold the data in: a width a row for numbers, a mask byte on top for a masked type, offsets and UTF-8 bytes and a bitmap for text, codes and categories for a category, a pointer a row for objects with deep adding each object, and the Python range and its three numbers for a RangeIndex. info spells text str and objects object, and writes pandas' + when object memory was left out. A column or frame built from columns that carry the plain range from zero now keeps a RangeIndex rather than turning it into stored integers, which also fixes their repr.

Changed: a series' var, std and sem are pandas' floats

Series.var, Series.std and Series.sem of a column of numbers take the squared distances from the mean numpy sums and leave its error alone, the steps pandas' nanvar takes, where the kernel corrected for that error. On five values near two to the fifty two that is pandas' 37.25 where exact arithmetic gives 37.2, and skew agrees with pandas' twelve percent on the same column. Without numpy the kernel still answers.

Changed: skew and kurt of whole numbers are pandas' answers

Series.skew and Series.kurt cast a column of whole numbers to float64 before taking its moments and add in numpy's order, step by step as pandas' nanskew and nankurt do, so near two to the sixty two they answer pandas' float rather than the exact value a few parts in ten million away. A float column's skew now goes the same way, which was already how its kurtosis went.

Changed: a text column's dtype is pandas' str

Series.dtype, Series.dtypes, Index.dtype and DataFrame.dtypes answer pandas' StringDtype with NaN for a gap for a column of text, which prints as str the way pandas 3 prints it and still equals string, so code comparing against either word keeps working. Everything below the surface reads a dtype back through one helper that gives the core's word, so no internal decision changed. A column made with dtype="string" is also spelt str, because firepanda does not yet keep the NA variant apart.

Added: a map's key and value types, and histogram's MAP(K, UBIGINT)

A MAP type now carries its key and value types, written MAP(K, V) as DuckDB writes them, and two maps are the same type only when both agree. histogram answers MAP(K, UBIGINT) for a key of any flat type, including the bins form, as DuckDB does, so the differential semantics check now compares its full type rather than the overload alone. A map whose key or value is itself nested is still a bare MAP, and says nothing about what it holds.

Changed: Rolling.max takes pandas' extra arguments

Rolling.max is the one window reduction pandas declares with *args before its engine arguments and **kwargs after them, and its body never reads either. firepanda declared the same three parameters as the other reductions and refused anything else, which was a registered divergence. The signature now matches pandas, and the extra arguments are dropped as pandas drops them. Expanding.max declares neither in pandas and still refuses them here.

Added: Styler.to_excel

Styler.to_excel now writes a styled frame to a workbook as pandas does. Each cell's CSS goes through pandas' CSSToExcelConverter, ported without numpy along with its CSS reader, and the openpyxl, xlsxwriter and odf writers turn the result into the same fonts, fills, borders, alignments and number formats pandas writes, including across merged cells. firepanda.io.formats.excel and firepanda.io.formats.css hold the converter and the reader where pandas keeps them. A loc key on rows labelled by several levels that holds a slice or a list for fewer levels than there are, such as loc[(slice(None),)], now keeps every level as pandas does, rather than dropping the ones it named.

Added: DataFrame.style

DataFrame.style now answers pandas' Styler, ported from pandas and run over firepanda frames, when jinja2 is installed. It writes a frame as HTML, LaTeX, text or Typst with the same formats, highlights, bars, gradients, tooltips, hidden rows and columns, sticky headers, table styles and concatenated footers as pandas, and the same text for each, down to the element ids. Without jinja2 the accessor raises pandas' own AttributeError. The gradients and colormapped bars need matplotlib, as they do in pandas. firepanda.io.formats.style.Styler is importable where pandas keeps it. Styler.to_excel is not ported yet. A column of a frame that was transposed twice now hands its labels back from items as the values they are, not as the text the extension holds them as.

Added: sequence answers in apply, level keys in loc and numbers compared with text

DataFrame.apply now frames a list, a tuple or an array answered for each column the way pandas does, labelled by the rows when the answers are as long as the frame. loc reads a row key with a slice or a list for some level, such as IndexSlice["A", :], level by level through MultiIndex.get_locs, which now answers rows in the order its lists name them, and loc[:, "a"] on rows with levels keeps every level. A column of numbers or flags compared with text is False for == and True for !=, and an order between them raises pandas' TypeError.

Added: DuckDB's .test files run against firepanda, with a floor

pixi run conformance runs DuckDB's own .test files from the vendored corpus through firepanda and checks each against what the file says DuckDB answers: every statement's success or error and every query's rows, whether sorted, hashed or labelled. The files are run unmodified. A file that needs an extension, a database file or the corpus's data directory is counted as not run, and every failing file is put in the bucket of its first failure: a refusal by name, a missing function, a divergence, a wrong answer or a crash. The per directory table over the target directories is in the README with its denominators, and tests/conformance_floor.txt holds the count that passes in each, which CI checks so a change that lowers it fails. --all runs every directory and writes the failures to build/conformance/failures.txt.

Added: HDFStore, read_hdf and to_hdf

HDFStore, read_hdf, DataFrame.to_hdf and Series.to_hdf read and write HDF5 files in pandas' fixed and table formats, and firepanda.io.pytables holds the storers where pandas keeps them. The module is pandas' own pandas.io.pytables ported class for class on top of PyTables, which is imported only when a store opens. Columns are grouped into blocks and consolidated by pandas' rules, so a file firepanda writes has the same nodes, attributes and types as the one pandas writes and each library reads the other's files. Where clauses, data columns, appends, removal, chunks, coordinates, several tables at once, copies, table indexes, the printed store and the errors all match pandas.

Added: read_sas and the SAS readers

read_sas reads SAS transport files and SAS7BDAT data sets the way pandas does, and firepanda.io.sas holds SASReader, XportReader and SAS7BDATReader where pandas keeps them. Both readers are ported to plain Python with struct, including the IBM float conversion of transport files and the run length and Ross decompressors of data sets, so they need no other package. Names, types, values, dates, chunks, reader attributes and errors match pandas on pandas' own SAS test files, with and without an encoding.

Added: DataFrame.to_stata and the Stata writers

DataFrame.to_stata, StataWriter, StataWriter117, StataWriterUTF8, StataStrLWriter, StataValueLabel and StataNonCatValueLabel now write Stata dta files in versions 114, 117, 118 and 119. The writer is a pure struct port of pandas' own, so with the same time stamp the file is byte for byte the one pandas writes, including type narrowing, renamed columns, value labels, dates in every Stata format, long strings, byte order and compression, and it raises pandas' warnings and errors in pandas' words.

Added: read_stata and StataReader

read_stata reads Stata dta files of versions 102 to 119, whole or in chunks through StataReader, which also lives at firepanda.io.stata with StataMissingValue. It is a port of pandas' reader that uses struct in place of numpy, and it follows pandas for missing values, the nine date formats, long strings, value labels as ordered categoricals, preserve_dtypes, columns, index_col and compressed files, with pandas' warnings and error sentences.

Added: convert_dtypes with the pyarrow backend

convert_dtypes(dtype_backend="pyarrow") now gives Arrow types as pandas does, such as int64[pyarrow], double[pyarrow], string[pyarrow] and timestamp[us][pyarrow], keeping each column's width and following the four switches, while categories and columns of objects stay as they are. A column of flags becomes whole numbers when convert_boolean is off, as it does in pandas. A column backed by Arrow hands out Timestamp and Timedelta for moments and durations. A frame built from a dict with moments next to a column of mixed objects no longer refuses the moments. read_spss(dtype_backend="pyarrow") now matches pandas through this.

Added: read_iceberg, to_iceberg and read_spss

read_iceberg and DataFrame.to_iceberg read and write Apache Iceberg tables through pyiceberg, with pandas' arguments for the catalog, the columns, the row filter, the snapshot and appending, and labels other than the default are kept as a column the way pandas keeps them. read_spss reads an SPSS file through pyreadstat with usecols, convert_categoricals and dtype_backend, and puts the file's metadata in attrs as pandas does. A missing package raises pandas' sentence.

Added: to_excel and ExcelWriter

DataFrame.to_excel, Series.to_excel and ExcelWriter write workbooks with pandas' signatures through openpyxl, xlsxwriter or odfpy, picking the engine from the extension and the io.excel.*.writer options as pandas does. A port of pandas' ExcelFormatter lays out the cells, so the header rows, the index names, merged MultiIndex labels, na_rep, inf_rep, float_format, startrow, startcol, freeze_panes and autofilter land where pandas puts them, with pandas' number formats for moments and durations. A writer holds several sheets, appends to an existing book with every if_sheet_exists choice, and fails with pandas' sentences.

Added: read_excel and ExcelFile

read_excel and ExcelFile read xlsx, xlsm, xls, xlsb and OpenDocument workbooks with pandas' signature and pandas' answers. The engine is picked from the first bytes the way pandas picks it, and openpyxl, xlrd, pyxlsb, odfpy and python-calamine each read the cells as pandas' reader for that engine does. The rows then go through a port of pandas' Python parser, so header rows, index_col, usecols in all its forms, skiprows, nrows, skipfooter, the missing value options, true_values and false_values, thousands, converters, dtype, parse_dates and dtype_backend type a column the way pandas does and fail with pandas' sentence. The engines are optional, and a missing one is named in pandas 3's words.

Added: pandas 3's sentence for a missing optional dependency

A reader or method that needs a package which is not installed now says so the way pandas 3 does, "Import xarray failed. Use pip or conda to install the xarray package.", naming the package to install, so odf is named odfpy. to_xarray, the sparse conversions to scipy, the Parquet, ORC and Feather readers and firepanda.test all word it through one helper, _optional, which the Excel readers will use too.

Added: to_xarray

DataFrame.to_xarray and Series.to_xarray answer an xarray Dataset or DataArray the way pandas does. Like pandas, they hand the object to xarray.Dataset.from_dataframe or xarray.DataArray.from_series. Those read pandas' internals, and xarray depends on pandas, so the object is first rebuilt as a pandas one with every column's type kept: categories and their order, the nullable types, text, and moments with a zone. A multi index is filled out to its full grid and a repeated label is refused, both by xarray itself. Without xarray the error is pandas' ImportError.

Added: arrays and categoricals in a frame mapping

DataFrame({"c": Categorical([...])}) now builds a category column with the categories and order it was given, where it used to build a text column, and an array from array(..., dtype="Int64"), "Float64" or "boolean" builds a column of that type with its missing values, where it used to be refused. Each array is read as a series of itself placed by position. Beside a series, it takes the labels the series make and has to be as long as they are, with pandas' error when it is not.

Added: select_dtypes reads nullable and object columns

select_dtypes now reads each column by the type dtypes shows rather than by how it is stored, so an Int64, UInt8, Float64 or boolean column is picked up by number, integer, floating and bool the way pandas picks it up. A nullable name on either side means its numpy twin, so include="Int64" takes an int64 column too. object, O and the object type select object columns and text columns, and string selects only text, which are pandas 3's rules.

Added: plotting with matplotlib

Series.plot and DataFrame.plot are pandas' PlotAccessor, with every kind pandas draws: line, bar, barh, box, hist, kde, area, pie, scatter and hexbin. Series.hist, DataFrame.hist, DataFrame.boxplot, the groupby plot, hist and boxplot, and the whole firepanda.plotting namespace are there too, including scatter_matrix, parallel_coordinates, table and plot_params. The matplotlib backend is a port of pandas' own, so a plot draws the same lines, bars, tick labels and legend, in the same colours, and a mistake raises the same error. matplotlib is imported only when a plot is drawn, and the plotting.backend option finds third party backends the way pandas does. An object column now hands back the object that was put in it, rather than a copy, for values of a class outside the standard library.

Added: sparse columns

SparseDtype, Series.sparse and DataFrame.sparse are implemented. A sparse column answers with pandas' dtype names and the fill value pandas picks for arithmetic, comparisons, where, fillna, map, astype and the rest, refuses the reductions pandas refuses with pandas' errors, and the accessors give the fill value, the stored points, the density and the dense column. DataFrame.__getattr__ now lets an accessor's own error through, as pandas does.

Added: pandas.test

firepanda.test(extra_args=None, run_doctests=False) runs the test suite with pytest the way pandas.test runs pandas', with the same default marks, the same printed command and the same exit with pytest's status. It points at the tests next to the package in a source checkout, and at the package itself otherwise.

Added: ewm(...).online()

ExponentialMovingWindow.online() returns an OnlineExponentialMovingWindow whose mean() remembers its state, so mean(update=...) carries on from the last row the way pandas does with numba, and reset() starts over. The kernel is plain Python, so no numba is needed. The other reductions, update_times and any engine but numba are refused with pandas' messages, and the window now prints the way pandas prints it.

Added: pandas.compat, core, io and errors submodules

firepanda.compat has pandas' platform checks and version flags, with the pyarrow flags worked out on first read. firepanda.core.api and firepanda.io.api name the core objects and the readers as the top level objects. firepanda.errors is now a package with cow holding pandas' chained assignment messages, and abc and ctypes showing through as they do in pandas.

Added: pandas.util

firepanda.util has hash_pandas_object and hash_array, which give the same uint64 row hashes pandas gives for every column kind, index and key, through a SipHash-2-4 port and pandas' bit mixing and row folding. firepanda.util.version reads and orders PEP 440 version strings like pandas' copy of packaging.version.

Added: pandas.tseries

firepanda.tseries is now a package with api, frequencies and offsets modules like pandas has. to_offset reads the same text as pandas and fails with the same nested words for retired aliases, bad suffixes and unknown names, get_period_alias maps offsets to period names, and guess_datetime_format is public. firepanda.pandas names the package itself.

Added: read_html

read_html reads the tables of an HTML document into frames the way pandas does, with match, attrs, displayed_only, header inference from thead and th rows, colspan and rowspan expansion, extract_links, and the read_csv options for typing the cells. It parses with the standard library, so it needs neither lxml nor BeautifulSoup.

Added: CREATE TABLE, CREATE VIEW, INSERT and DROP in SQL

firepanda.sql.execute runs one statement against a catalog it may change. CREATE TABLE with columns and types, with NOT NULL, and CREATE TABLE ... AS a query make a frame under the name; INSERT INTO adds rows from VALUES, a query or DEFAULT VALUES, by position, by a column list or BY NAME, casts each to the column's type, and answers DuckDB's one-row Count; CREATE VIEW keeps the query's text, which is parsed again wherever the view is named, so a view sees rows inserted after it; and DROP TABLE and DROP VIEW take the name away. OR REPLACE, IF NOT EXISTS and IF EXISTS behave as in DuckDB, and every refusal a mismatch earns is DuckDB's message: a name made twice, a count of values that does not fit, a column the table lacks, a null in a NOT NULL column, a drop of the wrong kind, and a view that reaches itself. What a frame cannot hold (a key, a default, a check, a generated column, ON CONFLICT, RETURNING, CASCADE) is refused by name rather than read and dropped.

Added: to_latex

DataFrame.to_latex and Series.to_latex write the LaTeX table pandas writes, as a booktabs tabular, inside a table float when there is a caption, label or position, or as a longtable. Cells follow pandas' formatting with six digits for floats, na_rep, float_format, formatters, decimal and LaTeX escaping, and the text is written without needing jinja2.

Added: pandas.array and pandas.arrays

firepanda.array is now available, as pandas.array. It infers the masked types for plain numbers and bools and text for strings, keeps the type of a Series, an Index or an array, and refuses a scalar as pandas does. Arrays now carry pandas' class names, IntegerArray, FloatingArray, BooleanArray, ArrowStringArray, NumpyExtensionArray, DatetimeArray, TimedeltaArray, PeriodArray, IntervalArray and ArrowExtensionArray, which firepanda.arrays exports with Categorical. They print as pandas prints them, and have arithmetic, comparisons, fillna, dropna, copy, astype, value_counts, to_numpy and the reductions. isna on an array now answers a numpy array of bools. Document 106 describes the design.

Added: to_period, to_timestamp and PeriodProperties

DatetimeIndex.to_period, Series.dt.to_period, and to_period and to_timestamp on Series and DataFrame labels are now available, as in pandas. With no frequency, to_period uses the index's own or an inferred one, read as its period frequency, and refuses as pandas does when there is none. s.dt on a period column answers a PeriodProperties with the period fields, start_time, end_time, freq, to_timestamp, asfreq and strftime. PeriodIndex.to_timestamp now infers the frequency of the starts. Document 105 describes the design.

Added: PeriodIndex and period_range

PeriodIndex and period_range are now available, as in pandas. The index is built from periods, their text or instants with a frequency or a period dtype, or with from_fields and from_ordinals. It answers the period fields, start_time, end_time, to_timestamp, asfreq, strftime, shift and integer arithmetic, and finds labels by a period or its text, including label slices. Series and frames labelled by periods print Freq:, and grouping by a period column, value_counts, pd.Index of periods, reindex onto periods and a category of periods all give a PeriodIndex. Document 104 describes the design.

Added: Period columns

A Series or DataFrame column of Period values of one frequency is now a period column with the dtype period[M], as in pandas. It can be asked for with dtype="period[M]" or astype, prints each period in its short form with NaT for gaps, sorts, groups and compares by period, moves by integers, and gives offsets when a period is subtracted. Scaling a period column and ordering it against another frequency are refused with pandas' messages. Document 103 describes the cell design and what waits for PeriodIndex.

Added: PeriodDtype

pd.PeriodDtype is the real type now, made from a frequency's text, period[...] text or an offset, with pandas' errors for the rest. It gives its frequency back as an offset, compares with text the way pandas does, pickles, and answers construct_from_string, is_dtype and the numpy style attributes pandas gives it. pandas_dtype and is_period_dtype in api.types read period text through it, so a frequency that does not exist is refused there too.

Added: Period

pd.Period is here, a span of time at a frequency, as pandas has it: made from text, a number, a date, a timestamp or its fields, with the same errors for frequencies it does not take. It moves between frequencies with asfreq, gives its first and last moments with to_timestamp, start_time and end_time, answers every field pandas answers, writes itself out with strftime and its period codes, and adds and subtracts whole steps, offsets and time spans the way pandas does. Timestamp.to_period now gives one too.

Added: Resampler upsampling

ffill, bfill, nearest, asfreq and interpolate on a resampler now fill the new bins the way pandas does, on the bin edges pandas picks for closed and label, with limit and fill_value. A column picked from the resampler fills within each bin on the original rows, and the methods pandas refuses there, after on=, and on a grouped resampler raise pandas' errors. interpolate(method="time") now works over dates and spans, method="index" reads dates as their counts, and Series.where with a condition on a date index no longer fails.

Added: GroupBy.resample

DataFrameGroupBy.resample and SeriesGroupBy.resample resample each group's rows on their own and stack the answers under the group's key, as pandas does. Columns can be picked with [] or by attribute, the key columns among them. group_keys=False and as_index=False stack the answers without the key, on= bins by a column, and include_groups=True is refused in pandas' words. transform is refused for now.

Added: to_clipboard and read_clipboard

DataFrame.to_clipboard, Series.to_clipboard and firepanda.read_clipboard work as pandas' do. They reach the clipboard through the same system commands pandas' copy of pyperclip finds, in the same order, and raise its error when there are none. to_clipboard writes what to_csv writes, tab separated, or the printed frame with excel=False, and warns where pandas warns. read_clipboard reads the text with read_csv, guessing tabs and row labels the way pandas guesses them.

Added: read_xml

read_xml reads the elements an xpath picks from an XML document into a frame, as pandas does, with each element's attributes, its own text and its children's text as the columns. The etree parser uses the standard library and the part of XPath it follows, and the default lxml parser, with its full XPath and stylesheets, works when lxml is installed and raises pandas' ImportError when it is not. elems_only, attrs_only, names, namespaces, dtype, converters, parse_dates, iterparse, compressed files and both dtype_backend choices work, and columns are typed the way read_csv types text.

Added: to_sql and read_sql over sqlite3

DataFrame.to_sql, Series.to_sql, read_sql, read_sql_query and read_sql_table work over a sqlite3 connection the way pandas works without SQLAlchemy. Writing gives the same CREATE TABLE text and column types, the same index on the row labels, the same if_exists, chunksize, dtype and method="multi" handling, and the same row count. Reading types each column the way pandas reads the values sqlite3 hands back, and takes params, index_col, parse_dates, dtype and chunksize. read_sql_table finds the table and then refuses as pandas does without SQLAlchemy. A URI or another kind of connection is refused, and a result column of nothing but NULLs reads as NaN gaps where pandas holds None.

Added: A LIMIT inside a LATERAL subquery

FROM t, LATERAL (SELECT x FROM s WHERE s.k = t.k ORDER BY x DESC LIMIT 1) now runs, where an ORDER BY or a LIMIT inside a lateral subquery used to be refused, and so does LEFT JOIN LATERAL over one and an OFFSET. The rows of each left row are one partition of the subquery's rows on the columns it correlates on, so they are numbered in the order the ORDER BY gives and the ones the LIMIT and OFFSET keep are joined back, which is DuckDB's answer. A limit whose correlation is not equalities alone, or that sits on a fold, is refused by name.

Added: A LATERAL subquery that folds runs

FROM t, LATERAL (SELECT sum(qty) FROM s WHERE s.k = t.k) and LEFT JOIN LATERAL (...) ON true now run when the subquery aggregates, where they used to be refused. The subquery is grouped by the columns its equalities correlate on and left-joined back, so a row with no match still gets its one row: a null sum, and a count of 0 (as DuckDB gives). An uncorrelated fold is one group joined to every row. A correlation other than equalities, or a fold that reads the outer row, is refused by name.

Added: StringDtype, DatetimeTZDtype, dt.timetz and selecting a group by's key

StringDtype and DatetimeTZDtype are pandas' dtype objects with pandas' attributes, repr, equality and mistakes, and each is equal to the text firepanda names the same type by, so either can be passed as dtype=. dt.timetz gives each instant's time of day with its clock. A group by now lets a key column be selected as a value, as in df.groupby("a")["a"].sum(), which used to fail inside the extension, reads its columns as attributes including the keys, and lists them in dir(). A key from outside the frame beside another key now keeps its name on the labels.

Added: DataFrame.unstack, Series.from_arrow and to_markdown

DataFrame.unstack moves a level of a MultiIndex into the column names, naming each column by the pair of its old name and the level's value, and on flat labels it answers a series labelled by column and then row, as pandas does. The one difference is that firepanda has no name for the column axis, so the header row pandas prints with the level's name is missing, as it already is for pivot and Series.unstack. Series.from_arrow reads an Arrow array or stream the way DataFrame.from_arrow reads a column. DataFrame.to_markdown and Series.to_markdown hand the frame to the optional tabulate package with pandas' defaults and raise pandas' ImportError when it is missing.

Added: interval_range, and cut by an IntervalIndex

interval_range builds an IntervalIndex of evenly spaced numbers from three of start, end, periods and freq, with whole number ends when every one given is whole, as pandas does. cut now takes an IntervalIndex as bins and answers an ordered category of those intervals, and it refuses intervals that overlap in pandas' words. Ranges of instants or spans still raise.

Added: interval columns, and cut and qcut label bins with intervals

A list of Interval values that share a side is now an interval column with pandas' interval[int64, right] or interval[float64, right] type, and it prints, casts to category, counts and groups the way pandas' does. IntervalDtype and IntervalIndex are new, with from_breaks, from_tuples, from_arrays, left, right, mid, length and pandas' repr, and cat.categories of a category of intervals is an IntervalIndex. cut and qcut without labels= now label each bin with an interval, rounded by pandas' precision rule, where they used to refuse. An interval column and a category of intervals or of numbers go to Arrow as pandas sends them, intervals as the pandas.interval type. Spec 102 has the details.

Added: DataFrame.from_arrow reads a date as pandas does

DataFrame.from_arrow, the pandas door, now reads a date32 column as an object column holding Python dates, which is what pandas.DataFrame.from_arrow hands back. firepanda.from_arrow still keeps Arrow's date type.

Added: a column named with empty text exports to Arrow

A frame with a column named with empty text, which get_dummies makes from an empty text value, used to crash inside pyarrow with a SystemError. The column now goes out under a spare name and takes its own name back in the batch, so pa.table and pa.record_batch read the frame as pandas' would.

Fixed: the bench driver builds again

From #1232 on, building the firepanda-bench driver never finished: the compiler stopped using CPU a couple of minutes in and waited forever. The check that skips text group keys a fixed width key already decides called itself on a coded column's codes, and that is what the compiler could not get past. It now calls a second function instead, and a coded column is still checked on its codes.

Added: pandas.wide_to_long

wide_to_long turns columns named by a stub and a suffix, such as A1970 and A1980, into one column per stub under a new index level holding the suffix, read as numbers when every suffix is one. It follows pandas' recipe over firepanda's melt, keeps pandas' row and column order for one id column and for several, and refuses a stub that names a column and ids that do not tell the rows apart in pandas' words.

Added: firepanda.testing, with assert_series_equal, assert_frame_equal and assert_index_equal

A test suite written against pandas imports pandas.testing first, and firepanda had no such module, so every such suite failed on its import line. firepanda.testing has pandas' four assertions with pandas' keywords and defaults. Each check runs in pandas' order and raises pandas' message, so a failing test points at the same place under either library: the length, the index, the dtype, the values with the share that differ and the first position that does, the gaps of a text, masked or date column before its values, the name, and for a frame the shape, the columns and then each column by its position and label.

Added: Series.list and Series.struct read off the class answer the accessor class

pandas answers the accessor class for Series.list and Series.struct, the way it does for Series.str, so a program can read the accessor's members without a column. firepanda answered a property object there, which is not callable, so both names now go through the same descriptor as str and dt, and the accessors take data=None as pandas' do.

Added: SQL PIVOT runs when every column lists its values

PIVOT t ON a IN (1, 2) USING sum(x) GROUP BY g and FROM t PIVOT (sum(x) FOR a IN (1, 2) GROUP BY g) now lower and run, where they used to be parsed and then refused. As in DuckDB, each cell is the USING aggregate over the rows holding that cell's values, one row per group, and a PIVOT with no USING counts the rows. Without a GROUP BY the groups are the columns that neither ON nor USING reads. Several ON columns make a cell for every combination of their values, and the cells are named the way DuckDB names them: the values joined with _, then the aggregate's alias, or the aggregate as printed when there is more than one. An ON column with no IN list, or with an IN over an enum or a subquery, is still refused, because its columns aren't known until the data is read.

Added: the masked types Int64, Float64 and boolean

pandas' capital letter types, Int8 to UInt64, Float32, Float64 and boolean, and their dtype classes, are taken by astype, the constructor and convert_dtypes where they were refused. A masked column keeps its type through a gap, prints a gap as <NA>, answers boolean with a gap from a comparison, follows Kleene's logic for &, | and ^, and skips gaps in reductions, as pandas does. It is held as an object column whose cells carry the type's name, and arithmetic, comparisons and reductions run over the lower case column of the same width. Document 99 of the compat notes describes the design.

Added: SQL UNPIVOT runs

UNPIVOT t ON a, b, c and FROM t UNPIVOT (v FOR n IN (a, b, c)) now lower and run, where they used to be parsed and then refused. Each row hands out one row per folded column, with the column's name in name and its value in value (or the names INTO NAME ... VALUE ... gives), and a null value drops its row, as in DuckDB. The columns not folded come along unchanged. A folded expression is named by its alias, or else by the one column it reads; an expression reading two columns is refused as DuckDB refuses it. More than one value column is refused by name.

Added: ArrowDtype, and the list and struct accessors

ArrowDtype names a column backed by Arrow, as pandas' does, and astype(ArrowDtype(t)) or Series(..., dtype=ArrowDtype(t)) makes one. The column is an object column whose cells carry the Arrow type, so it keeps the type through anything that moves rows, prints as pandas prints it with <NA> for a gap, and exports to Arrow with its own type. Series.list answers len, indexing, slicing and flatten, and Series.struct answers dtypes, field by name, position or path, and explode, each through the pyarrow compute function pandas uses.

Added: Categories of numbers and flags

astype("category"), Series(..., dtype="category") and Categorical now make a category column of whole numbers, floats or flags, whose categories keep the values' own type as pandas' do, where they refused anything but text. The categories are written as object cells, so the extension still encodes the column. The repr, CategoricalDtype, the cat methods, comparisons, isin, astype back, value_counts, grouping by the column and fillna all read them as values. unique and values on a category column now answer a Categorical, and drop_duplicates on one no longer raises.

Added: DataFrame.compare, and stack over columns with levels

DataFrame.compare is new and answers the differing values of two frames with columns named by the pair of the column and the side, or with align_axis=0 rows named by the pair of the label and the side, with keep_shape, keep_equal and result_names as in pandas. stack over a frame whose columns have levels moves the asked levels to the rows, keeping the order the labels first appear and reading a missing pair as missing, as pandas' new stacking does. droplevel, swaplevel and reorder_levels with axis=1 now work on the column levels. The repr of a MultiIndex right aligns each value to the widest at its place, as pandas prints it.

Added: Group by answers with two levels of column labels

DataFrameGroupBy.agg takes a list of reductions, and a mapping whose values are lists, and names each column by the pair of the column and the reduction, as pandas does, with lambdas numbered <lambda_0> and on and a repeated name raising SpecificationError. DataFrameGroupBy.describe and DataFrameGroupBy.ohlc are new and answer the same shape. Under as_index=False the keys come out as columns with an empty lower level. A header cut by max_columns now prints the whole label of the first column past the dots, as pandas does.

Added: Columns with more than one level

A frame whose column names are all tuples of one length of at least two now has MultiIndex columns, as in pandas. df[("a", "x")] selects one column, df["a"] selects every column under the first level a and drops that level, or answers a series when what is left of each name is empty text. The repr and to_string print one header line per level, sparsified the way pandas prints them, and to_csv writes one header row per level. Level names on the columns read as None for now.

Added: dt.to_pydatetime, str.encode, str.decode and DataFrame.to_records

dt.to_pydatetime answers an object column of Python datetimes with NaT for a gap, as pandas 3 does. str.encode answers an object column of bytes and str.decode reads bytes back as text, with anything that is not bytes as a gap. DataFrame.to_records answers a numpy record array with the row labels first, named and typed as pandas names and types them, including column_dtypes and index_dtypes.

Added: SQL ASOF RIGHT and FULL joins

ASOF RIGHT JOIN and ASOF FULL JOIN now lower, where they used to be refused. As in DuckDB, each left row still takes the nearest right row, and a right row no left row took comes out once more at the end with the left side's columns null. A full one also keeps a left row with no match, as ASOF LEFT JOIN does. With USING, a right one hands out the right side's values of the named columns, as any right join does. A full one with USING is refused by name, as every full USING join is so far.

Added: nested Arrow columns read with pandas' numbers

DataFrame.from_arrow reads a nested column through pyarrow's to_pandas, so a list of integers beside a null reads as floats with NaN and a struct field does the same, which is what pandas holds. The arrays pandas keeps in each cell are held as lists.

Added: explode, and nested Arrow columns read as object columns

Series.explode and DataFrame.explode spread each list, tuple or set over rows of its own and repeat the row labels, with ignore_index and several columns at once, and refuse an empty, repeated or mismatched list of columns as pandas does. DataFrame.from_arrow reads a table with list, map or struct columns by holding each nested column as an object column of lists and dicts, where it used to refuse the whole table.

Added: SQL LEFT JOIN LATERAL

LEFT JOIN LATERAL (...) ON true now lowers, where it used to be refused. A left row the subquery finds nothing for is kept once, with nulls where the subquery's columns would be, as in DuckDB. The subquery has to be correlated by equalities alone, its select list may not read the left side, and the ON has to be true; anything else is refused by name. A right or full join to a LATERAL is refused as before.

Added: SQL ASOF joins with USING

ASOF JOIN ... USING (a, b, t) and ASOF LEFT JOIN ... USING now lower, where they used to be refused. As in DuckDB, every named column but the last must be equal on both sides, and the last is the inequality: each left row takes the right row with the greatest value that is at most its own. The named columns come out once, as with any USING. ASOF RIGHT and ASOF FULL are still refused.

Added: Object columns cross to Arrow as their values

An object column exported through the Arrow PyCapsule interface, on its own or in a frame, now crosses as the values it holds, the way pandas hands one to pyarrow, rather than as the text firepanda writes them in. pyarrow picks the type, so a column of lists is a list column, and a mix of values with no one Arrow type is refused as it is for pandas. This needs pyarrow installed, as pandas does, and only for a frame or column that holds an object column.

Added: The string accessor on lists, and s.str[...]

s.str[i] and s.str[a:b:c] now work, as get(i) and slice(a, b, c) do in pandas. On an object column, len, get, slice, indexing and join read each value in Python the way pandas does, so a list, a tuple or a dict in a cell answers its length, its item or its joined text, and a value the method cannot take, such as a number, answers NaN. Other string methods read a list as a gap rather than refusing the column. Text read out of an object column comes back as an object column, and numbers with a gap come back as floats, as in pandas.

Added: Operations that answer object columns

Reading one row across columns of different types with iloc, loc or iterrows, transpose and melt over such columns, and concat of parts that share no type now answer an object column, as pandas does. describe answers count, unique, top and freq for text, flags and categories, and describe(include="all") puts those rows before the numbers' rows. str.split and str.rsplit without expand and the new str.findall answer a list per row, and dt.time answers the time of day. An object column's gap reads as None, NaN or NaT the way pandas' does, and a list in a cell prints as pandas prints it, as [a, b]. Timestamp of a number with a zone now counts from the epoch in UTC, as pandas does, where it used to read the number as a time on the zone's clock.

Changed: a grouped reduction stops replicating tables that outnumber its rows

A grouped reduction splits across cores with a private table per worker, and the number of workers was capped only by memory. It is now also capped so the tables together hold no more entries than there are rows, because each table is zeroed and merged and past that line those two passes are more work than the scatter. A reduction that loses its tables this way takes the partitioned route. TPC-H q20's sum of about 910,000 lines into 543,210 groups was given seven tables, and on a busy 8 core VM its mean over forty runs went from 118 ms on the replicated route to 54 ms on the partitioned one.

Added: Object columns

A column can hold any Python value, as pandas' object dtype does. A list whose values do not share one type, such as text mixed with numbers, a list in each cell, a dict, a Decimal or an integer too large for int64, now builds an object column where it used to be refused, and dtype=object and astype(object) make one on request. The values are written into a text column, so every operation that moves rows carries them unchanged, and they are read back as they were by tolist, iteration, one cell, to_dict and to_numpy. Comparisons, arithmetic, isin, fillna, sorting, min, max, sum and prod compute one value at a time in Python, as pandas does, and .str works on an object column of text. Document 95 of the compat notes describes the design and the few places the answer still differs from pandas.

Added: str.extractall

Series.str.extractall(pat, flags=0) answers one row for each match of the pattern in each value, labelled by the row it came from and the number of the match on a level named match, with one column per group named as extract names them. It runs Python's re over the values, as pandas does, and a pattern with no groups is refused with pandas' message.

Added: SQL unnest over a list written out

unnest([...]) in a select list writes each row once for every element of the list, so SELECT x, unnest([x, x * 2]) FROM t has two rows for each row of t. With several unnests the row is written as many times as the longest list is long and the shorter ones are padded with nulls, and an empty list or a NULL writes no rows, as in DuckDB. An unnest can be part of a larger expression, as in unnest([1, 2]) + 1, and its column is named unnest(main.list_value(1, 2)), the name DuckDB gives it. FROM unnest([...]) is a table with one column called unnest. A plan node, UNNEST, and an operator of the same name do the work. firepanda has no column that holds lists yet, so the list has to be written out in the query, and an unnest in a query that aggregates is refused for now.

Added: SQL unnest over a list written out

unnest([...]) in a select list writes each row once for every element of the list, so SELECT x, unnest([x, x * 2]) FROM t has two rows for each row of t. With several unnests the row is written as many times as the longest list is long and the shorter ones are padded with nulls, and an empty list or a NULL writes no rows, as in DuckDB. An unnest can be part of a larger expression, as in unnest([1, 2]) + 1, and its column is named unnest(main.list_value(1, 2)), the name DuckDB gives it. FROM unnest([...]) is a table with one column called unnest. A plan node, UNNEST, and an operator of the same name do the work. firepanda has no column that holds lists yet, so the list has to be written out in the query, and an unnest in a query that aggregates is refused for now.

Fixed: SQL ORDER BY ... DESC puts its nulls last

A descending sort key in an ORDER BY now puts its nulls last, as DuckDB does for both directions, where it used to put them first. NULLS FIRST still puts them first. A window's order already did this.

Added: SQL recursive CTEs

WITH RECURSIVE now runs. The anchor is everything left of the last UNION, and it runs once. The recursive term then runs again and again over the rows the last round kept, until a round keeps nothing. UNION ALL keeps every row, and UNION keeps only rows no earlier round kept, so a walk over a graph with a cycle stops. The term's rows are cast to the anchor's types, rounding the way a SQL cast does, as in DuckDB. The term may read the CTE more than once, for example to join it to itself, and a later CTE or the query may read the recursive one as often as it likes. A filter written above a recursive CTE stays above it. A recursive CTE whose union lines up by name, or which holds a WITH of its own, is refused by name.

Added: The operations that make number names

pivot, pivot_table, unstack and crosstab spread number and instant values across the columns, get_dummies encodes numbers, bools and dummy_na under their own values, and dot with a matrix, from_dict and from_records with rows, concat of unnamed series along the columns and Series.to_frame() name their columns 0, 1, 2 as pandas does. str.partition, str.rpartition, str.extract, str.split(expand=True) and str.rsplit(expand=True) name their columns by number rather than by text, read_csv(header=None) numbers the columns and names the index 0 with index_col=0, and read_json reads names like "0" as numbers under convert_axes. An index name can be a number too, through rename_axis, Index.rename and set_names. Each is compared with pandas in python/tests/test_made_names.py.

Added: SQL named windows

A WINDOW w AS (...) clause now lowers. OVER w is the window as it is written, and OVER (w ORDER BY ...) or OVER (w ROWS ...) starts from it and adds what w leaves out. One named window may start from another, written before or after it. As in DuckDB, a window that adds a partition or an order w already has, or starts from a w with a frame, is refused, as are a name given twice and a name that is not given. A column named after its call spells the window out, so sum(x) OVER w is called sum(x) OVER (PARTITION BY g), the name DuckDB gives it.

Added: Column names that are not text

A frame's columns and a series' name can be whole numbers, floats, bools, instants or spans, as in pandas. A frame built from a list of lists, a numpy array or a flat list names its columns 0, 1, 2 and so on, columns answers a RangeIndex or an int64 index for them, and transpose works for row labels of any of those kinds, dates included. Selecting, loc, at, drop, rename, sort_values, groupby, set_index, astype, merge, concat, drop_duplicates, nlargest, reindex and the repr all read such names, concat(axis=1, ignore_index=True) numbers the columns, and the Arrow export names each field with the text of its name. The extension still holds text, so a name that is not text is written into text on the way in and read back on the way out, as document 94 of the compat notes describes. Names that mix text with other kinds print and work, and only columns refuses them, because pandas answers an index of objects there.

Changed: a group by on two wide integer keys packs them without factorizing either

A group by whose keys are integers of one dtype packs the raw values into one uint32 when their ranges multiply out to more than 64 times what a table is laid over, as long as the product fits. The per key factorizes and the pass that packed their codes are skipped, and the packed column is hashed as before. TPC-H q20 groups about 910,000 lines by part and supplier, whose ranges multiply out to two billion. On a busy 8 core VM, interleaved best of 25 runs, its group ordinals went from 109, 119 and 184 ms to 79, 55 and 117 ms. The ordinals are the same first appearance order either route gives.

Added: The columns of a frame as an index

DataFrame.columns now answers an Index of the column names, and an empty RangeIndex for a frame without columns, as pandas does, rather than a list. Reading it as a list, by iterating, len, in or position, is unchanged, and the index methods such as tolist, get_loc, equals, intersection and difference now work on it. df[key] also takes an index, a column or an array of names, so df[df.columns[:2]] selects columns. Document 94 of the compat notes describes this and the next step, which is column names that are not text.

Added: Group by windows take pandas' arguments by name

GroupBy.rolling, GroupBy.expanding and GroupBy.ewm now have the same parameters as pandas, in the same order and with the same defaults, rather than passing any arguments through, so a signature check and a misspelled keyword both behave as they do in pandas.

Added: SQL windows with an ORDER BY, a frame and the window functions

A window in SQL can now be ordered and framed. OVER (PARTITION BY ... ORDER BY ...) runs the fold up to the row's last peer, and ROWS, RANGE and GROUPS frames take UNBOUNDED, n PRECEDING, n FOLLOWING and CURRENT ROW bounds with EXCLUDE CURRENT ROW, GROUP, TIES and NO OTHERS. row_number, rank, dense_rank, percent_rank, cume_dist, ntile, lag, lead, first_value, last_value and nth_value are lowered, the five that read another row take IGNORE NULLS, and a QUALIFY can filter on any of them. Sums, counts, averages, minimums and maximums over a frame are answered from blocks folded once, so a running total costs a logarithm per row rather than the width of the frame. An order key's nulls go last whichever way it sorts, as in DuckDB.

Added: moving windows and take on a group by, DataFrameGroupBy.value_counts, and concat with keys

rolling, expanding and ewm on a group by run the window over each group on its own and label the answer by the group and the row, as pandas does, and take does the same with the rows at the given places in each group. DataFrameGroupBy.value_counts and corrwith are in, and both kinds of value_counts now order ties the way pandas does, by where each combination is first seen. concat takes keys, or a dict, and names, and puts each part's key in front of its rows' labels as levels of a MultiIndex. Keys across the columns stay refused, since a column name is text.

Added: group keys in front of what apply answers, and GroupBy corr and cov

apply answering a series or a frame a group with group_keys=True, the default, now puts each group's key in front of the answer's labels as levels of a MultiIndex, as pandas does, rather than refusing. With as_index=False the answers stay group after group on their own labels. DataFrameGroupBy.corr and cov answer each group's matrix labelled by the group and the column, and SeriesGroupBy.corr and cov answer each group's value against another column.

Added: DataFrame.value_counts keeps pandas' order among equal counts

DataFrame.value_counts takes the combinations in the order they first appear and sorts them stably by count, as pandas does, so combinations with the same count come in the order pandas gives them rather than in label order.

Added: level methods and reshaping on rows labelled by a MultiIndex

loc takes a whole tuple, a prefix of one, or a row key and a column together on a frame or a column labelled by levels, and drops the levels a prefix names, as pandas does. droplevel, swaplevel, reorder_levels and xs are on frames and columns, sort_index takes level, DataFrame.value_counts, DataFrame.stack, Series.unstack and SeriesGroupBy.value_counts are in, and pivot and pivot_table take several row keys. Each works by rewriting the labels in Python over what the extension already does. A key column is no longer offered as an attribute of a group by, so groupby("a").a is an AttributeError as it is in pandas.

Changed: several comparisons against constants are anded in one pass

conjoin_compares in firepanda.kernel.binary takes a list of columns, comparisons and constants and answers the rows on which every comparison holds, reading each column once and writing one mask. Five comparisons that went through binary_value_any and conjoin wrote five masks and read them back, so a filter shaped like TPC-H q6's over six million rows goes from 59 ms to 30 ms on a shared six core box. A comparison it has no loop for, such as one on text or one whose column would need converting, is worked out the ordinary way and folded in. Nulls keep nothing, so the answer has no nulls.

Added: rows labelled by a MultiIndex

A frame or a column can now be labelled by a MultiIndex. It is built with index= given a MultiIndex, by set_index with several keys or with append=True, and by a group by on several keys, whose reductions, size, agg and apply now label their rows by every key as pandas does. .index answers the MultiIndex, reset_index puts all of its levels or the ones named by level= back as columns, and the repr prints the levels side by side and sparsified. Alignment, sorting, filtering, reindex, rename_axis with a name per level, to_csv, items, itertuples and in with a tuple all work on the tuples. Each tuple is held by the extension as one text label written to sort as the tuple sorts, which document 93 of the compat notes describes. A column of a group by can now also be read as an attribute, df.groupby("k").v.

Added: A zone given as a tzinfo object to tz_localize, and fixed offsets named after UTC

tz_localize on a column of instants or on a DatetimeIndex takes a tzinfo object, a zoneinfo zone or a fixed datetime.timezone, where it used to take only a zone name. A fixed offset is named after UTC, so a column read against seven hours east of it is datetime64[us, UTC+07:00], as pandas prints it, wherever the zone came from.

Added: Instants and spans refuse the casts pandas refuses, and elapsed times read from gaps are seconds

astype on a column of instants or spans refuses every float type and every whole number type but int64 with pandas' TypeError and words, in a series and in a frame, where it used to hand back counts. to_timedelta on an empty list or on nothing but missing values answers seconds, as pandas does, or nanoseconds for a column of floats, and an empty list no longer fails.

Added: Text joined and repeated by operators, lists beside a frame, and spans beside NaT

A column of text takes + with text, a list of text or another column of text, lined up by label with a missing row on either side missing in the answer, and * with a whole number or a column of them repeats each row, as pandas does. Every other arithmetic on text, and ordering text against a number, is refused with pandas' words, while a number is never equal to text. A list, a tuple or a numpy array beside a frame is read as a series along the columns, or along the rows for the named form on the rows, with pandas' message when the length is wrong. Spans scaled or divided by NaT are refused as pandas refuses them.

Added: Lists beside a series, numbers read as spans or instants, and rounding spans

A list, tuple or numpy array beside a series is read as a series on the same labels, with pandas' messages when the lengths differ. astype reads a column of numbers as counts of the unit it names when asked for timedelta64[unit] or datetime64[unit], spans round, floor and ceil to a fixed frequency with ties to even, a TimedeltaIndex shifts, rounds and hands back Python timedeltas, spans and instants take a quantile, a clip bound and an interpolation, the spread of instants is a span, a span over a missing span is a NaN rather than a missing value, and symmetric_difference of an index with a frequency infers the frequency of its answer as pandas does.

Added: Spans scaled by a number and divided by a span

A column or an index of spans times or over a number, s * 3, s / 7 or s // 2, is now spans, truncated toward zero as pandas truncates, with NaT over zero. A span over a span is floats, a floor of spans is whole numbers when nothing is missing, and a remainder of spans is spans in the finer unit, with the sign of the divisor. A number series times a span series lines up by label. A fixed offset such as Day(1) or Hour(2) added to spans is the span it lasts, and Timestamp and Timedelta read numpy's datetime64 and timedelta64, so these work on the right of an operator too. TimedeltaIndex keeps the scaled frequency, so timedelta_range("1h", periods=3, freq="h") * 2 holds 2h.

Added: An index of instants or spans passes its frequency on

A DatetimeIndex or a TimedeltaIndex now hands its frequency to the index an operation builds, by pandas' rules. unique, drop_duplicates, fillna, dropna, view and a same type astype keep it. take and sort_values keep it when the positions make a slice, times the slice's step, so a daily index sorted backwards is -1D, and a mask keeps it when the labels it keeps are one run. delete and insert keep it at the ends, union and intersection keep it or read it off the labels the way pandas does, append keeps it for pieces that follow on, and reindex takes the target's. normalize reads it off the labels, tz_localize keeps it on UTC, tz_convert keeps a fixed step, and arithmetic keeps, flips or scales it. Index.sort_values now takes key, and reindex of an index of instants takes a list of them.

Added: Index arithmetic and comparisons

An index now takes the arithmetic operators, +, -, *, /, //, % and ** on either side, and -, + and abs, against a number, a scalar instant or span, a list, another index or a series, and gives back an index, or a series for a series, the way pandas does. <, <=, > and >= join == and !=, and an index of instants or spans compares with a scalar, text or another index. An index of instants or spans stays a DatetimeIndex or a TimedeltaIndex through unique, delete, take, the set operations, append, diff and Index(series), to_timedelta of a list gives a TimedeltaIndex, and drop, insert, putmask and isin work on both. A column of spans takes a span in where, mask and fillna. Inserting a number into an index of instants, which pandas turns into an index of objects, is refused.

Added: DataFrame.to_xml

DataFrame.to_xml writes the frame as an XML document the way pandas does, with one element per row and each column as an element inside it, or as an attribute through attr_cols. It takes pandas' index, root_name, row_name, na_rep, elem_cols, namespaces, prefix, encoding, xml_declaration, pretty_print and compression. parser="etree" uses the standard library, and the default lxml parser, along with an XSLT stylesheet, needs lxml installed, as in pandas. The document is answered as text, or written as bytes to a path or a binary handle. to_pickle and to_xml now share one writer for compressed bytes.

Added: DatetimeIndex and TimedeltaIndex hold a frequency

An index made by date_range, bdate_range or timedelta_range holds the offset it was made at, as freq and freqstr, and its repr shows it as pandas does. The constructors take freq=, where infer takes the frequency the labels keep and any other frequency has to be one they keep, with pandas' error when it is not, and freq can be set the same way. A slice keeps the frequency, times its step, and a copy, a new name, as_unit and shift keep it too, where shift now uses the held frequency when none is given. Index of an index of instants or spans is now a DatetimeIndex or a TimedeltaIndex, as in pandas. The index of a series or a frame does not keep its frequency yet, because the core rebuilds it.

Added: infer_freq, inferred_freq and dt.freq

firepanda.infer_freq names the frequency a run of instants or spans keeps, as pandas' alias for it, by the same rules pandas' frequency inferer uses: years, quarters and months anchored at the calendar or business start or end, weeks anchored on a weekday, business days, business hours, the week of the month, and fixed steps from days down to nanoseconds, read on the wall clock for labels with a zone. It takes a DatetimeIndex, a TimedeltaIndex, a column of either or a list of dates, and raises pandas' errors for fewer than three labels or labels that are not instants. DatetimeIndex.inferred_freq, TimedeltaIndex.inferred_freq and Series.dt.freq give the same answer, or None where pandas gives None.

Added: asfreq on a frame and a column

DataFrame.asfreq and Series.asfreq put the rows on a fixed frequency from the first label to the last, as pandas does. A label the rows do not have is filled by method from a neighbour or with fill_value, normalize takes the labels to midnight, and the labels keep their unit, their zone and their name. It is pandas' own recipe, a date_range and then a reindex, so it fills exactly as reindex does.

Added: reindex fills from a neighbour and reindexes on instants

DataFrame.reindex, Series.reindex and both reindex_like methods take pandas' method, limit and tolerance. A label the frame does not have reads from the label before it under ffill or pad, the label after it under bfill or backfill, and the closer of the two under nearest, with pandas' rule for a tie, for a falling index, for how many rows one label may fill, and for how far away it may be. Row labels that are instants or spans reindex too, which they could not before, and a target given as an index keeps its own name, as it does in pandas. Index built from a list of instants is a DatetimeIndex, and from a list of spans a TimedeltaIndex, as in pandas. The mistakes are pandas' mistakes with pandas' words, including an unknown method, a limit that is not a positive whole number, a limit on labels out of order, labels that are neither rising nor falling, duplicate labels and a tolerance of the wrong size.

Added: tz_localize and tz_convert on a frame and a column

DataFrame.tz_localize, DataFrame.tz_convert, Series.tz_localize and Series.tz_convert move the row labels between time zones and leave the values alone, as pandas does. They take pandas' axis, level, copy, ambiguous and nonexistent arguments, warn about copy as pandas does, and raise pandas' errors for labels that are not instants, a level that is not there, and an axis a column does not have. An empty axis of any kind comes back as an empty axis of instants in the zone asked for.

Added: The grouped kurt and SeriesGroupBy.ohlc

DataFrameGroupBy.kurt and SeriesGroupBy.kurt give pandas' excess kurtosis for each group, including NaN for a group with fewer than four values and zero for one with no spread, and agg("kurt") reaches them. The core has no grouped kurtosis, so the answer is built from grouped sums of centred powers, centred twice so that a group far from zero keeps its digits. SeriesGroupBy.ohlc gives the first, highest, lowest and last value of each group as open, high, low and close.

Changed: a group by skips text keys another key already decides

A group by on several keys factorized every key, and a text key's factorize hashes and compares every row's bytes. When the keys include text and a fixed width key, the fixed width key is now grouped first and every other key is checked against the first row of each of its groups, one comparison a row. If they all agree, which is what grouping by a customer's key and then by its name, address, phone and comment looks like, those groups are the answer, and otherwise the grouping goes on as before without factorizing the first key twice. On the 13900K the grouping in TPC-H q10 goes from 5.8 to 1.7 ms and its group by from 8.5 to 4.7 ms.

Added: DataFrame.to_html and the notebook repr

DataFrame.to_html takes every one of pandas' arguments and writes the same markup, character for character, and DataFrame._repr_html_ gives a notebook the table pandas shows, under the same display options. The cells are the text to_string writes, so floats, dates, spans, gaps and cut rows and columns look as they do in pandas. sparsify is accepted and has nothing to do, because firepanda has no MultiIndex.

Fixed: uint64 values past the int64 range

A uint64 value above the int64 range came back from tolist(), to_csv and every other path through the column's values as a negative number, because the core hands the values out through a signed integer. They are now put back before anyone sees them. read_csv on a plain file also read such a value as float64, where pandas reads uint64, and now reads it as pandas does.

Changed: a join hands over a side it takes whole

When every row of one side comes out of a join once and in order, which is what an inner join of a fact table against a dimension it always finds looks like, the join now shares that side's columns instead of gathering each of them through the identity. On the 13900K, TPC-H q10's join of 114,705 lines against nation goes from 3.1 to 0.8 ms, and the whole query from 29 to 26 ms. q9 goes from 46 to 40 ms and q11 from 6.4 to 5.6.

Changed: a filter that keeps few rows reads only those rows

A filter copied every row of a column and advanced its output by the mask, so it read the whole column whatever the mask kept. When a stretch of the mask keeps fewer than one row in eight, the filter now finds the kept rows eight mask bytes at a time and reads only those. On the 13900K, filtering six million doubles down to one row in a hundred goes from 0.80 to 0.33 ms, and TPC-H q6's filter of two columns from 2.2 to 1.2 ms.

Added: read_csv takes every argument, plus read_table and read_fwf

read_csv now reads what pandas reads: a path, a handle, bytes or a URL, compressed or not, in any encoding, with every one of pandas' arguments. A plain file with pandas' defaults still goes to the core reader. Everything else goes to a new Python reader in _textread.py, which splits the text the way pandas' C parser does and picks each column's type the way it does. That covers separators, whitespace and patterns, header, names, skiprows, nrows, skipfooter, dtype, converters, missing text, parse_dates, thousands, decimal, comments, quoting and escapes, bad lines, and chunksize and iterator through TextFileReader. Blank and repeated names, an extra leading field, padded numbers, an empty file and a byte order mark now come out as in pandas as well. read_table and read_fwf are new, and a missing file is now FileNotFoundError.

Added: Parquet, Feather and ORC through pyarrow

DataFrame.to_parquet, to_feather and to_orc, and read_parquet, read_feather and read_orc, take pandas' parameters and read and write through pyarrow, which is imported only when one of them is called, so firepanda still has no runtime dependencies. A frame goes to pyarrow through the Arrow stream it already exports. The pandas metadata key is written in the layout pyarrow writes for pandas 3.0.3: the row labels as columns or as a range, each column's pandas type, and attrs. A file firepanda writes therefore reads back in pandas with its labels and types, and a file pandas writes reads back in firepanda. index, compression, partition_cols, columns, filters and filesystem are handed to pyarrow. dtype_backend keeps Arrow's types, so a missing integer stays an integer. ORC refuses labels other than the default range, as pandas does. An unknown engine, storage options on a local file and a missing file raise what pandas raises.

Added: frames, columns and indexes pickle, with to_pickle and read_pickle

A frame, a column or an index now pickles. It travels as the bytes of the Arrow C data it already exports, with the row labels as leading columns, so pickling needs neither pyarrow nor a Python object per value. attrs, the refusal of repeated labels, names that are numbers or tuples, and the class of an index (a range, instants, spans or pairs) all come back. DataFrame.to_pickle, Series.to_pickle, pandas.to_pickle and pandas.read_pickle take pandas' parameters. They write and read gzip, bz2, xz, zip, tar and zstd, chosen by the file's ending or by name, to a path or a handle. An unknown compression, storage options on a local file and a missing file raise what pandas raises. A firepanda pickle names firepanda's classes, so pandas cannot read it, and the other way round. A column name that is a tuple now prints as pandas prints it, as Name: (a, 1).

Added: floats print by display.precision and display.float_format

The repr and to_string of a frame or a column now read display.precision for how many digits a float gets, and display.float_format when no float_format is given, as pandas does, for float columns and for floats in text columns alike. set_eng_float_format sets that option to pandas' engineering formatter, so floats print as 1.23M or 1.230E+06, and show_versions prints the system and the versions of firepanda and what it reads through, as a table, as JSON or into a file.

Added: NA, IndexSlice, eval, col and functions as loc keys

firepanda.NA is pandas' missing value, with its three-valued logic and every operator answering as pandas 3.0 does against numbers, flags, text, bytes, dates, spans and numpy arrays, and isna knows it. IndexSlice hands back what goes between its brackets. firepanda.eval reads an expression over the caller's variables with the reader DataFrame.eval uses, assigns into a target, reads several lines into one, and refuses @ in pandas' words. firepanda.col makes an expression that assign, loc, iloc, [], where, mask and case_when read on the frame, and it prints as pandas prints it. loc and iloc on a frame or a column now take a function of the object on either axis, and a query or eval can call pandas' math functions such as sin, log10 and arctan2.

Added: attrs and flags carried as pandas carries them

Frames and columns have attrs, flags and set_flags, and firepanda.Flags is exported. Which answers keep attrs follows lists measured on pandas 3.0 by calling every method of a frame, a column, a group by, a resample, the windows and the str, dt and cat accessors on an object holding them: most answers keep a deep copy, an operator takes the other side's when it has some, merge, join and concat keep them only when every input holds the same, and the reductions pandas runs in compiled code drop them. allows_duplicate_labels=False is carried the same way and raises pandas' DuplicateLabelError with its table of positions. Until some object holds attrs or flags the wrappers call straight through, so programs that never use them pay nothing measurable. The @name lookup in query and eval skips the wrappers' frames.

Added: Rolling windows measured as a span of time

rolling takes a window given as text such as "30s", a Timedelta, a timedelta or a fixed offset, and reaches back that far along the row labels or along the column named by on, which may hold dates or durations. Each window starts and stops where pandas' own bounds put it, under every closed and with center, over times that rise or fall. on works for a window of rows too, leaves its column out of the reduction and puts it back into the answer where pandas does. Picking columns out of a window with [] keeps the window, obj, on and window answer what pandas answers, and the window prints pandas' repr. The mistakes are pandas' own words: a month, which is not a fixed span, times out of order or with a gap, a step, and an on that is not a column of dates.

Changed: a join onto a tall side with repeated keys builds on every core

The side of a join that is bucketed by key, when its keys repeat, was bucketed on one thread, and on TPC-H q13 that was 10.6 of the outer join's 13 ms. From 262,144 rows the bucketing now runs on every core, keeping a count per key for each core when the keys are few and moving the rows into ranges of keys first when they are many, and every bucket still holds its rows in order. On the 13900K q13's 1.48 million orders by 150,000 customers go from 10.2 ms to 2.6 ms, and 4.2 million rows over a million keys from 93 ms to about 10.

Changed: grouping half a million rows or more into many groups uses every core

A factorize under four million rows stayed on one core whatever its group count, and when the groups outgrow a core's own cache every probe of that table is a trip to memory. Such a column now takes the partitioned route from half a million rows, where each core owns a disjoint share of the groups. On the 13900K a million rows of a hundred and twenty eight thousand groups goes from 25.6 ms to 11.7 ms, three million rows of sixty five thousand from 108 ms to 13 ms, and TPC-H q20's group by of nine hundred thousand lines into half a million part and supplier pairs from about 37 ms to about 24.

Fixed: cat.codes has no name

Series.cat.codes answered an empty name, and pandas answers none. It now has no name, whatever the column is called.

Changed: cat.codes answers pandas' width

Series.cat.codes answers the codes in the narrowest integer type that holds them, int8 for fewer than 128 categories and int16 up to 32767, as pandas does. The column still stores int32 codes, for the reason document 26 gives, and the answer is cast to the narrow width.

Added: Categorical, CategoricalDtype and CategoricalIndex

firepanda.Categorical, CategoricalDtype and CategoricalIndex follow pandas. A category column's dtype is now a CategoricalDtype that carries its categories and order flag and still equals the word "category", astype takes one with categories decided, Categorical.from_codes and codes hold the codes as narrow as pandas does, a Series built from a Categorical keeps its categories, and each prints the way pandas prints it.

Added: grouping by a category column, and observed=False

A frame groups by a category column in the order of its categories, and the key comes back as a category column, or as labels that print as pandas' CategoricalIndex, carrying every category. observed=False makes a group of each category that has no rows, and each reduction answers for it what pandas answers for an empty group: zero for size, count, sum and nunique, one for prod, false for any, true for all, and a gap for the rest, with agg reading each function it is given. idxmax and idxmin raise pandas' error when a category has no rows, and apply, a function passed to agg, and several keys with observed=False are refused. An index of categories now has categories, ordered and codes.

Added: grouping by keys from outside the frame, and Series.groupby

groupby takes every key pandas takes: a column of values lined up on the row labels, an array, list or Index as long as the frame, a function or a dictionary of the row labels, and the labels themselves by level or by their name. Each key is named as pandas names it, including index and level_N for keys with no name under as_index=False, and a key from outside the frame is never reduced or handed back by head, filter or iteration. Series.groupby takes the same keys. Grouping a frame by one of its own columns leaves that column out when the values are the column's, where pandas asks whether it is the same object. A NaN in a key of floats is now left out under dropna, a column's size carries the column's name, and pd.Grouper reads (*args, **kwargs) as pandas' does.

Changed: a join reads its keys through a table indexed by value more often

A join on an integer key could look its keys up in a table indexed by the key value only when the range of values was under half the larger side's height, and otherwise it hashed both sides, with the build on one core. The bound is now both sides' height together. TPC-H q10 joins a quarter's orders to all customers on a key range just past the old bound, and on the 13900K the key alignment for that join goes from 1.03 ms to 0.24 ms and the whole join of the key columns from 1.86 ms to 1.08 ms.

Added: describe over a column's groups, and pd.Grouper

SeriesGroupBy.describe answers the count, mean, spread, extremes and percentiles of each group as float64 columns on the keys, with the same percentile labels as Series.describe. firepanda.Grouper groups by a column, and on its own it brings its own sort and dropna as pandas reads them, so a plain one leaves the groups in the order they are first seen. A grouper with a frequency bins a column of moments as resample(freq, on=key) does, and prints as pandas' TimeGrouper.

Added: LATERAL subqueries

A LATERAL subquery written after a comma, or on the right of an inner or a cross join, now runs, as in DuckDB. It is not run once per left row. Its FROM is joined to what is on its left, an equality in its WHERE between the two sides becomes a join key, and its select list is worked out over each pair, so FROM shops, LATERAL (SELECT qty FROM sales WHERE sales.shop = shops.shop) is one hash join. A left row the subquery finds nothing for is dropped. A condition on a JOIN LATERAL is a filter over the result. An aggregate, a DISTINCT, an ORDER BY or a LIMIT inside a lateral subquery is refused by name, as is a left join to one. A LATERAL with nothing to its left is an ordinary subquery.

Added: a Python function over the groups

GroupBy.agg, GroupBy.transform and a new GroupBy.apply take a Python function and run it once a group, as pandas does. agg hands it a column's values in each group and reads the type from the answers, and a function can sit in a list, a mapping or the named form beside the names, with lambdas numbered as pandas numbers them. transform fits each answer to its group's rows and puts them back in the frame's order, with a row whose key is missing answering missing. apply answers one row a group for one value or for a series with the same text labels each time, and the rows on their own labels when group_keys=False, which groupby now takes. An answer pandas would put on a MultiIndex is refused.

Added: Timestamp reads a year, a month or a quarter

Timestamp("2024"), Timestamp("2024-3"), Timestamp("2024/03") and Timestamp("2024Q1") are the first day of the span they name, as in pandas, where they used to be refused as text that is not ISO 8601.

Added: offset anchors by position, and taking no zone off

The month anchored offsets name their anchor in their signature, as pandas' YearEnd shows month, and take it as the third positional argument. Series.dt.tz_localize(None) and DatetimeIndex.tz_localize(None) on moments that carry no zone hand back the readings as pandas does, where they used to raise.

Added: offsets as resample rules and as spans

resample takes a tick or Day from firepanda.offsets as its rule, the same as its frequency text, and refuses a calendar offset the way it refuses the calendar text. A rule that is neither text nor an offset raises pandas' TypeError. Timedelta reads a tick, keeping the tick's unit for milliseconds and finer and seconds for anything coarser, and refuses Day and calendar offsets with pandas' words.

Added: pandas.offsets and DateOffset

firepanda.offsets now carries every name in pandas.tseries.offsets, and firepanda.DateOffset is the same class pandas exports. Month, quarter, half year and year offsets and their business forms, the semi month, week, week of month, Easter and 52 or 53 week fiscal offsets, business days and business hours with custom week masks and holidays, Day and the ticks all move a Timestamp the way pandas 3 does, including zero steps, negative steps, normalize, zoned moments and the time unit. They print, compare, hash, roll forward and back and read their own frequency text as pandas does. An offset adds to and subtracts from a column of moments and an index of moments, and date_range takes one as its freq. Business hours that run over midnight and a holiday calendar other than numpy's are refused. test_offsets.py checks 567 cases against pandas.

Changed: a comparison against a coded column spreads its answers on every core

Comparing a dictionary coded text column with a constant, or asking it is_in, answers once per distinct value and then copies each row's answer out by its code. That copy ran on one core, and on TPC-H q10's l_returnflag == "R" over six million rows it took 6.9ms of the query. A column of 65536 rows or more now copies in morsels on every core, and a shorter one keeps the old path.

Fixed: astype("category") of a column that is not text refuses cleanly

A category of numbers, flags or instants raised a bare RuntimeError from inside the core's cast. It now raises UnsupportedError, which is a NotImplementedError, naming the column's type, on a series and on a frame. pandas keeps such categories in the values' own type, and firepanda holds categories as a text column.

Fixed: A row of values written through loc is read as numpy reads it

df.loc[label] = [1, 2.5] now reads whole numbers beside a float as floats, as pandas does when it turns the list into a numpy array, so a new row made from them makes whole number columns float64. A list with text, a flag or a gap in it keeps each value's own type, as numpy's object array does.

Added: Writing into a frame through loc, iloc, at and iat

A frame now takes df.loc[rows, columns] = value and the same through iloc, at and iat. Each named column is written the way a series is, so a value that does not fit raises pandas' TypeError and a gap in some rows of whole numbers makes them float64, while a write that covers every row by a slice refuses to change the type, as pandas 3 does. A list goes one value a column, rows of values one row a row, a series by column name into one row, and a frame by name. loc with a new name makes a column, float64 for numbers and text for text, and with a new row label puts a row on the end with pandas' types. iloc with a frame of values, and cells pandas would hold as objects, are refused.

Added: Writing into a series with s[key] = value, loc, iloc, at and iat

A series now takes writes as pandas 3 does. A label, a list of labels, a slice, a mask, or a position through iloc and iat picks the rows, and one value, a list, or a series goes in. A series is lined up on the labels under loc and a mask, and read by position under iloc. A value that does not fit the column raises pandas' TypeError, a gap widens whole numbers to float64, and loc, at or s[label] with a new label puts a row on the end. The write rebinds the series, so copies taken before keep their values.

Added: kurt of a float column summed in pandas' order

Series.kurt and DataFrame.kurt on a float column now take pandas' nankurt sums in numpy when numpy is installed, over the whole column with its gaps as zeros, as pandas does. Far from zero the fourth moment moves with the order of adding, and at values near 4.6e18 with a spread of 4e9 two orders differ by 2e-7, which is more than pandas' own distance from the exact answer, so only the same order agrees. Without numpy the answer comes from the column's own sums as before, and whole number columns keep the exact shift from their least value.