v0.8.34

Built against Mojo 1.0.0 (ed45d567).

A patch release. On the pandas side, a frame and a column gain interpolate, corr, cov, autocorr, unique, factorize, describe, kurt, melt, get_dummies, query, eval and resample, a group by gains ffill, bfill, pct_change and filter, and iloc reads one row across columns and raises pandas' IndexError past the end. In SQL, a semi or anti join checks the rest of its condition, so TPC-H q21 runs. Text is hashed sixteen bytes a step in two lanes, which makes hashing a short string about four times faster and helps every text group by and join.

Fixed: a gap interpolate leaves is NaN

The gaps interpolate leaves behind, at the front of a forward fill, past a limit or outside a limit_area, and every row of a column with no value at all, are NaN now rather than null, since NaN is the only missing value a float column has in pandas. Before this they crossed to Arrow as nulls and a consumer counting nulls saw a different column from the one pandas answers.

Added: interpolate on a frame and a column

Series.interpolate and DataFrame.interpolate fill each gap on the straight line between the values either side of it, with numpy's arithmetic so the answers agree to the last bit, along row positions for linear and along the labels for index and values. limit, limit_direction and limit_area leave the gaps pandas leaves, whole numbers or flags with no gap come back untouched, whole numbers with a gap come back as float64, and inplace=True hands back the object as pandas does. Text, flags with a gap and a category are refused with pandas' errors, every bad argument with pandas' message in pandas' order, and the methods pandas hands to scipy as not supported.

Added: corr, cov and autocorr on a frame and a column

DataFrame.corr and DataFrame.cov answer the square frame pandas does, labelled by the numeric columns both ways, pairing each two columns over the rows where both hold a value. Series.corr, Series.cov and Series.autocorr answer one number, lining the two columns up by label first. Pearson and Spearman are both there, min_periods and ddof behave as in pandas, including the difference in DataFrame.cov between a frame with a gap and one without, and text is refused with numpy's words unless numeric_only is set. Kendall is refused as not supported yet.

Added: a row position past the end raises pandas' IndexError

df.iloc[9999] on a short frame raises IndexError with pandas' message, "single positional indexer is out-of-bounds", from either end. It used to pass the position to the core and raise the core's slice message.

Added: one row of a frame as a series, and names that are numbers

df.iloc[0], df.iloc[-1, [1, 0]] and df.loc[label] read one row across several columns, which used to raise. The answer is a series labelled by the column names and named by the row's label, and it holds the one type every column fits: columns of one type keep it, and numbers meet the way numpy's do, so int32 next to int8 is int32 and float32 next to int64 is float64. A number next to a flag or to text is an object column in pandas, and that is still refused with NotImplementedError, since firepanda has no object column.

A series name can be a number now. The core keeps a name as text, so the wrapper keeps the number next to it and name answers the number while the core's text still spells it. rename(5).name is 5, a row's name is its label, and DataFrame.quantile(0.5).name is 0.5, where it used to be None. quantile(1) is named 1.0, as in pandas. An operation that makes a new column carries the name as text, so rename(5) + 1 is still named "5".

Fixed: a column labelled with empty text exports to Arrow

A frame with a column labelled "" could not be read by pyarrow, which raised a SystemError from pa.table, because the column's schema went out with a null name, the form Arrow keeps for an array with no name at all, and pyarrow refuses a struct child without one. get_dummies makes that label whenever a column holds an empty string. A frame's column now always carries its name, empty or not, and a lone column or index with no name still goes out with a null one.

Added: unique and factorize

Series.unique and firepanda.unique give each value once in the order first seen, with a missing value kept once, and a categorical column keeps every one of its categories, used or not, as in pandas. The answer is a FirepandaArray, which is pandas' array answer here: values with a type and no row labels or name, with a length, positions, slices and the Arrow PyCapsule export, and Series.array gives one too. Series.factorize, Index.factorize and firepanda.factorize give an int64 array of codes and the uniques, as an index for a column or an index and as an array for an array, with sort and use_na_sentinel doing what they do in pandas, a float NaN included. The codes are the group numbers of a group by, so no Python loop runs over the rows. A list is refused with pandas' own TypeError, and factorize on a categorical column is refused, because its uniques are a categorical index, which firepanda does not hold yet.

Added: pandas' aliases and small helpers on a frame and a column

A frame and a column now have div, divide, rdiv, multiply and subtract as the names pandas gives the arithmetic already there, isnull and notnull beside isna and notna, and add_prefix, add_suffix, pipe, first_valid_index, last_valid_index, pop, equals and to_dict. A frame has insert, which puts a column at a position and refuses a label it already holds, as pandas does, and a column has item, to_list and is_unique, where two missing values count as the same value. firepanda.isna, notna, isnull and notnull answer for a scalar, a column or a frame. to_dict takes each of pandas' orients for a frame and a mapping class for into, and each signature matches pandas' parameter for parameter. UndefinedVariableError is now built from the name and whether it was the caller's, the way pandas builds it, so the compat signature check agrees.

Added: pandas.get_dummies

firepanda.get_dummies turns a column, a list, or the text and categorical columns of a frame into one column of flags per level, the way pandas does. The levels are the categories in order for a categorical column, unused ones included, and the distinct values sorted for anything else. prefix and prefix_sep take one value, a list with one per encoded column, or a mapping, and dummy_na, columns, drop_first and dtype all work, with pandas' messages for a prefix list of the wrong length, a columns that is not a list and an object dtype. sparse=True is refused, and so is a level that would label a column with something other than text, such as a number with no prefix, since a firepanda column label is text.

Added: a semi or anti join asks the rest of its condition, and TPC-H q21 runs

A semi or an anti join, and the correlated EXISTS and NOT EXISTS that become one, now take a condition with more than equalities in it, such as EXISTS (SELECT 1 FROM lineitem l2 WHERE l2.l_orderkey = l1.l_orderkey AND l2.l_suppkey <> l1.l_suppkey). The equalities pair the rows up and the rest is asked of each pairing, and a left row is kept when some pairing passes (semi) or none does (anti). The plan shows it as JOIN semi [a = k] where k > b and the JSON form writes it under "where". This is the last piece TPC-H q21 needed, so all 22 TPC-H queries now answer the same as DuckDB (#816).

Added: DataFrame.query and DataFrame.eval

query keeps the rows where an expression over the columns is True, and eval answers the value of one, or the frame with a column assigned for c = a + b. The expression is read with Python's own parser and pandas' three changes: & and | bind looser than a comparison, a name in backticks may hold spaces, and @name reads a variable of the caller, or of local_dict and global_dict. Chains such as 1 < a < 4, membership with in, not in and == against a list, the row labels as index or by their name, abs, and methods such as b.isna() and s.str.startswith('x') all work, each node worked out over whole columns. A name that is nothing the query can read raises the new firepanda.errors.UndefinedVariableError, a subclass of NameError, and the nodes pandas does not read, such as is and if else, raise NotImplementedError as pandas does. inplace works for both.

Added: DataFrame.resample, Series.resample and the resampler

resample bins a datetime index, or the column on= names, into fixed steps of time, from nanoseconds to days, with multiples and fractions such as 90min and 1.5h. It is the group by firepanda already has, keyed on the bin number of every timestamp, with every bin from the first to the last in the answer. An empty bin holds what pandas puts there: 0 for sum, count, size and nunique, 1 for prod, and NaN for everything else, which widens an integer column to float64. closed, label and the origins start_day, start and epoch place the bins the way pandas places them, and a rule in days always starts at midnight of the first day. The resampler has every reduction the group by has plus ohlc on a series, agg and apply by name or by a mapping of columns to names, transform by name, pipe, asfreq, get_group, a column by key or by attribute, and groups, indices, ngroups, ndim, ax, binner and obj. Calendar rules such as weeks and months, offset=, the end origins, zoned timestamps, functions, lists of names and upsampling are refused by name. python/tests/test_resample.py checks all of this against pandas.

Fixed: a group fill leaves NaN, not a null, in a float column

A row that groupby(...).ffill() or bfill() could not fill, because its key is missing or its group has no value yet, is NaN in a float column now, which is what pandas holds there. Before it was a null, which reads the same through isna but is a different value in the column.

Added: DataFrame.melt and pandas.melt

melt turns a frame long. The id columns repeat once for each value column, a new column holds the name of the column each row came from, and another holds the values, stacked one column after another. It picks columns by default the way pandas does, takes var_name from the column labels' own name, and keeps the row labels with ignore_index=False, repeated labels included. The stacked values widen the way concat widens them. A mix that pandas answers with an object column is refused, and that includes a bool column next to a number column, which pandas' melt turns into object even though its concat of frames does not. A missing column raises pandas' KeyError, and a value_name that names an existing column raises its ValueError. python/tests/test_melt.py checks all of this against pandas.

Fixed: kurt over a column with an infinity answers NaN

An infinity makes the mean NaN and every moment NaN with it, and the sums skipped those NaNs, so kurt answered zero where pandas answers NaN. The moments are now summed without skipping, and python/tests/test_describe.py checks two columns with infinities against pandas.

Added: group ffill, bfill, pct_change and filter

groupby(...).ffill() and bfill() fill each missing value from the last or next value in its group, and limit stops after that many missing rows in a row. A row whose key is missing answers missing, and an integer column that gets a missing value widens to float64, as it does in pandas. pct_change divides each value by the group's own shift and subtracts one, with no fill first, as pandas 3 does, and refuses freq. filter calls the function once per group on the group's rows and keeps the rows of the groups it answers True for. On a frame, an answer that is not one flag raises pandas' TypeError, and on a column any truthy value keeps the group. dropna=False is refused. All four carry repeated row labels through unchanged, and python/tests/test_group_fills.py checks them against pandas.

Changed: text is hashed sixteen bytes a step in two lanes

hash_bytes, which every text key goes through on its way into a group by, a join, a factorize or an isin, used to take a word of eight bytes at a time through the whole splitmix finalizer, so a key was one long chain of multiplies. It now takes sixteen bytes a step in two lanes of xxHash64's round, reads the bytes left over as overlapping words rather than one at a time, and finishes with the same mix. A 90 byte URL hashes in 5 ns rather than 20 and a short key in 1.5 ns rather than 2.2, with the same spread over both the low and the high bits.

Added: describe and kurt on a series and a frame

Series.describe and DataFrame.describe answer the count, mean, standard deviation, smallest value, percentiles and largest value of a column of numbers as float64, labelled the way pandas labels them, with percentiles checked, sorted and printed by pandas' rules, so 0.333333 next to 0.25 is 33.3%. A frame describes its columns of numbers and leaves the rest out, as pandas does by default. Text, flags, moments and spans are refused by name, because pandas describes them with a column of mixed values that only an object column holds. kurt and kurtosis answer the unbiased excess kurtosis with pandas' own sum and its guard for a column that is constant up to rounding, and a column they cannot read raises pandas' error for its type. python/tests/test_describe.py checks 59 cases against pandas.