v0.8.36

Built against Mojo 1.0.0 (ed45d567).

A patch release. On the pandas side, df[key] = value writes columns into a frame, groupby(...).sample and json_normalize arrive, an index, a frame and a column print what pandas prints, NaT and missing values read out the way pandas spells them, and to_datetime takes dayfirst, yearfirst, the separators ISO8601 allows, numbers among moments and moments mixed with text. The ClickBench group by got faster again: the text walk prefetches the row a pending slot names, and a tuple of keys is read back from the map once at the end rather than gathered from every chunk, which together take about a tenth off q34, a quarter off q39 and more than a quarter off q18 at 1M rows.

Added: Writing a column into a frame with df[key] = value

DataFrame.__setitem__ and DataFrame.__delitem__ now exist. A name puts one column, lined up on the row labels when it is a series, a list of names puts several from a frame, from rows or from one value, and a column or frame of true and false puts the value in the rows or cells it marks. The write rebinds the frame's one slot to what assign or mask would answer, as inplace does, so a copy or a column taken before the write keeps what it had, which is pandas' copy on write. None and a function are refused, since pandas holds them in an object column.

Added: Exact kurtosis of whole numbers far from zero

Series.kurt and DataFrame.kurt on a whole number column now measure the values from the column's least value as whole numbers before the cast to float, as skew already does. On values near two to the sixty two the cast alone moved each value by up to 512 and the kurtosis in its seventh digit. The answer is now within a few bits of exact arithmetic, and within the suite's tolerance of pandas.

Added: Names and text read from a DatetimeIndex with a gap

day_name, month_name and strftime on a DatetimeIndex with a gap raised since the gap handling for whole number parts, because the text went through the float path. They now answer the text with the gap shown as NaN, as pandas does. time, timetz and to_pydatetime hand out NaT for a gap where they handed out None.

Added: Numbers among moments in to_datetime, and text among spans in to_timedelta

to_datetime of a list mixing a Timestamp, datetime or date with a number now reads the number as pandas does, as nanoseconds since 1970, and holds the column in nanoseconds. A NaN beside a moment is a gap, a flag raises pandas' TypeError, and a zoned moment beside a number raises pandas' mixed timezones ValueError unless utc is set. to_timedelta of text mixed with spans now answers microseconds, or the finer unit of the spans, where it answered nanoseconds, and keeps nanoseconds when a number is mixed in.

Added: to_datetime of moments mixed with text

to_datetime of a list mixing Timestamp, datetime or date values with text now reads as pandas reads it, taking each moment as it is and reading the text with the format worked out from the first piece of text, where it refused the list as a column of mixed kinds. format, dayfirst, errors="coerce" and utc apply to the text as before, and under utc the moments are put on UTC too.

Added: Instants and spans among objects in a frame's values

When a frame's values or to_numpy() is an object array, an instant or a span in it is now a Timestamp or a Timedelta and a gap is NaT, as pandas gives them, where numpy had put a datetime, a timedelta and None.

Added: Instants and spans in different units meet as in pandas

concat of instant or span columns, or of instant row labels, in different units now keeps the finest unit, as pandas does, where it refused them as pandas' object column. A mix of zones, or of instants and spans, is still refused, since pandas answers an object column there. astype to the same kind in another unit, such as astype("datetime64[ms]"), is now as_unit on a series and on a frame, and dropping a zone that way raises pandas' TypeError.

Added: Calendar parts of a column with a gap, typed as in pandas

A whole number read through dt or a DatetimeIndex from instants or spans with a gap, such as year, hour, dayofweek or days, is now float64 with NaN in the gap, as pandas gives it, where it was int32 with a gap. The yes or no fields, such as is_month_start and is_leap_year, answer False for a gap as pandas does, where they answered missing. Without a gap nothing changes.

Added: NaT, the missing moment and span

firepanda.NaT is pandas' missing moment and span: one value, a datetime that never equals anything, sorts against nothing and turns the arithmetic it meets into itself, with its fields NaN and the methods pandas refuses refused with pandas' words. A gap in a column of instants or spans now reads out as NaT through tolist, iteration, iat, a frame cell and an index, Timestamp(None), Timedelta(None), to_datetime(None) and to_timedelta(None) answer it, and a reduction over only gaps answers it where it answered NaN. NaT goes back in wherever pandas takes it, so a column built from it, fillna, where, replace, isin, get_loc, in and adding it to a span column all match pandas. Replacing a moment or a span, which never matched a row before, now does.

Added: Read a missing value out the way pandas spells it

A gap read out of a column through tolist, iteration, iloc, iat, loc, at or items now comes back as pandas hands it out for the column's type. A float column, an integer column with a gap, a text column and a category column hand out nan, and a column of flags hands out None, as an object column does in pandas. A frame cell and an index read the same way, a category's missing row has a code of -1, and a NaN given back among text or asked for by get_indexer finds the gap. A missing instant or span still reads as None, since firepanda has no NaT yet.

Changed: a tuple of keys is read back from the map at the end

A group by on several keys with text among them writes each row's keys out as bytes and looks those up in the text map, which already stores every distinct tuple. Each chunk also gathered its new groups' keys out of its own columns, a second copy of the same values, and then found its new groups again with a serial pass over the ordinals. The text map now hands back the rows it already knew were new, and the key columns are read out of the stored tuples once, on the cores, when the operator finishes. On a six core Linux machine, alternating the old and new driver over eight rounds, ClickBench q18 went from 106 to 84 ms at best, and every answer matched.

Added: groupby sample

groupby(...).sample draws rows at random from each group, the same rows pandas draws for the same seed. pandas walks the groups in order and asks numpy's choice for each one's positions from one random state, and this walks them the same way, so n, frac, replace, weights as a list or a column name, sort=False and dropna=False all give pandas' rows in pandas' order, and a size or weights pandas refuses are refused with its words. The size and weight checks moved out of DataFrame.sample into two helpers both now share. 11 calls and 7 mistakes match pandas.

Added: json_normalize

firepanda.json_normalize flattens nested records into a frame the way pandas.json_normalize does. It is a port of pandas' own code: nested dicts become dotted columns, sep and max_level change how, record_path pulls the rows out of lists nested in each object, meta repeats outer fields on every row, the two prefixes rename columns, and errors='ignore' fills a missing field with NaN. The error messages are pandas' words. A cell left holding a dict or a list needs a column of objects, which firepanda does not have, so that case still raises. 14 calls and 6 mistakes match pandas.

Added: An index prints what pandas prints

repr and str of an Index, a DatetimeIndex and a TimedeltaIndex used to print the core's summary, so a zoned index printed the counts behind its instants. They now go through a port of pandas' format_object_summary: the labels in brackets, wrapped at display.width, cut to ten from each end past display.max_seq_items with dots between, then the type, the name, the length when cut, and freq for instants and spans. Labels that are still a range print as a RangeIndex. Dates at midnight print as dates, whole days as days, text is quoted with its tabs and newlines escaped, and a gap prints as pandas prints it for each kind. 24 shapes and 16 option combinations match pandas.

Changed: the text walk prefetches the row a pending slot names

A chunk that brings most of its own keys compares most of its rows against the row of the same chunk that first opened the key's slot, not against a stored key, and that comparison was the one the walk did not prefetch. The walk now asks for that row's view and then its bytes ahead of time, as it does for a stored key, and compares the two rows by their view words without the validity checks a key column with no nulls does not need. On a six core Linux machine that was busy with other work, alternating the old and new driver over eight rounds, ClickBench q18 went from 108 to 96 ms at best, q34 from 88 to 79 and q39 from 169 to 131, and every answer matched.

Added: A frame and a column print what pandas prints

repr and str of a DataFrame printed a summary of its columns, and a column of instants with a zone printed their counts. Both now go through the text formatter under pandas' display options, so a frame prints its rows, a long one loses its middle to dots, a wide one is cut to fit the terminal or wrapped at display.width, and a zoned instant prints as a time with its offset. to_string takes max_rows, min_rows, max_cols and line_width instead of refusing them, and cuts and wraps the way pandas does.

Added: format="ISO8601" reads the separators pandas reads

to_datetime with format="ISO8601" now reads a date split by a slash, a dot, a space or a backslash, such as 2024/01/02 10:00, as long as both splits are the same, which is what pandas' own ISO 8601 reader accepts. It also refuses a comma before the fraction, as pandas does under ISO8601, while format="mixed" still reads one.

Added: dayfirst and yearfirst in to_datetime and DatetimeIndex

to_datetime and DatetimeIndex used to refuse dayfirst and yearfirst. They now read the way pandas does, which hands both to dateutil. dayfirst reaches the guesser too, so 01/02/2024 is the first of February and, as in pandas, so is 2024-01-02. yearfirst changes rows read on their own, such as 10/11/12 with no format to guess. An explicit format and ISO8601 ignore both. The same work fixed two things found on the way: a row with no year, such as 01/02 or Jan 2, is now in the year 1 as in pandas rather than in the current year, and str of a Timestamp before the year 1000 pads the year to four digits, as pandas does, while its repr still does not.