v0.8.31

Built against Mojo 1.0.0 (ed45d567).

A patch release. read_parquet now holds a string column that repeats as int32 codes into its distinct values by default, with dtype still string, so TPC-H sf1 lineitem reads into 813 MB instead of 1150 in the same time, which is less than DuckDB's 1138 MB for the same table. To get there, sorts, Arrow export, the IPC writer, every per row Series.chars_* method and the rest of the string kernels learned the encoding: the text methods run once per distinct value, and concat, cast, the fills, pick, the text reductions, joins, grouped aggregation and the per cell readers decode first. encode_strings=False on Session.run gives the old flat columns. A group by on a text column now inserts its new keys on every core. On the pandas side, pd.merge and DataFrame.merge answer a merge on key columns the way pandas does. SQL reads a NEAREST join and a slice whose end is written as a minus.

Changed: read_parquet holds a string column that repeats as codes by default

Session.run(encode_strings=...) now defaults to on, so read_parquet and everything built on it hand back a string column where each value repeats at least sixteen times on average as int32 codes into its distinct values (#979). The dtype still says string, and every frame and Series method reads the encoding or decodes first, so nothing a caller does changes except memory and speed. On TPC-H sf1 lineitem the frame is 813 MB rather than 1150 MB, and the read takes the same time. encode_strings=False gives every column flat, for a caller that reads the buffers directly.

Added: merge and DataFrame.merge on key columns

The core pairs the rows through a new join_on binding and the Python layer does the rest the way pandas does it. on, left_on and right_on work, as does a merge on the shared columns when no key is named. A suffix goes on both sides of an overlapping column, an int key against a float key becomes a float64 key, and a missing key matches a missing key. Inner and left joins keep the left order, a right join keeps the right order, and an outer join is sorted on the key, all as pandas 3 does. sort, suffixes, validate and a named series on either side work, and the mistakes raise the pandas class with the pandas message, with MergeError added to firepanda.errors. A cross or asof join, left_index, right_index and indicator are refused by name for now.

Added

• A NEAREST join, such as JOIN products t APPROX NEAREST 2 BY SIMILARITY f(q.e, t.e), now parses and prints back, with or without an alias on the right side and with the join type, APPROX or EXACT and the count all optional. The plan still refuses it as a kind of join firepanda does not run.

• A slice whose end is written as a minus, such as a[1:-:2] or a[:-:-1], which is how DuckDB says the end of the list when a step follows. It now parses and prints back as written, and the plan refuses it the way it refuses any other step. a[1:-] with no second colon is a syntax error, as it is in DuckDB.

Added: the text methods and the rest of the string kernels read a column held as codes

Every Series.chars_* method that answers a row from its value alone (case, strip, pad, slice, find, replace, the regex and folded matches, the is_* tests, translate and the rest) now runs once per distinct string of a column held as codes (#979), and the answer is spread over the rows with through_codes. On six million rows of seven values that is seven calls rather than six million. through_codes takes a string answer too, and hands it back flat, because two categories can map to one answer ("a" and "A" upper cased) and codes into a list with a repeat would break a group by on the codes. The methods that are not a row at a time (chars_extract_regex, chars_partition, chars_dummies, chars_join, to_datetime) decode first.

Concat, cast, coalesce, forward and backward fill, pick, a text reduction (min, max, any, all), a grouped aggregation of an encoded value column, a join on an encoded key, a column against column comparison and the lasting hash now decode such a column rather than raising. A reduction does not work over the categories, because a filter keeps them all and the smallest may be a value no row still holds. distinct_count_any counts the codes the rows still use and never compares a string. AnyArray.text_at(i) reads one row however it is held, and the CSV writer, the display, the index label compare, python_list and the Parquet DESCRIBE read use it.

Added: a sort and an Arrow export read a string column held as codes

A sort on a column held as codes (#979) sorts its few distinct strings once and then radix sorts the rows on each one's rank, an int32, rather than comparing strings row by row. argsort_any, argsort_multi, is_sorted_any, DataFrame.argsort, sort_values and the bounded argsort_limit all go through it, and the order is the one the decoded column gives, ties and nulls included. export_array, export_array_borrowed and the IPC writers hand such a column out as the strings it stands for, since a consumer handed an Arrow dictionary would call it a category. DataFrame.is_flat() and DataFrame.decoded() are new, for a writer that reads buffers as they lie.

Changed: grouping by a text column inserts the new keys on every core

LastingText, the table that gives each distinct string of a text key its group number across the chunks of a scan, did all its inserts on one thread, and on ClickBench q33 that serial insert was most of the main thread's time. It now splits the table into 32 parts by the top bits of the hash. Each chunk hashes its rows in parallel, sorts the row numbers into their parts, and walks the parts on all cores, so a row either finds its string or claims a slot. A chunk that a filter left small takes one worker per 8K rows rather than one per part. Rows that claim one are then numbered in row order, so new groups still get consecutive numbers in the order their first row appears and nothing downstream sees a difference. The distinct strings are kept as one piece per chunk instead of being appended to a single column, and are stacked once at the end. On the URL column of the first million ClickBench rows, split into 128K row chunks, the best of 30 runs goes from 73.6 ms to 58.6 ms. On the planner route over the million rows, q33 goes from 109 ms to 74 ms at best and q18 from 127 ms to 96 ms, and q13, q14, q16, q17 and q39 do not get slower. All 43 queries give the answers they gave before (#997).