Skip to content

v0.1.3

Choose a tag to compare

@randyzwitch randyzwitch released this 20 Sep 23:46
· 846 commits to main since this release
2c79432

Parallel grouping, join and sort, plus two timestamp fixes that matter more than any of the speedups for common data.

[workspace]
preview = ["pixi-build"]

[dependencies]
mojo = ">=1.1.0,<1.2"
dataframe_mojo = { git = "https://github.com/randyzwitch/dataframe_mojo.git", tag = "v0.1.3" }

Fixed

  • Datetimes accept an ISO 8601 zone designator: a trailing Z means UTC, and +HH:MM, -HH:MM, +HHMM and +HH convert to UTC. Both previously raised, which rejected the most common datetime spelling in real data (#107).
  • Format directives with no separator between them take their exact width, so %Y%m%d reads 20240228 instead of letting %Y consume six digits and fail (#107).

Changed

At 1,000,000 rows on a 32-core Threadripper, against v0.1.2:

workload v0.1.2 v0.1.3
grouping, ~100k groups 51.7 ms 24.5 ms
join, inner 283 ms 179 ms
sort, two keys 594 ms 345 ms
CSV read 1,260 ms 1,081 ms
  • Grouping partitions rows by key hash so no dictionaries or reduction states are ever merged, which is what made earlier attempts lose at high cardinality (#104).
  • Joins encode both sides' keys one hash bucket at a time instead of building one dictionary over everything (#105).
  • sort, arg_sort, top_k and bottom_k sort one row range per worker and merge the runs, in the same stable order as before for any worker count (#108).
  • CSV parses plain decimal floats in one exact pass rather than validating and then re-parsing (#107).

Added

  • pixi run -e oracle bench-polars: a head-to-head benchmark against Polars on identical inputs with thread counts pinned equal, which produced every number above (#102).

Targets Mojo 1.1. A compiled Mojo package only loads in the Mojo version that produced it, and pixi builds a source dependency with the newest Mojo that [package.host-dependencies] allows rather than the consumer's pin, so each tag targets one Mojo version.