Repository navigation
v0.1.3
Parallel grouping, join and sort, plus two timestamp fixes that matter more than any of the speedups for common data.
[workspace]
preview = ["pixi-build"]
[dependencies]
mojo = ">=1.1.0,<1.2"
dataframe_mojo = { git = "https://github.com/randyzwitch/dataframe_mojo.git", tag = "v0.1.3" }Fixed
- Datetimes accept an ISO 8601 zone designator: a trailing
Zmeans UTC, and+HH:MM,-HH:MM,+HHMMand+HHconvert to UTC. Both previously raised, which rejected the most common datetime spelling in real data (#107). - Format directives with no separator between them take their exact width, so
%Y%m%dreads20240228instead of letting%Yconsume six digits and fail (#107).
Changed
At 1,000,000 rows on a 32-core Threadripper, against v0.1.2:
| workload | v0.1.2 | v0.1.3 |
|---|---|---|
| grouping, ~100k groups | 51.7 ms | 24.5 ms |
| join, inner | 283 ms | 179 ms |
| sort, two keys | 594 ms | 345 ms |
| CSV read | 1,260 ms | 1,081 ms |
- Grouping partitions rows by key hash so no dictionaries or reduction states are ever merged, which is what made earlier attempts lose at high cardinality (#104).
- Joins encode both sides' keys one hash bucket at a time instead of building one dictionary over everything (#105).
sort,arg_sort,top_kandbottom_ksort one row range per worker and merge the runs, in the same stable order as before for any worker count (#108).- CSV parses plain decimal floats in one exact pass rather than validating and then re-parsing (#107).
Added
pixi run -e oracle bench-polars: a head-to-head benchmark against Polars on identical inputs with thread counts pinned equal, which produced every number above (#102).
Targets Mojo 1.1. A compiled Mojo package only loads in the Mojo version that produced it, and pixi builds a source dependency with the newest Mojo that [package.host-dependencies] allows rather than the consumer's pin, so each tag targets one Mojo version.