Skip to content

v0.6.0

Choose a tag to compare

View all tags
@mousebrains mousebrains released this 25 Mar 22:52
· 8 commits to main since this release
v0.6.0
1ea1133

What's New

Bulk parser for 10-20x faster multi-file loading

New loadQfiles() function uses numpy structured dtypes to parse all records per file in a single C-level call, then builds one xr.Dataset at the end — eliminating per-file xarray overhead and per-record struct.unpack calls.

Benchmarks (773 Q-files, 101K records):

v0.5.1 v0.6.0 Speedup
Load + merge 3.5s 0.27s 13x
Write 5.8s 1.6s 3.6x
Total 9.3s 2.1s 4.4x

Handles mixed v1.2/v1.3 files, heterogeneous channels/spectra/config keys, varying frequency dimensions, multi-segment files, and empty files.

Bug fix

  • mergeDatasets() no longer crashes when freq coordinate is missing from some datasets

Other changes

  • Default CLI compression lowered from level 5 to 1 (15 MB vs 9 MB, but 2x faster write)
  • loadQfile, loadQfiles, mergeDatasets now exported from q2netcdf package
Assets 4
Loading