v0.17.0

Files keep what they declare

Saving a loaded file used to drop or change several things it declares. Each of these is now read, written back and settable, so masters and archive files come back as they were.

• GZIP compression level: CDF.compression_level, Variable.compression_level, and add_variable(compression_level=...). CDFpp always wrote 9 while compressing at 6.
• Sparse records mode: Variable.sparse_records (pycdfpp.SparseRecords).
• Pad value: Variable.pad_value. A resaved master got the default pad value of its type instead (#38).
• Majority and byte order: CDF.majority and CDF.encoding (pycdfpp.Encoding). Files can be written column major or big endian, and a loaded file keeps its own. They all came back row major, in the host byte order.
• MD5 checksum: CDF.checksum (pycdfpp.Checksum). It is computed while the file is written. It isn't checked when loading.
• Declared variable attributes, in their order: CDF.declared_variable_attributes. Attributes no variable uses were dropped, and the others came back in another order.

Right dates when writing datetime64

• NaT was stored as a real date: 2262-04-11 as TT2000, 1677-09-21 as CDF_EPOCH. It is now stored as the fill value, which reads back as NaT.
• Dates before 1970 read back as NaT as CDF_EPOCH16, and one millisecond late as CDF_EPOCH.
• Dates before 1707, which TT2000 can't hold, overflowed. They become its illegal value, as with NASA's library.

Fixes

• HP encoded files are read as big endian: their values had their bytes swapped.
• Column major string arrays of two dimensions or more, in several records, read with the strings of different records mixed.
• CDF.add_attribute("A", []) raised an error. Masters declare such attributes.
• A loop over variables or attributes sees the collection as it was when it started. Adding a variable during the loop could leave it reading freed memory.

Faster

Measured on an AMD Ryzen 7 5800X (AVX2), btrfs on NVMe.

• Saving over an existing file: 40 → 8 ms for 83 MB. btrfs and ext4 flushed the file to disk when it was closed.
• Column major files with multi-dimensional records: an MMS FPI burst distribution file saves in 32 ms instead of 75, and loads in 58 ms instead of 81.
• Files with an MD5 checksum: MD5 runs at 970 MB/s instead of 660, on another thread while the file is written.
• Big-endian files, and files with much metadata (THEMIS ESA, 383 variables: 18.7 → 10.8 ms).
• datetime64 time axes are converted to TT2000 with SIMD and threads: 3 to 6 times faster for dates before 2017.
• Loading eagerly, or adding variables, no longer copies the values of every variable already there each time the CDF grows: eager loads of MMS FGM, MMS FPI and Wind MFI files take 40 to 56% less time, on every platform.
• save(cdf)'s result can be written, viewed or loaded again without bytes(), which copies it, and has a length.

CDFpp Explorer

• Its WebAssembly module uses WebAssembly SIMD, which every browser has since 2023 (Safari 16.4): gzip saves are 10 to 17% faster, and big-endian files load up to a third faster. The Pyodide wheels don't use it yet.

Faster on Apple Silicon

Measured on a MacBook Air M2. pycdfpp is now faster than spacepy and cdflib on every task of the comparison there, writing without compression included.

• Time conversions use NEON on every ARM CPU: TT2000 both ways, CDF_EPOCH and EPOCH16.
• Files are written with write() on macOS, whose std::fstream copies big writes through a small buffer: the uncompressed MMS FGM file saves in 10.1 ms instead of 29.1.
• Loops over attributes and variables no longer end by throwing a C++ exception, which is slow on macOS: opening the MMS FPI file and reading all its attributes takes 0.5 ms instead of 3.4.

Documentation

• New page, Under the hood: how reading, writing and time conversions are made fast, on x86, on Apple Silicon and in the browser, with the measurements behind each choice.