Skip to content

v2.1.0

Choose a tag to compare

@xmival00 xmival00 released this 31 Oct 01:41
· 97 commits to main since this release

mef3io 2.1.0

Released 2026-10-31

A minor release that makes writing many channels much faster. Nothing that
worked in 2.0 changes.

What's new

  • Write many channels in one call. Writer.write_int32_many(channels, data, ufact, start_uutc, fs) (integer counts) and Writer.write_many(...)
    (float data, NaN gaps, precision decided per channel) write the same stretch
    of time for all channels at once. The files are the same as a loop of
    write_int32 / write calls would produce (a fresh segment differs only
    in its random UUIDs). The channels are encoded and written in parallel on
    the Writer's n_threads, and the disk is waited for once per call
    instead of once per channel. Also in C++ (SessionWriter::write_int32_many,
    write_float_many), the C ABI (mef3io_writer_write_int32_many) and
    MATLAB (Writer.writeInt32Many, integer counts only).
    Speed: on synthetic data (64 channels of 10 minutes at 2048 Hz, appended
    in 4 rounds, 12 threads, one SSD, one 12-core machine) a loop of
    write_int32 took 5.2 s and write_int32_many 0.76 s. Your gain depends on
    your disk and channel count; it is largest with many channels and a disk
    where waiting for a sync is slow. Numbers and the setup are in the
    benchmarks.
  • Writer(flush="close") (opt-in). No wait for the disk during writes;
    close() or flush() does one combined wait. If the power is cut while
    writing, the whole session can become unreadable, and recover_session
    cannot be relied on to fix it.
    Use it only for output you rebuild after
    any failure. The default, flush="each", is unchanged. It is refused
    together with wait_for_disk=True and on a session that already existed
    when the Writer opened, so earlier recordings are never put at risk.
    Batching with write_int32_many already removes most of the waiting; in the
    same synthetic test flush="close" saved only about 10 % more on top of it.
    See When is my data on disk?.

Failures in a multi-channel write

Every refusal (a duplicate channel, a different rate or conversion factor
than what is stored, data starting before the stored end) is decided before
any file is touched. A failure before the metadata files are published rolls
every channel back. A failure while they are being published raises
PartialWriteError, whose written_channels and incomplete_channels say
what was written. If a roll-back itself fails, RollbackIncompleteError names
the channels whose files may be torn, never a silent "nothing written": run
validate and recover on them.

Changed

  • Writing a fresh segment copies the encoded blocks once less.

Upgrading from 2.0

Nothing is required: no signature, default or file format changed, and the
new calls and option are additive. To get the speed-up, replace a loop of
write_int32 / write over channels with one write_int32_many /
write_many call, and write long runs in chunks of minutes (each call holds
its channels' data and encoded blocks in memory).