Child metadata scattered throughout file causes excessive S3 range requests for partial reads

[nephdev]

Problem

When reading individual variables from a .om file stored on S3 (or any HTTP range-request backend) via OmFileReader.from_fsspec(), each get_child_by_index(i) call triggers a separate range request because each child's variable message is stored immediately after that child's compressed data — scattered throughout the file.

Even reading a single pixel from a handful of variables requires many small metadata fetches first. For a file with N children, the reader must make ~2N small range requests (one for each child's variable message, one for each LUT entry) just to discover where the data chunks live, before it can fetch the actual data. Each of these reads is tiny (77–131 bytes) but at unpredictable offsets spread across the entire file.

A pre-fetch of the file's tail region (e.g., last 2 MB) captures the root group message and trailer, but not the per-child metadata, because that metadata is interleaved with data chunks throughout the file body.

On S3, each range request has ~400ms of latency overhead, so these scattered metadata reads dominate total query time — turning what should be a fast point lookup into a multi-second operation.

Root Cause

The .om file layout stores each child's metadata right after that child's compressed data chunks:

[child 0 data chunks] [child 0 variable msg] [child 0 LUT entries]
[child 1 data chunks] [child 1 variable msg] [child 1 LUT entries]
...
[child N data chunks] [child N variable msg] [child N LUT entries]
[root group message]
[trailer]

For local file access this layout is fine — the OS page cache handles it transparently. But for S3/HTTP range-request access, every metadata read at a new file offset requires a separate HTTP round trip.

Suggested Fix

Consolidate all child metadata into a contiguous region at the end of the file, before the root group message and trailer:

[child 0 data chunks]
[child 1 data chunks]
...
[child N data chunks]
[child 0 variable msg] [child 0 LUT entries]   <- all metadata grouped together
[child 1 variable msg] [child 1 LUT entries]
...
[child N variable msg] [child N LUT entries]
[root group message]
[trailer]

This way, a single tail-region read would capture all child metadata + root + trailer. Any subsequent variable read would only need requests for the actual compressed data chunks — the metadata is already cached.

Alternatively, an option in OmFileWriter to control metadata placement would work. The current "inline after data" layout is fine for local access and could remain the default, with an opt-in "consolidated metadata" mode for files intended to be served over HTTP/S3.

Environment

• omfiles Python package version 1.1.0
• Reading via OmFileReader.from_fsspec() with an S3 range-request backend

[terraputix]

Hi,
some follow up questions from my side:

1. Can you point to me to the file that you are experiencing these problems with?
2. Are you using fsspec in combination with the blockcache as done in the example scripts:

   python-omfiles/examples/plot_map.py

   Line 35 in b8978dd

   f"blockcache::{S3_URI}",

   ?
3. Did you try deactivating the check_files property of the fsspec cache: https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.implementations.cached.CachingFileSystem ?

While it is true that LUT entries might not currently be consolidated at the end of the file (depends on how the file was written), the single variable messages should be. Therefore get_child_by_index() and get_child_by_name() should normally be fast operations and I assume the problem is that the correct usage is not documented well enough.

[nephdev]

Thanks for the follow-up questions, they helped me narrow down the actual issue.

The file:
I'm reading .om files that I write myself using OmFileWriter from the Python bindings — processed weather data on S3, dozens to hundreds of children per file. I serve them for point queries via a custom fsspec-compatible filesystem on top of boto3 (direct range requests, no blockcache).

When I profiled it, child variable messages start near the beginning and are interleaved with data chunks across the entire file body. A 2 MB tail fetch covers almost nothing. I built a companion .om.idx file with pre-extracted metadata byte ranges as a workaround.

After your reply I checked the OpenMeteo source files on S3 (e.g. s3://openmeteo/data_spatial/dwd_icon_eu/...) and you're right — on those files all child variable messages and LUTs sit in the last ~150 KB. Consolidated, exactly as you described. So the difference is between files written by the Swift server vs files written by the Python OmFileWriter.

I traced it to how the Python bindings wrap the Rust writer:

The Rust library has a three-phase API: prepare_array() → write_data() → finalize(), and then separately write_array() to emit the variable message. The Swift server uses this to write all data first, then all variable messages at the end.

But the Python write_array() bundles all three phases into a single call. So when I write many variables sequentially, each variable's metadata gets written right after its data:

[var1 data] [var1 LUT] [var1 variable msg]
[var2 data] [var2 LUT] [var2 variable msg]
...
[root] [trailer]

vs what the Swift server produces:

[var1 data] [var2 data] ...
[var1 LUT] [var2 LUT] ...
[var1 variable msg] [var2 variable msg] ...
[root] [trailer]

I confirmed by writing a small test file with OmFileWriter from Python — child metadata at 10%, 36%, 63% of the file.

To answer your questions directly:

1. blockcache — No, I'm using a custom boto3 backend. Even with blockcache it wouldn't help for my files since metadata is spread across the whole file, not concentrated in a few blocks.

2. check_files — Not applicable, not using CachingFileSystem.

3. Variable messages should be consolidated — They are in files written by the Swift server, but not in files written by the Python OmFileWriter. The underlying Rust library supports both patterns, but the Python bindings only expose the bundled single-call API.

Would it be possible to either:

• Expose prepare_array() / write_data() / finalize() to Python so users can control the write order, or
• Have write_array() defer writing the variable message until close() is called?

Either approach would let Python-written files have the same consolidated layout as what the Swift server produces.