remfile-cpp: a read-only VFD for streaming remote HDF5 files over HTTP

[bendichter]

We work with NWB neurophysiology data, which is HDF5, and a lot of it now lives in cloud archives like DANDI. Files are routinely hundreds of GB, so downloading them to read a small piece is not practical. In Python we use remfile for this, but we needed the same thing in C++, so I wrote one as a VFD.

Sharing it here in case it's useful, and because I have a question at the end for people who know the VFL better than I do.

Repo: https://github.com/catalystneuro/remfile-cpp
Write-up: https://catalystneuro.com/blog/remfile-cpp-streaming-hdf5-from-cpp

What it does

It answers HDF5's reads with HTTP range requests through libcurl, and keeps an in-memory chunk cache with read-ahead. It's a C++ port of Jeremy Magland's Python remfile, and the caching strategy is his.

hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
H5Pset_fapl_remfile(fapl, NULL);
hid_t file = H5Fopen("https://example.com/file.h5", H5F_ACC_RDONLY, fapl);

It also builds as a dynamic plugin, so you can load it with HDF5_PLUGIN_PATH and H5Pset_driver_by_name(fapl, "remfile", NULL) without recompiling anything.

It only needs libhdf5 and libcurl. CI runs against HDF5 1.10, 1.12, 1.14, and 2.1 on Linux and macOS.

Comparison with ROS3

ROS3 already does remote reads, so the fair question is why write another one.

Reading a 23 MB slice from a DANDI file, alternating drivers, means of 6 runs:

Phase	ROS3	remfile-cpp
open (H5Fopen)	3.26 s	0.33 s
metadata	0.09 s	0.00 s
bulk read (23 MB)	1.06 s	2.11 s
total	4.40 s	2.44 s

Almost all of the difference is in H5Fopen. Opening one of these files is hundreds of small scattered reads (superblock, B-trees, object headers), and without a cache each one costs a round trip. ROS3 is faster on the bulk read, because remfile over-fetches about 24% on a single contiguous read. For our workload that trade is worth it, since opening the file is where the time actually goes.

Two other differences that matter to us in practice:

• ROS3 is a compile-time option and is often just not there. It's missing from Homebrew's HDF5, and it wasn't in any conda h5py on my machine. I had to build a separate environment to benchmark it. remfile only needs libcurl.
• It uses plain HTTP rather than S3, so presigned URLs work. Presigned URLs reject HEAD, so it gets the file size from a one-byte range request and reads Content-Range.

A performance thing that might be worth knowing

The first version was 2.4x slower than ROS3 on bulk reads. I logged the actual HTTP requests to find out why.

The read-ahead only ever grew its window (100 KiB, then 170 KiB, then 290 KiB, and so on) and never looked at how much the current read was asking for. So a large H5Dread did not issue one large request. It ramped up to it, which meant 11 sequential round trips for a 23 MB read.

The fix was to never fetch less than the current read needs, and keep the read-ahead on top of that. It went from 11 requests to 5, and bulk reads are now about the same speed as ROS3, while H5Fopen is still ~10x faster.

I'm mentioning it because the cache was tuned for the metadata pattern and it made the streaming pattern worse, and I only noticed by counting requests. Timing it would not have told me.

Testing

Testing a network-backed VFD is awkward, so the tests run an HTTP server inside the test process and serve a generated HDF5 file to it. No network, ~2 seconds. The server counts requests and records byte ranges, so the tests check things like "a cache hit issues zero requests" and "a large read is coalesced into at most two range requests", instead of checking timing. Everything is also compared against the same file read with sec2.

That's what found the round-trip problem above, plus two retry bugs.

Question

I left read_vector NULL and let HDF5 fall back to read. Is it worth implementing? It looks like it could reduce round trips on chunked datasets, where HDF5 knows up front that it wants several separated pieces, but I don't have a good sense of when HDF5 actually calls it or how much it would help. If someone knows, I'd appreciate the pointer.

[jhendersonHDF]

Hi @bendichter,

first of all thanks for sharing this! It's always interesting to see what people build for use in the HDF5 ecosystem. It's also great timing as 1325d30 was just merged today to fix some of the issues you've seen with the ROS3 VFD.

The ROS3 VFD initially used only libcurl, but we found that the complexity of dealing with return values from S3 caused significant usability issues with the VFD, so we switched its backend over to https://github.com/awslabs/aws-c-s3 instead. The ROS3 VFD also previously cached several of the initial MiB of the file on file open, which is likely where you were seeing the overhead. After the commit I referenced previously, the VFD caches those bytes on the first read instead. The VFD also now has an internal LRU I/O block cache performing similar duties as your VFD, though likely in a much simpler fashion.

I think one important thing worth noting (as you did) is that, by default, HDF5 metadata can end up scattered across the file, resulting in poor access patterns for cloud storage. There are different ways to approach optimizing HDF5 files for cloud storage; I'd recommend giving https://www.youtube.com/watch?v=R5ok4fdYqBs a watch. Without taking steps to optimize HDF5 files for cloud storage, you end up paying for the poor file layout even with a very good caching strategy.

As for your question about the VFL, there are two main cases where read_vector() is called. The first is when a read_selection() call would be made, but the VFD doesn't support that callback, in which case it will be translated into a read_vector() call or series of calls to read() if read_vector() isn't implemented. read_selection() is called from several places, including I/O on chunked datasets, as long as the feature we refer to as 'selection I/O' is being used. For chunked datasets, selection I/O mostly gets disabled if:

• the feature was requested to be off
• the dataset uses data filters
• the library's "page buffering" feature is enabled
• the size of a dataset's chunks mean they're small enough to be cached in the chunk cache

Though this currently restricts the feature to a fairly small set of use cases, ideally the read_selection() callback should still be implemented if possible. Otherwise, implementing read_vector() is a good alternative to read_selection(). The other main case where read_vector() is used is during parallel I/O when a read for multiple filtered chunks is issued. It appears your VFD is intended for serial I/O, so you probably don't need to worry about the latter case.

[bendichter]

This is very helpful, thank you. If ROS3 was caching the first several MiB at open, that's almost certainly what I measured as open overhead, and I read it as missing caching rather than eager caching. My numbers describe the old behavior. I'd like to rebuild against develop with 1325d30 and rerun the same benchmark, and I'll post the updated table here.

[ajelenak]

it wasn't in any conda h5py on my machine

The hdf5 package from conda-forge comes with the ROS3 driver. How do you install h5py in a conda environment?

[bendichter]

You're right, thanks for the correction. I checked, and every environment where ROS3 was missing had h5py from PyPI, which bundles its own HDF5 built without the driver. The one env where h5py came from conda-forge does have ROS3. So I wrote "any conda h5py" when what I actually had was pip-installed h5py inside conda environments. My mistake.

Homebrew's HDF5 doesn't have ROS3, and neither do PyPI h5py wheels. conda-forge and Ubuntu's libhdf5-dev both do. I'll fix this in the blog post.

[bendichter]

Followed up on this. I rebuilt against develop with 1325d30 and the aws-c-s3 backend, and reran the same benchmark (23 MB slice, alternating drivers, medians of 8 to survive network noise):

Phase	ROS3 before (2.1)	ROS3 develop	remfile-cpp
open	3.26 s	1.51 s	0.35 s
read	1.06 s	1.58 s	1.20 s
total	4.40 s	3.08 s	1.56 s

The block cache change is clearly visible: ROS3 open dropped by roughly half. So the open-time gap I reported is now about 4x rather than 10x, and the read times are close enough that I wouldn't call a winner given the variance I'm seeing. remfile is still ahead end-to-end on this open-heavy workload, but the margin narrowed and will keep narrowing as the block cache matures. Thanks for the pointer, this was worth redoing.