Skip to content

feature, add HDF5 S3 / ROS3 VFD support #188

@steven-varga

Description

@steven-varga

Problem

h5cpp has no bindings for the HDF5 Read-Only S3 Virtual File Driver (ROS3 VFD), despite the driver being compiled into every modern HDF5 distribution (H5_HAVE_ROS3_VFD). A user cannot open an S3-hosted HDF5 file through h5cpp today. Additionally, the existing fapl_rest_vol / kita alias in H5Pall.hpp wraps a pre-1.12 API (H5Pset_fapl_rest_vol) that no longer exists in HDF5 1.12's REST VOL.

Gap summary

Capability HDF5 1.12.x h5cpp today
Read-only S3 (ROS3 VFD) ✅ since 1.10.6 ❌ not wrapped
Temp credentials (session token) ✅ 1.12.1 v2 FAPL ❌ not wrapped
VOL connector API (H5Pset_vol) ✅ 1.12 ❌ absent
Read-write S3 (REST VOL + HSDS) ✅ via VOL ❌ stale pre-1.12 alias
Build-time ROS3 feature detection H5_HAVE_ROS3_VFD ❌ not in CMakeLists

Scope (this issue — ROS3 VFD only)

VOL connector / read-write S3 is a larger effort and is explicitly out of scope here. This issue covers the read-only S3 path.

H5Pall.hpp

  • Add fapl_ros3 property class guarded by #ifdef H5_HAVE_ROS3_VFD, wrapping H5Pset_fapl_ros3(hid_t, H5FD_ros3_fapl_t*).
  • Support both FAPL struct versions:
    • v1 (version, authenticate, aws_region, secret_id, secret_key) — HDF5 ≥ 1.10.6
    • v2 (adds session_token) — HDF5 ≥ 1.12.1; guard with H5_VERSION_GE(1,12,1)
  • Deprecate / remove the fapl_rest_vol / kita aliases (or reroute them to H5Pset_vol once the VOL issue is opened); they reference H5Pset_fapl_rest_vol which no longer links against HDF5 1.12.

H5config.hpp

  • Add #ifdef H5_HAVE_ROS3_VFD guard block; expose a h5cpp::have_ros3_vfd constexpr bool for runtime capability queries.

CMakeLists.txt

  • Add find_file(HDF5_ROS3_HEADER H5FDros3.h ...) detection.
  • Pass -DH5CPP_HAVE_ROS3_VFD when found so the guard in H5Pall.hpp can also be set from the build rather than relying solely on the HDF5 public config header.

Tests

  • Add test/ros3.cpp with a doctest suite:
    • Unauthenticated open of a public S3 bucket (authenticate=false)
    • Confirm H5Pget_fapl_ros3 round-trips the struct correctly
    • Skip cleanly when H5_HAVE_ROS3_VFD is not defined

Documentation

  • Add docs/s3.md with a minimal working example:
    #include <h5cpp/all>
    #include <H5FDros3.h>
    
    auto fapl = h5::fapl_ros3{ H5FD_CURR_ROS3_FAPL_T_VERSION, false, "", "", "" };
    auto fd   = h5::open("s3://my-bucket/data.h5", H5F_ACC_RDONLY, h5::fapd{fapl});

Acceptance criteria

  • fapl_ros3 compiles against HDF5 ≥ 1.10.6 with H5_HAVE_ROS3_VFD
  • v2 session-token path compiles under H5_VERSION_GE(1,12,1) guard
  • Stale fapl_rest_vol / kita alias removed or guarded
  • CMake detects ROS3 availability and sets the compile definition
  • ctest passes (ros3 tests skip gracefully when driver absent)
  • docs/s3.md documents public API and example

Out of scope

  • Read-write S3 via REST VOL + HSDS (H5Pset_vol) — separate issue
  • Credential file / env-var helpers — separate issue

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions