Skip to content

feature, implement threaded_pipeline_t using H5Qall ring-backed worker pool #197

@steven-varga

Description

@steven-varga

Motivation

threaded_pipeline_t in H5Zpipeline.hpp has been a do-nothing stub since the pipeline was introduced. H5Qall.hpp (#195) provides the correct substrate: a bounded ring to pass chunk descriptors between the split-to-chunk tiler, filter worker threads, and the HDF5 chunk writer.

Design

split_to_chunk_write ──► ring_in (spsc / mpsc)
                              │
                     N worker threads:
                     pop raw chunk → apply filter chain → push ring_out
                              │
                         ring_out (spsc)
                              │
                     writer thread: H5Dwrite_chunk

Thread topology selection:

  • n_threads == 1bounded::ring::spsc_t (zero CAS on both sides)
  • n_threads > 1bounded::ring::mpsc_t (CAS on producer head, single consumer)

Controlled by H5CPP_PIPELINE_WORKERS cmake option (default 1, preserving single-threaded behavior as the ABI-stable baseline).

Ring capacity is fixed at set_cache() time:
size = filter_scratch_bound(block_size) * 2 * n_workers

Because ring::adaptor_t uses a compile-time n_bytes_v, the implementation caps at H5CPP_MAX_CHUNK_BYTES (new compile-time constant, proposed default 32 MiB) and asserts at set_cache() if the actual chunk exceeds this.

Changes

  • h5cpp/H5Zpipeline.hpp: replace stub write_chunk_impl / read_chunk_impl in threaded_pipeline_t with ring-backed implementation
  • New h5cpp/H5Zpipeline_threaded.hpp (implementation file, included from H5Zpipeline.hpp)
  • CMakeLists.txt: add H5CPP_PIPELINE_WORKERS option; add H5CPP_MAX_CHUNK_BYTES constant
  • test/H5Zpipeline.cpp: add threaded pipeline round-trip test with filter chain

Depends on

#194 (C++20), #195 (H5Qall)

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew functionality

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions