Streaming API / Writing refactors

[Blackclaws]

@Apollo3zehn Thought this might make sense to discuss first.

I've done further changes the PureHDF in our internal fork. Many improvements to memory usage / write speed that could potentially simply be applied here, but also a new streaming API that allows you to stream data into an H5 file that's being written to a stream, also covering groups and staying low on memory usage (depending on how many pending groups you have < 100Mb resident size for 1.2M datasets total).

Should I open PRs for all of these changes? It might make sense to get them into the 3.0.0 branch still if you're interested.

[Apollo3zehn]

Hi @Blackclaws, yes this sound interesting as long as these features were implement in a generic way, not application specific. If I may ask, for what you case did you add streaming write support? I guess #180 is a requirement for the streaming support, right? I have to review that first.

[Blackclaws]

They are generic, we are streaming a large amount of data from a postgres database into an S3 type backend via multipart uploads, so the datasets can then be downloaded as .h5. The dataset is too large to fit in RAM and multipart uploads require you to finish parts (at least 5Mb size) as a whole. Keeping those in Ram introduces the same problem meaning only streaming datasets in doesn't help if metadata isn't contiguous.

The solution is to stream and use the Aggregated mode from #180 to keep Metadata blocks together.

A number of other things had to be changed such as keeping global heap together and reducing overhead on numerous internal data structures to get below the RAM requirements we had.

In the end writing got faster and uses less memory. I can try to divvy this up into a number of PRs splitting pure enhancements from the streaming approach which we also added later this might take a moment even with LLM assistance but I'd like to contribute it back if it helps others as well.

[Apollo3zehn]

Sounds very good, but I am also not very fast in reviewing, so no rush. I hope to be able to review the big endian PR today evening or tomorrow morning and then I have to see when I can go on with all the PRs related to streaming. A big thank you for your work so far, this has pushed PureHDF way forward (and it was already on a good level compared to my other projects ;-)).

[Apollo3zehn]

Multiple smaller PRs are preferred

[Blackclaws]

I've split the changes into smaller PRs as suggested. All of them target dev. A stacked PR names the PR it builds on, and its diff includes that PR's commits until it is merged.

Independent of the rest:

• test: delete the temporary files the tests create #191 test: delete the temporary files the tests create
• perf(reading): trim fixed-length strings before decoding, and bulk-decode in bounded batches #192 perf(reading): trim fixed-length strings before decoding, and bulk-decode in bounded batches. This overlaps with Improve string array read performance #149, which also touches the string read path, so whichever is merged first, I'll rebase the other onto it.
• perf(writing): checksum structures from their encoded bytes #193 perf(writing): checksum structures from their encoded bytes

The writing stack:

• feat(writing): keep file structure together, opt-in per placement #180 metadata placement, now builds on perf(writing): checksum structures from their encoded bytes #193. It front-loads in a single pass and keeps Interleaved as the default. The description lists what changed since your review.
  • perf(io): skip seeks that would not move the stream #194 skip seeks that would not move the stream
  • perf(writing): hold less per object while writing #195 hold less per object while writing. One small breaking change in H5Group, described in the PR.
    • perf(writing): give a write's variable-length values a heap of their own #196 a heap of its own for a write's variable-length values
      • refactor(writing)!: remove GlobalHeapFlushThreshold #197 remove GlobalHeapFlushThreshold (breaking, 3.0.0)
      • fix(writing): size compact datasets by everything they place in the header #198 size compact datasets by everything they place in the header
        • feat(writing)!: write only small datasets compact, and leave the smallest out of the default filters #199 smaller compact datasets and no default filters for the smallest (breaking, 3.0.0)
          • perf(writing): write chunks as soon as they can be written, and finish filtered datasets early #200 write chunks as soon as they can be written
            • perf(writing): write every byte once, in the order it is allocated #201 write every byte once, in allocation order
              • feat(writing): stream a file piece by piece with BeginStreaming #202 BeginStreaming, closes Writing data as a stream #146

The breaking changes are kept in PRs of their own, so they can be declined or held for 3.0.0 without blocking the rest. #200–#202 currently sit on #199. If you'd rather not take #199, I'll rebase them without it.

🤖 Generated with Claude Code