Skip to content

Blosc1 gpu#4

Merged
nclack merged 28 commits into
mainfrom
blosc1-gpu
May 7, 2026
Merged

Blosc1 gpu#4
nclack merged 28 commits into
mainfrom
blosc1-gpu

Conversation

@nclack
Copy link
Copy Markdown
Owner

@nclack nclack commented May 7, 2026

closes #2

Adds blosc-zstd and blosc-lz4 decoding on the gpu. Supports the various bit/byte shuffle things.

Uses nvcomp for the raw lz4/zstd parts.

Allows for mixed sources - a batch can draw samples from zarrs with different encodings. supporting zstd, blosc-lz4, blosc-zstd, and none for now.

Not really optimized yet.

There's a d2h early in the pipeline and a sync point that's required for getting parameters for the nvcomp launch. The blosc kernels themselves probably need an optimization pass too. I did some basic checking on register pressure etc, but more to do there.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 7, 2026

Codecov Report

❌ Patch coverage is 54.22343% with 168 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.91%. Comparing base (81b0bed) to head (c321ace).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/damacy.c 53.81% 46 Missing and 69 partials ⚠️
src/decoder/decoder_lz4.c 53.06% 7 Missing and 16 partials ⚠️
src/decoder/decoder_zstd.c 53.06% 5 Missing and 18 partials ⚠️
bench/main.c 0.00% 4 Missing ⚠️
src/zarr/zarr_metadata.c 78.57% 1 Missing and 2 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main       #4      +/-   ##
==========================================
+ Coverage   40.80%   41.91%   +1.11%     
==========================================
  Files          28       29       +1     
  Lines        4220     4454     +234     
  Branches     1008     1081      +73     
==========================================
+ Hits         1722     1867     +145     
- Misses       1995     2028      +33     
- Partials      503      559      +56     
Flag Coverage Δ
unittests 41.91% <54.22%> (+1.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/planner/planner.c 68.96% <100.00%> (+0.80%) ⬆️
src/zarr/zarr_metadata.c 49.61% <78.57%> (+10.30%) ⬆️
bench/main.c 0.00% <0.00%> (ø)
src/decoder/decoder_lz4.c 53.06% <53.06%> (ø)
src/decoder/decoder_zstd.c 53.06% <53.06%> (ø)
src/damacy.c 59.25% <53.81%> (-0.65%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nclack nclack merged commit 906318c into main May 7, 2026
2 of 3 checks passed
@nclack nclack deleted the blosc1-gpu branch May 7, 2026 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

blosc support

1 participant