Skip to content

Fix cut-off sentence in GPU blog post #808

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dcherian merged 2 commits into from
Oct 18, 2025
Merged

Fix cut-off sentence in GPU blog post #808

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f9a6451
Fix cut-off sentence in GPU blog post
maxrjones Oct 6, 2025
c393478
Update index.md
dcherian Oct 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/posts/gpu-pipeline/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ PyTorch’s `DataLoader` includes options like `num_workers`, `pin_memory`, and

## Hackathon: Strategies Explored!

During the hackathon, we tested the following strategies to improve the data loading performance. In the end, we were able to achieve
During the hackathon, we tested the following strategies to improve the data loading performance. In the end, we were able to achieve at least ~17x improvement on 1 GPU in training throughput by optimizing data loading and preprocessing steps.

### Step 1: Optimized Chunking & Compression

Expand Down Expand Up @@ -198,7 +198,7 @@ With nvCOMP, all steps of data loading including reading from disk, decompressio

To unlock this, we would need zarr-python to support GPU-based decompression codecs, with one for Zstandard (Zstd) currently being implemented in [this PR](https://github.com/zarr-developers/zarr-python/pull/2863).

We tested the performance of GPU-based decompression using nvCOMP with Zarr-Python 3 and KvikIO, and compared it to CPU-based decompression using [this data reading benchmark here](https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus/tree/v1.0/benchmarks/era5_zarr_benchmark.py).
We tested the performance of GPU-based decompression using nvCOMP with Zarr-Python 3 and KvikIO, and compared it to CPU-based decompression using [this data reading benchmark here](https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus/blob/v1.0/benchmarks/era5_zarr_benchmark.py).

Here are the results:

Expand Down