-
Notifications
You must be signed in to change notification settings - Fork 0
blake3 hashing code optimization #219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR consolidates BLAKE3 hashing functionality into a single location and optimizes file hashing performance. The key changes include:
- Upgrading
lukechampine.com/blake3from 1.4.0 to 1.4.1 to fix a critical hash calculation bug - Consolidating BLAKE3 hash functions from
pkg/utils/utils.goandpkg/crypto/hash.gointo a newpkg/utils/hasher.gomodule - Implementing adaptive buffer sizing for file hashing that achieves up to 7× performance improvement
Reviewed Changes
Copilot reviewed 10 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| go.mod, cmd/sncli/go.mod, tests/system/go.mod | Updated blake3 dependency to v1.4.1 |
| pkg/utils/hasher.go | New unified module with optimized BLAKE3 hashing functions |
| pkg/utils/hasher_test.go | Comprehensive test coverage for new hashing functions |
| pkg/utils/utils.go | Removed BLAKE3 functions (migrated to hasher.go) |
| pkg/crypto/hash.go | Deleted file (functionality moved to pkg/utils/hasher.go) |
| pkg/crypto/hash_test.go | Deleted file (tests moved to pkg/utils/hasher_test.go) |
| supernode/cascade/download.go, sdk/action/client.go | Updated to use new Blake3HashFile function |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 10 out of 13 changed files in this pull request and generated 2 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 10 out of 13 changed files in this pull request and generated 1 comment.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 10 out of 13 changed files in this pull request and generated 2 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 10 out of 13 changed files in this pull request and generated no new comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
moved blake3 hash enhancements from network-maker
increased lukechampine.com/blake3 version from 1.4.0 to 1.4.1, 1.4.0 has serious bug that leads to incorrect hash calculation - "writing data in two calls where the first call ends exactly on a chunk boundary produced a wrong hash".
unified all blake3-related hash functions in pkg/utils/hasher.go
blake3 file hashing optimization:
// hashReaderBLAKE3 computes a BLAKE3 hash using an adaptive,
// manual buffered read loop to avoid the *os.File.WriteTo fast-path
// that limits throughput when using io.Copy/io.CopyBuffer.
//
// The buffer size is chosen based on data size:
//
// ≤ 4 MiB → 512 KiB buffer
// 4–32 MiB → 1 MiB buffer
// 32 MiB–2 GiB → 2 MiB buffer
// > 2 GiB → 4 MiB buffer
//
// Buffers are reused from a concurrent-safe pool to reduce allocations.
// This approach achieved the following throughput in benchmarks
// on AMD Ryzen 9 5900X (Linux, lukechampine.com/blake3):
//
// Data size | Adaptive | Manual(1MiB) | io.Copy(~32KiB)
// ----------|-------------|--------------|----------------
// 1 MiB | 1.80 GB/s | 1.26 GB/s | 0.52 GB/s
// 32 MiB | 3.00 GB/s | 3.02 GB/s | 0.50 GB/s
// 256 MiB | 3.79 GB/s | 3.35 GB/s | 0.48 GB/s
// 1 GiB | 3.91 GB/s | 3.27 GB/s | 0.53 GB/s
//
// Compared to io.Copy/io.CopyBuffer, the adaptive manual loop is
// up to ~7× faster on large files, with fewer allocations.
added unit tests for all blake3 hashing functions