Skip to content

Conversation

@akobrin1
Copy link
Contributor

  • moved blake3 hash enhancements from network-maker

  • increased lukechampine.com/blake3 version from 1.4.0 to 1.4.1, 1.4.0 has serious bug that leads to incorrect hash calculation - "writing data in two calls where the first call ends exactly on a chunk boundary produced a wrong hash".

  • unified all blake3-related hash functions in pkg/utils/hasher.go

  • blake3 file hashing optimization:
    // hashReaderBLAKE3 computes a BLAKE3 hash using an adaptive,
    // manual buffered read loop to avoid the *os.File.WriteTo fast-path
    // that limits throughput when using io.Copy/io.CopyBuffer.
    //
    // The buffer size is chosen based on data size:
    //
    // ≤ 4 MiB → 512 KiB buffer
    // 4–32 MiB → 1 MiB buffer
    // 32 MiB–2 GiB → 2 MiB buffer
    // > 2 GiB → 4 MiB buffer
    //
    // Buffers are reused from a concurrent-safe pool to reduce allocations.
    // This approach achieved the following throughput in benchmarks
    // on AMD Ryzen 9 5900X (Linux, lukechampine.com/blake3):
    //
    // Data size | Adaptive | Manual(1MiB) | io.Copy(~32KiB)
    // ----------|-------------|--------------|----------------
    // 1 MiB | 1.80 GB/s | 1.26 GB/s | 0.52 GB/s
    // 32 MiB | 3.00 GB/s | 3.02 GB/s | 0.50 GB/s
    // 256 MiB | 3.79 GB/s | 3.35 GB/s | 0.48 GB/s
    // 1 GiB | 3.91 GB/s | 3.27 GB/s | 0.53 GB/s
    //
    // Compared to io.Copy/io.CopyBuffer, the adaptive manual loop is
    // up to ~7× faster on large files, with fewer allocations.

  • added unit tests for all blake3 hashing functions

@akobrin1 akobrin1 requested a review from Copilot October 23, 2025 16:05
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR consolidates BLAKE3 hashing functionality into a single location and optimizes file hashing performance. The key changes include:

  • Upgrading lukechampine.com/blake3 from 1.4.0 to 1.4.1 to fix a critical hash calculation bug
  • Consolidating BLAKE3 hash functions from pkg/utils/utils.go and pkg/crypto/hash.go into a new pkg/utils/hasher.go module
  • Implementing adaptive buffer sizing for file hashing that achieves up to 7× performance improvement

Reviewed Changes

Copilot reviewed 10 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
go.mod, cmd/sncli/go.mod, tests/system/go.mod Updated blake3 dependency to v1.4.1
pkg/utils/hasher.go New unified module with optimized BLAKE3 hashing functions
pkg/utils/hasher_test.go Comprehensive test coverage for new hashing functions
pkg/utils/utils.go Removed BLAKE3 functions (migrated to hasher.go)
pkg/crypto/hash.go Deleted file (functionality moved to pkg/utils/hasher.go)
pkg/crypto/hash_test.go Deleted file (tests moved to pkg/utils/hasher_test.go)
supernode/cascade/download.go, sdk/action/client.go Updated to use new Blake3HashFile function

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@akobrin1 akobrin1 requested a review from Copilot October 23, 2025 16:15
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 10 out of 13 changed files in this pull request and generated 2 comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@akobrin1 akobrin1 requested a review from Copilot October 23, 2025 16:25
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 10 out of 13 changed files in this pull request and generated 1 comment.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@akobrin1 akobrin1 requested a review from Copilot October 23, 2025 16:39
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 10 out of 13 changed files in this pull request and generated 2 comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@akobrin1 akobrin1 requested a review from Copilot October 23, 2025 17:21
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 10 out of 13 changed files in this pull request and generated no new comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@mateeullahmalik mateeullahmalik merged commit 0ef6403 into master Oct 24, 2025
7 checks passed
@akobrin1 akobrin1 deleted the blake3-optimize branch October 24, 2025 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants