feat: Detach chunk identity from chunk name#56
Merged
define-null merged 5 commits intoMay 18, 2026
Conversation
kalabukdima
reviewed
May 18, 2026
In order to be able to detouch chunk ids from the data they reference the worker code should not parse the chunk id format to obtain the last_block. It may not match with the actual data in the chunk, and that prohibits chunk-id format changes.
ChunkRef.chunk becomes an opaque Arc<String>; DataChunk is reduced to a layout parser. The on-disk format gains an optional trailing suffix so multiple chunks can cover the same block range.
define-null
force-pushed
the
defnull/NET-385/detach-chunk-identity-from-chunk-name
branch
from
85f2b0e to
2f95e5b
Compare
kalabukdima
reviewed
May 18, 2026
kalabukdima
approved these changes
May 18, 2026
define-null
deleted the
defnull/NET-385/detach-chunk-identity-from-chunk-name
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements: https://linear.app/sqd-ai/issue/NET-385/detach-chunk-identity-from-chunk-name-in-sqd-network
What is this PR about?
PR changes the way worker is treating chunks. The logic which deduce block boundaries from the chunk id is mostly removed, and only kept in the layout for now. The idea is to allow chunk replacement, where a single chunk could be shadowed by another chunk of the same shape.
Read path for the worker has been modified to drop the branch which returned the last_block in certain cases. The chunk id shouldn't be used as a source of truth for the data that it corresponds to.
How does it work?
Chunk id is no longer analyzed in the read path and works purely as an id to lookup if the worker has corresponding chunk or not. The sqd-network was updated to support lookup based on the chunk ref.
According to @kalabukdima block range is always provided in the current implementation, so it's safe to expect the range.