data files: add version into meta-header#18226
Merged
AskAlexSharov merged 4 commits intomainfrom Dec 14, 2025
Merged
Conversation
eastorski
requested review from
AskAlexSharov,
Giulio2002 and
sudeepdino008
as code owners
3 tasks
AskAlexSharov
changed the title
AskAlexSharov
requested changes
Dec 10, 2025
Collaborator
AskAlexSharov
left a comment
AskAlexSharov
left a comment
There was a problem hiding this comment.
AccountsDomain.History page-level compression enable - it's for another PR
db/seg/compress.go
Outdated
| return err | ||
| } | ||
|
|
||
| if c.featureFlagBitmask&PageLevelCompressionEnabled == PageLevelCompressionEnabled { |
Collaborator
There was a problem hiding this comment.
- add type for
featureFlagBitmaskand add method.Has(flag)for that type featureFlagBitmask uint8- max 8 feature flags - not limiting us too much?compPageValuesCount uint8- 256 keys per page - not limiting us too much?
Member
Author
There was a problem hiding this comment.
at this moment only 3 bits are reserved in bitmask, and we use max 64 values per page, this is why I decided to use uint8 for both types.
We always can upgrade if we need - because we still control version byte.
db/seg/decompress.go
Outdated
| d.version = d.data[0] | ||
|
|
||
| // 1st byte: version, | ||
| // 2nd byte: defines how exactly the file is compressed |
Collaborator
There was a problem hiding this comment.
this comment is valid not for all versions - better move it inside if
db/state/statecfg/state_schema.go
Outdated
| Hist: HistCfg{ | ||
| ValuesTable: kv.TblAccountHistoryVals, | ||
| CompressorCfg: seg.DefaultCfg, Compression: seg.CompressNone, | ||
| CompressorCfg: seg.DefaultCfg.WithValuesOnCompressedPage(16), Compression: seg.CompressNone, |
Collaborator
There was a problem hiding this comment.
AccountsDomain.History page-level compression enable - is for another PR
| Accessors: AccessorHashMap, | ||
|
|
||
| HistoryLargeValues: false, | ||
| HistoryValuesOnCompressedPage: 64, |
Collaborator
There was a problem hiding this comment.
seems this field can be removed - because HistoryCompressCfg.WithValuesOnCompressedPage added. (up to you)
eastorski
force-pushed
the
eastorski/upgraded-file-compression-format
branch
3 times, most recently
from
8c964af to
6f6f616
Compare
eastorski
force-pushed
the
eastorski/upgraded-file-compression-format
branch
from
6f6f616 to
a4cbce4
Compare
AskAlexSharov
approved these changes
Dec 13, 2025
AskAlexSharov
pushed a commit
that referenced
this pull request
Jan 22, 2026
Cherry-picked this PR: #18226 and also changed snapshots format version from v1 to v0 in compressor settings because new format will be used as default only since 3.4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR changes the internal format of snapshot files, specifically how the files are written to disk:
the first indicates the version, the second is a feature bitmask used during compression of the file.
If the page-level compression flag is enabled, a third additional byte follows the first two bytes in the file. It indicates the compression level — the number of elements per page.