feat: make MAX_MINIBLOCK_VALUES configurable via env var#6340
Merged
westonpace merged 3 commits intolance-format:mainfrom Mar 30, 2026
Merged
Conversation
Allow tuning the maximum number of values per mini-block chunk through the LANCE_MINIBLOCK_MAX_VALUES environment variable (default 4096). This helps reduce read amplification for workloads that read small contiguous row ranges from object storage where bandwidth is the bottleneck. Closes lance-format#6140 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Xuanwo
approved these changes
Mar 30, 2026
- Clamp LANCE_MINIBLOCK_MAX_VALUES to [1, 4096] to prevent zero (infinite loops) or values exceeding the log_num_values format constraint - Extract parse_max_miniblock_values() for testability - Fix stale comment on log_num_values field to reflect configurability - Add 5 serial tests covering: default, custom value, zero clamping, above-max clamping, and invalid input fallback - Add serial_test dev-dependency to lance-encoding Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ments The default is appropriate for local disks and same-region cloud object storage. Only consider lowering LANCE_MINIBLOCK_MAX_VALUES after profiling confirms read amplification is saturating available bandwidth. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
eddyxu
pushed a commit
that referenced
this pull request
Mar 31, 2026
## Summary - Converts `MAX_MINIBLOCK_VALUES` from a compile-time constant to a `LazyLock<u64>` that reads from the `LANCE_MINIBLOCK_MAX_VALUES` environment variable (default `4096`) - Updates all 6 usage sites across the encoding crate to dereference the `LazyLock` - Adds documentation in `docs/src/format/file/encoding.md` explaining the tuning knob and when it's useful Closes #6140 ## Test plan - [x] All existing miniblock tests pass (10 tests) - [x] All existing RLE tests pass (22 tests) - [x] `cargo clippy -p lance-encoding --tests -- -D warnings` clean - [ ] ~~Verify with a custom `LANCE_MINIBLOCK_MAX_VALUES` value that smaller mini-blocks are produced~~ 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MAX_MINIBLOCK_VALUESfrom a compile-time constant to aLazyLock<u64>that reads from theLANCE_MINIBLOCK_MAX_VALUESenvironment variable (default4096)LazyLockdocs/src/format/file/encoding.mdexplaining the tuning knob and when it's usefulCloses #6140
Test plan
cargo clippy -p lance-encoding --tests -- -D warningscleanVerify with a customLANCE_MINIBLOCK_MAX_VALUESvalue that smaller mini-blocks are produced🤖 Generated with Claude Code