Skip to content

Comments

Add layered zip indexes#17

Merged
klauspost merged 3 commits intominio:mainfrom
klauspost:layered-indexes
Jan 7, 2026
Merged

Add layered zip indexes#17
klauspost merged 3 commits intominio:mainfrom
klauspost:layered-indexes

Conversation

@klauspost
Copy link
Collaborator

This will allow to specify multiple indexes and have them searchable as a single entity.

Indexes are added in order and the layered index will represent the combined state.
That means that on a file with a given path the newest version will be returned.

A generic comparable type can be provided as the index reference type, so for any given file this can be seen in the results.

An index can be added as a "delete" index. This means that any files in this will be removed from the underlying layers.

  • Add LayeredIndex[T] for combining multiple zip indexes with generic references
  • Implement file override semantics: newer layers override older ones
  • Add delete layers to remove files from previous layers
  • Automatic empty directory cleanup when files are deleted
  • Binary search optimization: files sorted by name on layer addition
  • Find operations use O(log n) binary search instead of O(n) linear search
  • Concurrent serialization/deserialization with user-provided RefSerializer[T]
  • Manual msgpack format: version + layers with ref data as binary arrays
  • Comprehensive test suite with 15+ test scenarios and benchmarks
  • Iterator support for Go 1.23 range syntax
  • Complete API: layer management, file access, conversion, serialization
  • Updated README with usage examples and API documentation

  - Add LayeredIndex[T] for combining multiple zip indexes with generic references
  - Implement file override semantics: newer layers override older ones
  - Add delete layers to remove files from previous layers
  - Automatic empty directory cleanup when files are deleted
  - Binary search optimization: files sorted by name on layer addition
  - Find operations use O(log n) binary search instead of
   O(n) linear search
  - Concurrent serialization/deserialization with user-provided RefSerializer[T]
  - Manual msgpack format: version + layers with ref data as binary arrays
  - Comprehensive test suite with 15+ test scenarios and benchmarks
  - Iterator support for Go 1.23 range syntax
  - Complete API: layer management, file access, conversion, serialization
  - Updated README with usage examples and API documentation
@klauspost klauspost merged commit 3e52ca2 into minio:main Jan 7, 2026
5 checks passed
@klauspost klauspost deleted the layered-indexes branch January 7, 2026 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants