Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cover mmap storages in compatibility tests #3715

Merged
merged 3 commits into from
Feb 28, 2024
Merged

Cover mmap storages in compatibility tests #3715

merged 3 commits into from
Feb 28, 2024

Conversation

agourlay
Copy link
Member

@agourlay agourlay commented Feb 28, 2024 •
edited

I have noticed that our storage compatibility tests do not cover our mmap storage implementations.

This PR tries to mitigate the issue by introducing new collections with mmap configurations.

The archive size grows quickly so it becomes difficult to test all combinations.

Also the refreshed archive now contains information about clock maps.

e.g.

./storage/collections/test_collection_vector_on_disk/payload_index.json
./storage/collections/test_collection_vector_on_disk/0/
./storage/collections/test_collection_vector_on_disk/0/segments/
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/OPTIONS-000035
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/OPTIONS-000021
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/000027.sst
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/000032.log
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/LOCK
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/LOG
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/000025.sst
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/000026.sst
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/config.json
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/IDENTITY
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/000023.sst
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/000024.sst
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/CURRENT
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/MANIFEST-000033
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/000028.sst
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/payload_index/000029.sst
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/000014.log
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_storage-image/
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_storage-image/deleted/
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_storage-image/deleted/flags_a.dat
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_storage-image/deleted/status.dat
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_storage-image/vectors/
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_storage-image/vectors/chunk_1.mmap
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_storage-image/vectors/config.json
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_storage-image/vectors/chunk_0.mmap
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_storage-image/vectors/status.dat
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/LOCK
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/OPTIONS-000017
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/OPTIONS-000007
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/LOG
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/000009.sst
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/000011.sst
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/IDENTITY
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/000008.sst
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_index-image/
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_index-image/hnsw_config.json
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_index-image/graph.bin
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_index-image/links.bin
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/segment.json
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/CURRENT
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/version.info
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/000010.sst
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/MANIFEST-000015
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_index-text/
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_index-text/inverted_index.data
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_index-text/inverted_index_config.json
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_index-text/indices_tracker.json
./storage/collections/test_collection_vector_on_disk/0/segments/440ad0d4-61dc-4605-bd05-747b8ed28094/vector_index-text/sparse_index_config.json
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/payload_index/
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/payload_index/OPTIONS-000021
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/payload_index/LOCK
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/payload_index/MANIFEST-000005
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/payload_index/LOG
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/payload_index/config.json
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/payload_index/OPTIONS-000019
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/payload_index/IDENTITY
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/payload_index/CURRENT
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/payload_index/000004.log
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/vector_storage-image/
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/vector_storage-image/deleted/
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/vector_storage-image/deleted/flags_a.dat
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/vector_storage-image/deleted/status.dat
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/vector_storage-image/vectors/
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/vector_storage-image/vectors/config.json
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/vector_storage-image/vectors/status.dat
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/LOCK
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/OPTIONS-000007
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/MANIFEST-000005
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/LOG
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/IDENTITY
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/segment.json
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/CURRENT
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/version.info
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/000004.log
./storage/collections/test_collection_vector_on_disk/0/segments/1b71a28e-177f-4e24-b5ad-09259ed8bb63/vector_index-text/
./storage/collections/test_collection_vector_on_disk/0/temp_segments/
./storage/collections/test_collection_vector_on_disk/0/clock_map_highest.json
./storage/collections/test_collection_vector_on_disk/0/clock_map_cutoff.json
./storage/collections/test_collection_vector_on_disk/0/shard_config.json
./storage/collections/test_collection_vector_on_disk/0/replica_state.json
./storage/collections/test_collection_vector_on_disk/0/wal/
./storage/collections/test_collection_vector_on_disk/0/wal/first-index
./storage/collections/test_collection_vector_on_disk/0/wal/open-2
./storage/collections/test_collection_vector_on_disk/0/wal/open-1
./storage/collections/test_collection_vector_on_disk/config.json
./storage/collections/test_collection_vector_on_disk/version.info

@generall
Copy link
Member

Which version is used to generate current snapshot?

@agourlay
Copy link
Member Author

@generall I used dev assuming we are releasing soon.

@generall
Copy link
Member

@generall I used dev assuming we are releasing soon.

I mean, we should test compatibility with previous version, not current dev

@agourlay
Copy link
Member Author

@generall fair point 👍

I am proposing the following fix for it #3717

coszio
coszio approved these changes Feb 28, 2024
View reviewed changes
Copy link
Contributor

@coszio coszio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming the changes from #3717 are merged, this looks good to me ✅

@agourlay agourlay merged commit a56277d into dev Feb 28, 2024
17 checks passed
@agourlay agourlay deleted the mmap-storage-compat branch February 28, 2024 15:05
timvisee pushed a commit that referenced this pull request Mar 5, 2024
@agourlay @timvisee
Cover mmap storages in compatibility tests (#3715)
9d4bc13 
* Cover mmap storages in compatibility tests

* regen archive from scratch

* update collection used for detecting up
@agourlay agourlay mentioned this pull request Mar 13, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@generall generall generall approved these changes

@coszio coszio coszio approved these changes

@timvisee timvisee Awaiting requested review from timvisee

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@agourlay @generall @coszio