Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not scan_keys on getting labelsets #1863

Merged
merged 10 commits into from
Feb 22, 2024
Merged

Do not scan_keys on getting labelsets #1863

merged 10 commits into from
Feb 22, 2024

Conversation

lferran
Copy link
Contributor

@lferran lferran commented Feb 21, 2024

Description

Current implementation of getting labelsets uses tikv scan keys to get the list of labelsets.
Tikv doesn't behave very well when scanning keys.

This PR does:

  • materializes the list of labelsets in a separate key in the KV
  • makes sure to update the list on additions and deletions
  • adds a migration step to materialize labelset lists for existing kbs
  • runs backward compatible code to make sure the feature works while the migration hasn't run yet

How was this PR tested?

Integration tests

Copy link

codecov bot commented Feb 21, 2024

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (f5f1bd2) 84.35% compared to head (bdf16c5) 84.34%.
Report is 2 commits behind head on main.

Files Patch % Lines
nucliadb/nucliadb/ingest/orm/knowledgebox.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1863      +/-   ##
==========================================
- Coverage   84.35%   84.34%   -0.02%     
==========================================
  Files         328      328              
  Lines       18757    18747      -10     
==========================================
- Hits        15823    15812      -11     
- Misses       2934     2935       +1     
Flag Coverage Δ
ingest 69.74% <50.00%> (+0.01%) ⬆️
sdk 87.85% <ø> (ø)
utils 81.81% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@lferran lferran marked this pull request as ready for review February 21, 2024 13:35
@lferran lferran requested a review from a team February 21, 2024 13:35
@lferran lferran merged commit 9fa8194 into main Feb 22, 2024
83 checks passed
@lferran lferran deleted the labelsets-do-not-scan branch February 22, 2024 08:11
@lferran
Copy link
Contributor Author

lferran commented Feb 22, 2024

[sc-8994]

Copy link

This pull request has been linked to Shortcut Story #8994: reduce tikv scans.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants