Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: rewrite handling of missing keys for indexers #1628

Merged
merged 1 commit into from
Jan 8, 2021

Conversation

cristianmtr
Copy link
Contributor

@cristianmtr cristianmtr commented Jan 8, 2021

This will be required for the CRUD integration tests suite

  • handling of missing keys should not Raise
  • missing keys are filtered out of the request
  • we do log a warning - up to the user to interpret if that's expected or not

@cristianmtr cristianmtr requested a review from a team as a code owner January 8, 2021 12:23
@jina-bot jina-bot added size/S area/core This issue/PR affects the core codebase area/helper This issue/PR affects the helper functionality area/testing This issue/PR affects testing component/executor executor/indexer labels Jan 8, 2021
@cristianmtr cristianmtr requested review from JoanFM, florian-hoenicke and CatStark and removed request for rutujasurve94 January 8, 2021 12:25
jina/executors/indexers/keyvalue.py Outdated Show resolved Hide resolved
jina/helper.py Outdated Show resolved Hide resolved
@github-actions
Copy link

github-actions bot commented Jan 8, 2021

Latency summary

Current PR yields:

  • 😶 index QPS at 1898, delta to last 3 avg.: +0%
  • 😶 query QPS at 37, delta to last 3 avg.: +0%

Breakdown

Version Index QPS Query QPS
current 1898 37
0.9.2 1853 37
0.9.1 1987 37
0.8.22 1880 36

Backed by latency-tracking. Further commits will update this comment.

@github-actions
Copy link

github-actions bot commented Jan 8, 2021

Latency summary

Current PR yields:

  • 😶 index QPS at 1758, delta to last 3 avg.: +0%
  • 😶 query QPS at 33, delta to last 3 avg.: -2%

Breakdown

Version Index QPS Query QPS
current 1758 33
0.9.2 1776 33
0.9.1 1723 33
0.8.22 1732 33

Backed by latency-tracking. Further commits will update this comment.

jina/executors/indexers/keyvalue.py Outdated Show resolved Hide resolved
jina/executors/indexers/keyvalue.py Outdated Show resolved Hide resolved
@cristianmtr
Copy link
Contributor Author

Not sure why I can't reply to this comment: #1628 (comment)
but @JoanFM here's the reply

When we call an .update, the keys will be computed for the update anyway. The update will then call the delete with only the filtered out keys.

If we don't do it this way, the .delete method will have to recalculate and filter out the missing keys.

When .delete is called on its own (not part of an update), then the optimization will not happen.

@cristianmtr cristianmtr changed the title fix: rewrite handling of missing keys for kv indexer fix: rewrite handling of missing keys for indexers Jan 8, 2021
jina/helper.py Outdated Show resolved Hide resolved
jina/helper.py Outdated Show resolved Hide resolved
@cristianmtr cristianmtr force-pushed the fix-kv-indexer branch 3 times, most recently from 8b93831 to 3d8d678 Compare January 8, 2021 13:50
@jina-bot jina-bot removed the size/S label Jan 8, 2021
@jina-bot jina-bot added the size/M label Jan 8, 2021
@cristianmtr cristianmtr mentioned this pull request Jan 8, 2021
8 tasks
jina/executors/indexers/cache.py Outdated Show resolved Hide resolved
jina/executors/indexers/cache.py Outdated Show resolved Hide resolved
jina/executors/indexers/vector.py Outdated Show resolved Hide resolved
jina/executors/indexers/vector.py Outdated Show resolved Hide resolved
.gitignore Outdated Show resolved Hide resolved
@cristianmtr cristianmtr force-pushed the fix-kv-indexer branch 4 times, most recently from ab1ae68 to 4e98eb5 Compare January 8, 2021 14:58
Copy link
Member

@maximilianwerk maximilianwerk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just organizational/cleanup comments.

tests/unit/executors/indexers/test_numpyindexer.py Outdated Show resolved Hide resolved
jina/helper.py Outdated Show resolved Hide resolved
jina/executors/indexers/__init__.py Outdated Show resolved Hide resolved
jina/helper.py Outdated Show resolved Hide resolved
@cristianmtr cristianmtr merged commit 4b67bce into master Jan 8, 2021
@cristianmtr cristianmtr deleted the fix-kv-indexer branch January 8, 2021 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/core This issue/PR affects the core codebase area/helper This issue/PR affects the helper functionality area/testing This issue/PR affects testing component/executor executor/indexer size/M
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants