feat: index drivers update and delete #1460

florian-hoenicke · 2020-12-14T16:01:36Z

Related ticket https://jinaai.atlassian.net/browse/JINACORE-552

add UpdateRequest and DeleteRequest
add update/delete driver for KVIndexer and VectorIndexer
when sharding, errors are not fixed but handled as we currently handle different indexers: include in response, but do not repair them (yet).
add tests for the drivers

jina/proto/jina.proto

github-actions · 2020-12-14T16:14:20Z

Latency summary

Current PR yields:

😶 index QPS at 1682, delta to last 3 avg.: +1%
😶 query QPS at 26, delta to last 3 avg.: +0%

Breakdown

Version	Index QPS	Query QPS
current	1682	26
`0.8.16`	1667	26
`0.8.15`	1633	26
`0.8.14`	1663	26

Backed by latency-tracking. Further commits will update this comment.

jina/enums.py

# Conflicts: # jina/clients/__init__.py

JoanFM

watch out! U are bringing changes to the hub submodule in this PR

florian-hoenicke · 2020-12-15T20:25:24Z

I reverted the changes thanks.

jina/clients/request.py

jina/types/request/__init__.py

# Conflicts: # jina/clients/__init__.py

jina/executors/indexers/keyvalue.py

codecov · 2020-12-27T21:23:28Z

Codecov Report

Merging #1460 (ebd9bbb) into master (afbc44e) will increase coverage by 0.52%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1460      +/-   ##
==========================================
+ Coverage   84.26%   84.79%   +0.52%     
==========================================
  Files         108      108              
  Lines        6336     6417      +81     
==========================================
+ Hits         5339     5441     +102     
+ Misses        997      976      -21

Impacted Files	Coverage Δ
jina/clients/request.py	`91.66% <100.00%> (+0.49%)`	⬆️
jina/drivers/index.py	`100.00% <100.00%> (+5.26%)`	⬆️
jina/enums.py	`96.57% <100.00%> (+0.14%)`	⬆️
jina/executors/indexers/vector.py	`88.94% <100.00%> (+0.23%)`	⬆️
jina/proto/jina_pb2.py	`100.00% <100.00%> (ø)`
jina/logging/profile.py	`68.33% <0.00%> (-0.58%)`	⬇️
jina/drivers/craft.py	`100.00% <0.00%> (ø)`
jina/types/ndarray/generic.py	`100.00% <0.00%> (ø)`
jina/drivers/encode.py	`94.91% <0.00%> (+0.08%)`	⬆️
jina/jaml/__init__.py	`95.93% <0.00%> (+0.09%)`	⬆️
... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update afbc44e...ebd9bbb. Read the comment docs.

JoanFM

missing unit tests for driver with delete and update method

jina/proto/jina.proto

jina/types/request/__init__.py

jina/proto/jina.proto

…rivers-update-and-delete # Conflicts: # jina/executors/__init__.py # jina/executors/indexers/vector.py

JoanFM

Watch out there are changes in the hub

JoanFM

Just minor things! Did not notice we did not have test_vector_index_driver

JoanFM · 2020-12-30T07:31:35Z

tests/unit/drivers/test_kv_index_driver.py

@@ -18,6 +18,14 @@ def add(self, keys: Iterator[int], values: Iterator[bytes], *args, **kwargs):
        for key, value in zip(keys, values):
            self.docs[key] = value

+    def update(self, keys: Iterator[int], values: Iterator[bytes], *args, **kwargs):


u can pass with these, I think these 2 methods won't be used

They are used here: SimpleKVIndexDriver(method='update')
However, I van remove the other methods: query, get_query_handler...

JoanFM · 2020-12-30T07:33:10Z

tests/unit/drivers/test_vector_index_driver.py

+        for key, value in zip(keys, vectors):
+            self.docs[key] = value
+
+    def update(self, keys: Iterator[int], values: Iterator[bytes], *args, **kwargs):


are also these used?

yes in line 63

@pytest.fixture(scope='function') def simple_kv_indexer_driver_update(): return SimpleVectorIndexDriver(method='update') @pytest.fixture(scope='function') def simple_kv_indexer_driver_delete(): return SimpleVectorIndexDriver(method='delete')

but this is MockGroundTruthVectorIndexer not SimpleVectorIndexDriver

True, but the executor is attached to that driver and sets the _exec function accordingly.
simple_vector_indexer_driver_add.attach(executor=mock_groundtruth_indexer, pea=None)

…rivers-update-and-delete # Conflicts: # tests/unit/drivers/test_kv_index_driver.py

feat: index drivers update and delete proto definition

5d3d163

florian-hoenicke requested review from maximilianwerk, cristianmtr and JoanFM December 14, 2020 16:01

jina-bot added size/S area/core This issue/PR affects the core codebase area/entrypoint This issue/PR affects the entrypoint codebase area/helper This issue/PR affects the helper functionality area/network This issue/PR affects network functionality component/proto labels Dec 14, 2020

JoanFM reviewed Dec 14, 2020

View reviewed changes

jina/proto/jina.proto Outdated Show resolved Hide resolved

feat: index drivers alias for request type

2e88df8

jina-bot added the component/type label Dec 14, 2020

florian-hoenicke added 2 commits December 14, 2020 23:55

feat: index drivers alias fix linter

2665895

feat: index update and delete in clients enums flow

6501ec8

JoanFM reviewed Dec 15, 2020

View reviewed changes

jina/enums.py Outdated Show resolved Hide resolved

florian-hoenicke added 3 commits December 15, 2020 20:44

Merge branch 'master' into index-drivers-update-and-delete

ace50b8

# Conflicts: # jina/clients/__init__.py

feat: index update and delete in clients

f2fe1be

feat: request enum change order

049d8a4

JoanFM requested changes Dec 15, 2020

View reviewed changes

fix: revert subproject version

40dca12

jina-bot added size/M component/client component/flow and removed size/S labels Dec 15, 2020

cristianmtr reviewed Dec 16, 2020

View reviewed changes

jina/clients/request.py Outdated Show resolved Hide resolved

cristianmtr reviewed Dec 16, 2020

View reviewed changes

jina/types/request/__init__.py Outdated Show resolved Hide resolved

feat: test for flow update and delete

d88445c

jina-bot added size/L and removed size/M labels Dec 22, 2020

florian-hoenicke added 5 commits December 23, 2020 17:57

Merge branch 'master' into index-drivers-update-and-delete

6e1aae2

fix: index driver test delete

1cfdac7

fix: drivers index test update and delete

f2ec2a9

feat: crud test deletion no content

80a9445

Merge branch 'master' into index-drivers-update-and-delete

b803463

# Conflicts: # jina/clients/__init__.py

florian-hoenicke marked this pull request as ready for review December 27, 2020 18:07

florian-hoenicke requested a review from a team as a code owner December 27, 2020 18:07

fix: indexer touch

d12e2a8

jina-bot added the executor/indexer label Dec 27, 2020

JoanFM reviewed Dec 27, 2020

View reviewed changes

jina/executors/indexers/keyvalue.py Outdated Show resolved Hide resolved

feat: index driver decorator update

1adfe45

JoanFM requested changes Dec 27, 2020

View reviewed changes

jina/proto/jina.proto Outdated Show resolved Hide resolved

jina/proto/jina.proto Outdated Show resolved Hide resolved

jina/types/request/__init__.py Outdated Show resolved Hide resolved

fix: cosmetic changes

a6cfe83

JoanFM requested changes Dec 28, 2020

View reviewed changes

jina/proto/jina.proto Outdated Show resolved Hide resolved

jina/proto/jina.proto Outdated Show resolved Hide resolved

florian-hoenicke added 4 commits December 28, 2020 14:52

feat: index drivers unit test

929d2a2

fix: vector index unit test

f8ee58c

Merge branch 'master' of https://github.com/jina-ai/jina into index-d…

c5758a9

…rivers-update-and-delete # Conflicts: # jina/executors/__init__.py # jina/executors/indexers/vector.py

fix: subproject

06c60a7

JoanFM reviewed Dec 29, 2020

View reviewed changes

fix: subproject

b80821e

JoanFM reviewed Dec 30, 2020

View reviewed changes

florian-hoenicke added 3 commits December 30, 2020 10:35

feat: index driver cleanup test

c2c91c7

Merge branch 'master' of https://github.com/jina-ai/jina into index-d…

5003ac4

…rivers-update-and-delete # Conflicts: # tests/unit/drivers/test_kv_index_driver.py

feat: test rename vars

ebd9bbb

JoanFM approved these changes Dec 30, 2020

View reviewed changes

florian-hoenicke merged commit 3fe2145 into master Dec 30, 2020

florian-hoenicke deleted the index-drivers-update-and-delete branch December 30, 2020 13:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: index drivers update and delete #1460

feat: index drivers update and delete #1460

florian-hoenicke commented Dec 14, 2020

github-actions bot commented Dec 14, 2020 •

edited

Loading

JoanFM left a comment

florian-hoenicke commented Dec 15, 2020

codecov bot commented Dec 27, 2020 •

edited

Loading

JoanFM left a comment

JoanFM left a comment

JoanFM left a comment

JoanFM Dec 30, 2020

florian-hoenicke Dec 30, 2020

JoanFM Dec 30, 2020

florian-hoenicke Dec 30, 2020

JoanFM Dec 30, 2020

florian-hoenicke Dec 30, 2020 •

edited

Loading

JoanFM Dec 30, 2020

florian-hoenicke Dec 30, 2020

feat: index drivers update and delete #1460

feat: index drivers update and delete #1460

Conversation

florian-hoenicke commented Dec 14, 2020

github-actions bot commented Dec 14, 2020 • edited Loading

Latency summary

Breakdown

JoanFM left a comment

Choose a reason for hiding this comment

florian-hoenicke commented Dec 15, 2020

codecov bot commented Dec 27, 2020 • edited Loading

Codecov Report

JoanFM left a comment

Choose a reason for hiding this comment

JoanFM left a comment

Choose a reason for hiding this comment

JoanFM left a comment

Choose a reason for hiding this comment

JoanFM Dec 30, 2020

Choose a reason for hiding this comment

florian-hoenicke Dec 30, 2020

Choose a reason for hiding this comment

JoanFM Dec 30, 2020

Choose a reason for hiding this comment

florian-hoenicke Dec 30, 2020

Choose a reason for hiding this comment

JoanFM Dec 30, 2020

Choose a reason for hiding this comment

florian-hoenicke Dec 30, 2020 • edited Loading

Choose a reason for hiding this comment

JoanFM Dec 30, 2020

Choose a reason for hiding this comment

florian-hoenicke Dec 30, 2020

Choose a reason for hiding this comment

github-actions bot commented Dec 14, 2020 •

edited

Loading

codecov bot commented Dec 27, 2020 •

edited

Loading

florian-hoenicke Dec 30, 2020 •

edited

Loading