Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(pydantic): add oneof validations for correct document creation #2146

Merged
merged 6 commits into from Mar 11, 2021

Conversation

deepankarm
Copy link
Member

@deepankarm deepankarm requested a review from a team as a code owner March 10, 2021 20:36
@jina-bot jina-bot added size/M area/core This issue/PR affects the core codebase area/network This issue/PR affects network functionality area/testing This issue/PR affects testing component/peapod labels Mar 10, 2021
@jina-bot jina-bot added size/M and removed size/L labels Mar 10, 2021
@github-actions
Copy link

github-actions bot commented Mar 10, 2021

Latency summary

Current PR yields:

  • 😶 index QPS at 1134, delta to last 3 avg.: +2%
  • 😶 query QPS at 18, delta to last 3 avg.: -3%

Breakdown

Version Index QPS Query QPS
current 1134 18
1.0.8 1115 18
1.0.7 1103 18

Backed by latency-tracking. Further commits will update this comment.

@codecov
Copy link

codecov bot commented Mar 10, 2021

Codecov Report

Merging #2146 (8ad29d0) into master (441966b) will increase coverage by 0.10%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2146      +/-   ##
==========================================
+ Coverage   90.38%   90.48%   +0.10%     
==========================================
  Files         211      211              
  Lines       11199    11216      +17     
==========================================
+ Hits        10122    10149      +27     
+ Misses       1077     1067      -10     
Flag Coverage Δ
daemon 50.37% <0.00%> (-0.04%) ⬇️
jina 90.94% <100.00%> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
jina/peapods/runtimes/asyncio/rest/app.py 93.52% <ø> (-0.56%) ⬇️
jina/peapods/runtimes/asyncio/rest/models.py 100.00% <100.00%> (ø)
jina/peapods/pods/helper.py 98.92% <0.00%> (-0.03%) ⬇️
jina/docker/hubio.py 84.11% <0.00%> (ø)
jina/drivers/index.py 96.15% <0.00%> (ø)
jina/jaml/__init__.py 96.25% <0.00%> (ø)
jina/drivers/delete.py 100.00% <0.00%> (ø)
jina/drivers/search.py 98.18% <0.00%> (ø)
jina/parsers/optimizer.py 100.00% <0.00%> (ø)
jina/executors/compound.py 89.31% <0.00%> (ø)
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 441966b...8ad29d0. Read the comment docs.

@deepankarm deepankarm changed the title fix(pydantic): add oneof validations for correct document building from json fix(pydantic): add oneof validations for correct document creation Mar 10, 2021
@deepankarm
Copy link
Member Author

Still missing scenario:

Document allows custom dict key-value pairs and updates them to tags

d = Document({'id': '123', 'hello': 'world', 'tags': {'good': 'bye'}})

assert d.id == '123'  # true
assert d.tags['hello'] == 'world'  # true
assert d.tags['good'] == 'bye'  # true

Pydantic model skips these key-value pairs, which means if the client's (frontend) json payload has additional key-value pairs, they wouldn't get passed to Document()

PROTO_TO_PYDANTIC_MODELS.DocumentProto(hello='world')
# DocumentProto(id='', content_hash='', granularity=0, adjacency=0, level_name='', parent_id='', chunks=[], weight=0.0, length=0, matches=[], mime_type='', uri='', tags={}, location=[], offset=0, embedding=None, score=None, modality='', evaluations=[])

@deepankarm
Copy link
Member Author

Still missing scenario:

Document allows custom dict key-value pairs and updates them to tags

d = Document({'id': '123', 'hello': 'world', 'tags': {'good': 'bye'}})

assert d.id == '123'  # true
assert d.tags['hello'] == 'world'  # true
assert d.tags['good'] == 'bye'  # true

Pydantic model skips these key-value pairs, which means if the client's (frontend) json payload has additional key-value pairs, they wouldn't get passed to Document()

PROTO_TO_PYDANTIC_MODELS.DocumentProto(hello='world')
# DocumentProto(id='', content_hash='', granularity=0, adjacency=0, level_name='', parent_id='', chunks=[], weight=0.0, length=0, matches=[], mime_type='', uri='', tags={}, location=[], offset=0, embedding=None, score=None, modality='', evaluations=[])

This is done.

@florian-hoenicke florian-hoenicke merged commit 77b222a into master Mar 11, 2021
@florian-hoenicke florian-hoenicke deleted the fix-pydantic-oneof branch March 11, 2021 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/core This issue/PR affects the core codebase area/network This issue/PR affects network functionality area/testing This issue/PR affects testing component/executor component/jaml component/logging component/peapod executor/meta size/M
Projects
None yet
Development

Successfully merging this pull request may close these issues.

REST API - "oneof" validation
4 participants