Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling null fields in the filtertransformer and validator #796

Merged
merged 2 commits into from May 14, 2021

Conversation

ml-evs
Copy link
Member

@ml-evs ml-evs commented May 7, 2021

Closes #792, as reported by @JPBergsma.

This PR:

  • adds validation that no filters on a field returns entries with null values for that field. This can occur in MongoDB as null values match filters that test for inequality (i.e. null != 1 evaluates to true).
  • add additional "$ne": null expressions to all "$ne" filters in the MongoTransformer.
  • add additional "exists": True expression to all "!=" filters in the ElasticsearchTransformer.
  • add some null fields to our server test data so that we can verify this for other backends
  • updates CODEOWNERS

@ml-evs ml-evs requested a review from CasperWA as a code owner May 7, 2021 15:10
@codecov
Copy link

codecov bot commented May 7, 2021

Codecov Report

Merging #796 (0671b69) into master (44593e1) will decrease coverage by 0.04%.
The diff coverage is 71.42%.

❗ Current head 0671b69 differs from pull request most recent head 7ef9117. Consider uploading reports for the commit 7ef9117 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master     #796      +/-   ##
==========================================
- Coverage   92.72%   92.68%   -0.05%     
==========================================
  Files          68       68              
  Lines        3669     3675       +6     
==========================================
+ Hits         3402     3406       +4     
- Misses        267      269       +2     
Flag Coverage Δ
project 92.68% <71.42%> (-0.05%) ⬇️
validator 92.68% <71.42%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
optimade/validator/validator.py 82.14% <50.00%> (-0.25%) ⬇️
optimade/filtertransformers/elasticsearch.py 83.78% <100.00%> (ø)
optimade/filtertransformers/mongo.py 97.59% <100.00%> (+0.02%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 44593e1...7ef9117. Read the comment docs.

@ml-evs ml-evs requested a review from shyamd as a code owner May 7, 2021 16:15
@ml-evs ml-evs force-pushed the ml-evs/closes_#792 branch 2 times, most recently from b3957ab to d810c1a Compare May 7, 2021 17:14
@ml-evs ml-evs requested a review from markus1978 as a code owner May 7, 2021 17:14
@ml-evs ml-evs added priority/high Issue or PR with a consensus of high priority transformer/MongoDB Related to filter transformer for MongoDB labels May 11, 2021
@ml-evs ml-evs requested a review from JPBergsma May 11, 2021 07:42
optimade/filtertransformers/elasticsearch.py Outdated Show resolved Hide resolved
optimade/validator/validator.py Show resolved Hide resolved
CasperWA
CasperWA previously approved these changes May 11, 2021
@ml-evs
Copy link
Member Author

ml-evs commented May 11, 2021

I'll wait for @JPBergsma to have a look at this before merging!

@JPBergsma
Copy link
Contributor

JPBergsma commented May 12, 2021

I think Mongo DB uses null for missing values.
In the file mongo.py line 91 it should be null instead of None.
The same goes for lines 95,143,162,461 in test_mongo.py
Other than that I did not spot anything wrong.

@CasperWA
Copy link
Member

CasperWA commented May 12, 2021

I think Mongo DB uses null for missing values.
In the file mongo.py line 91 it should be null instead of None.
The same goes for lines 95,143,162,461 in test_mongo.py
Other than that I did not spot anything wrong.

The Python special NoneType value None is parsed as null for the MongoDB backend when needed through pymongo. So I think this should be fine.

Essentially, MongoDB null is represented as None in Python.

Or did you mean that it should be the string "null"?

@JPBergsma
Copy link
Contributor

@CasperWA
Yes you are right. Pymongo does the conversion from None to null for us.

@ml-evs ml-evs requested a review from CasperWA May 13, 2021 12:37
JPBergsma
JPBergsma previously approved these changes May 13, 2021
Copy link
Contributor

@JPBergsma JPBergsma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appart from the minor point in tests/server/routers/test_structures.py , I did not see any mistakes in this code. Although I did not fully understand the code in validator.py

@ml-evs ml-evs force-pushed the ml-evs/closes_#792 branch 2 times, most recently from 84e94e6 to 0661287 Compare May 13, 2021 15:35
@ml-evs ml-evs requested a review from JPBergsma May 13, 2021 15:35
JPBergsma
JPBergsma previously approved these changes May 13, 2021
CasperWA
CasperWA previously approved these changes May 14, 2021
@ml-evs ml-evs dismissed stale reviews from CasperWA and JPBergsma via 0671b69 May 14, 2021 09:27
…lidator

- Add validation of null/missing fields and negated queries
- Add null fields to test data
- Add router tests for filters on null fields
- Mongo transformer workaround for negative queries on null fields
- Fix != queries returning null fields for elasticsearch
@ml-evs ml-evs merged commit 3f2f404 into master May 14, 2021
@ml-evs ml-evs deleted the ml-evs/closes_#792 branch May 14, 2021 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority/high Issue or PR with a consensus of high priority transformer/MongoDB Related to filter transformer for MongoDB
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Verify that missing values are not returned in comparisons
3 participants