Added default mongo implementations for HAS ALL/ANY/ONLY #173

ml-evs · 2020-02-10T01:02:51Z

Long overdue PR that closes #98. The only list queries that remain would be e.g. HAS ALL <3, >3, which will be trickier (in fact, I would support this type of query being demoted to optional in the spec).

Implement ALL/ANY/ONLY for mongo transformer
Check filtertransformer errors are sensible (i.e. return "not implemented") rather than just checking that a VisitError is raised (can be a false positive).

In the course of this PR, a bug was found in mongomock which is why we are now testing with MongoDB directly. The fix has been merged into mongomock but not yet released. You can have a look at the edit history of this PR description for more detail.

ml-evs · 2020-02-15T14:58:42Z

Can't resolve this myself, so raised an issue with mongomock: mongomock/mongomock#594

ml-evs · 2020-02-24T14:06:41Z

I've raised a PR with mongomock/mongomock#597 that fixes the issues with HAS ONLY. Not sure how long it will take to get merged... thinking about it, should we make the CI use a real Mongo? The tests can just check if they can see a real mongo or not, and decide which to use (so local tests don't need a running mongo).

CasperWA · 2020-02-24T19:30:57Z

I've raised a PR with mongomock/mongomock#597 that fixes the issues with HAS ONLY. Not sure how long it will take to get merged... thinking about it, should we make the CI use a real Mongo? The tests can just check if they can see a real mongo or not, and decide which to use (so local tests don't need a running mongo).

I guess, yeah - although we (Materials Cloud) do use mongomock for our index meta-database. But maybe we should instead use a proper MongoDB, and that should be our go-to recommendation, i.e., also what is tested in CI?
Since you know more about Mongo and using it, do you think this recommendation is fair or is it "normal" to use mongomock in production?

ml-evs · 2020-02-24T20:47:24Z

I guess, yeah - although we (Materials Cloud) do use mongomock for our index meta-database. But maybe we should instead use a proper MongoDB, and that should be our go-to recommendation, i.e., also what is tested in CI?
Since you know more about Mongo and using it, do you think this recommendation is fair or is it "normal" to use mongomock in production?

I think it's fine for something as trivial as the meta-db, but as we see in this case, there are features that are not supported by mongomock (currently), and we can easily add a mongo to our CI with just a few lines. It's not even clear that we're testing against a real database at any point (except in our own implementations).

I had a look earlier and we'd have to change our config a little bit, I think it makes sense to disentangle the use_real_mongo option from importing the test data (test_structures.json etc). It would make sense to change the default database name from optimade to _optimade_testing or some such, to prevent dropping some poor user's main database when the app is loaded in test mode.

CasperWA · 2020-02-25T09:53:58Z

I think it's fine for something as trivial as the meta-db, but as we see in this case, there are features that are not supported by mongomock (currently), and we can easily add a mongo to our CI with just a few lines. It's not even clear that we're testing against a real database at any point (except in our own implementations).

Good point. And I indeed think we should.

I had a look earlier and we'd have to change our config a little bit, I think it makes sense to disentangle the use_real_mongo option from importing the test data (test_structures.json etc). It would make sense to change the default database name from optimade to _optimade_testing or some such, to prevent dropping some poor user's main database when the app is loaded in test mode.

Sounds good.
I think I will have to see the solution (or a proper outline) before I can put my finger on specific parts, if needed.

CasperWA

Much needed implementation! Thanks @ml-evs.

I have some questions, suggestions, and very minor clean ups.

optimade/filtertransformers/mongo.py

tests/filterparser/test_filterparser.py

tests/filtertransformers/test_mongo.py

tests/server/test_query_params.py

CasperWA · 2020-02-27T11:20:28Z

As per the discussion in this PR - if you can, maybe just use a real MongoDB - otherwise let's skip the failing test, keeping the comments you've made and add that it works for a real MongoDB. Maybe also referencing your issue/PR @ mongomock?

ml-evs · 2020-02-28T11:38:41Z

As per the discussion in this PR - if you can, maybe just use a real MongoDB - otherwise let's skip the failing test, keeping the comments you've made and add that it works for a real MongoDB. Maybe also referencing your issue/PR @ mongomock?

I think I would rather make another PR to use mongo for all tests, then hopefully watch this PR turn green :)

codecov · 2020-02-28T12:41:34Z

Codecov Report

Merging #173 into master will increase coverage by 0.18%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #173      +/-   ##
==========================================
+ Coverage   86.43%   86.62%   +0.18%     
==========================================
  Files          39       39              
  Lines        1851     1854       +3     
==========================================
+ Hits         1600     1606       +6     
+ Misses        251      248       -3

Flag	Coverage Δ
#unittests	`86.62% <100%> (+0.18%)`	⬆️

Impacted Files	Coverage Δ
optimade/filtertransformers/mongo.py	`95.29% <100%> (+3.83%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f0fc998...4257de4. Read the comment docs.

CasperWA · 2020-02-28T12:43:27Z

It's green!!! 😮
Edit: Ah ... you've rebased on your other PR ... I see.... cheating ;)

ml-evs · 2020-02-28T12:56:18Z

It's green!!! open_mouth
Edit: Ah ... you've rebased on your other PR ... I see.... cheating ;)

I still have an "un-rebased" (debased?) local copy that we can revert to, should we decide not to merge the Mongo CI PR, so only half cheating :P

Co-Authored-By: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>

CasperWA

On the home stretch, thanks @ml-evs

tests/server/test_query_params.py

CasperWA

Upwards and onwards - next stop OPTiMaDe v1.0.0-rc.1 😄

@CasperWA

Up to v0.6.0. **New features**: - GitHub Action validator that runs `optimade_validator` for a locally running OPTiMaDe server (#191, @CasperWA, tested by @ml-evs) - Support filter queries for `HAS ALL`, `HAS ANY` and `HAS ONLY` and value lists on MongoDB backends (#173, @ml-evs) Note: `OPERATOR` use in value lists are still _not_ supported. - Debug mode. Start the server in debug mode to enable `debug` log-level in `uvicorn` and get a full Python traceback in the JSON response (#190, @CasperWA) - Add testing of mandatory `filter` queries to `optimade_validator` (#205, @ml-evs) **Updates**: - Allow Cross-Origin requests from anywhere (`*`), i.e., enable CORS for both servers (#194, @CasperWA) - Updates to models (correct misspelling, more transparent model class naming, streamline models with respect to python class inheritance, update field descriptions) (#195, @CasperWA) - API change: Rename `optimade.models.toplevel.py` to `optimade.models.responses.py` (#195, @CasperWA) - Update dependencies to newest versions (as of 02.03.2020) (#202, #196, @CasperWA) - Move imports from `starlette` to `fastapi`, where possible (#202, @ml-evs) - Remove custom middleware to redirect slashed URLs in favor of `starlette` implementation (#202, @ml-evs) - CI tests are now performed with a real MongoDB in the backend. CI tests are also performed with a `mongomock` backend for the tests in `server/test_middleware.py`, `server/test_server_validation.py`, and `server/test_config.py` (#196, @ml-evs, additional testing by @CasperWA ) - Remove `/optimade` from base URLs. This was especially important for the OpenAPI schema (#201, #216, @CasperWA, @ml-evs) - Check integrity of query part of the raw URL using a custom middleware (#209, @CasperWA) - New `optimade/server/exceptions.py` to contain custom `HttpException`s, moved `BadRequest` here (#209, @CasperWA) - Pattern and regex testing for `data.available_api_versions` parts in `/info` endpoint and fix tests for the same (#211, @CasperWA) - Restructure import of test data for regular server (#212, @shyamd) **Bug fixes**: - New retrieval URL for Materials-Consortia's list of providers (#187, @CasperWA) - Skip local `HAS ONLY` tests with a `mongomock` backend, since v3.19.0 does not support these (#206, @ml-evs) - Resource ID's can now contain slashes (`/`) (#183, @ml-evs, @CasperWA) - Remove _valid_ version part of base URL in `meta.query.representation` (#201, @CasperWA)

ml-evs requested review from CasperWA, ltalirz and shyamd February 10, 2020 01:03

ml-evs changed the base branch from ml-evs/add_landing_page to master February 10, 2020 09:19

ml-evs changed the title ~~Added default implementations for HAS ALL/ANY/ONLY~~ Added default mongo implementations for HAS ALL/ANY/ONLY Feb 10, 2020

ml-evs added priority/medium Issue or PR with a consensus of medium priority transformer/MongoDB Related to filter transformer for MongoDB labels Feb 15, 2020

ml-evs force-pushed the ml-evs/list_queries branch from 0e7f41d to 5edbe34 Compare February 24, 2020 14:04

CasperWA requested changes Feb 27, 2020

View reviewed changes

ltalirz mentioned this pull request Feb 27, 2020

Checklist for OPTiMaDe v0.10.1 #29

Closed

16 tasks

ml-evs force-pushed the ml-evs/list_queries branch from f3120d7 to 76988e2 Compare February 28, 2020 12:22

CasperWA mentioned this pull request Feb 28, 2020

Use mongo for CI #196

Merged

ml-evs force-pushed the ml-evs/list_queries branch from 2cde421 to 0134082 Compare March 4, 2020 17:01

ml-evs and others added 7 commits March 4, 2020 17:03

Enable value_list and HAS ANY/ALL/ONLY

4bb2f55

Bumped filterparser test version and added HAS ONLY test

94284be

Enable mongo transformer tests for HAS ONLY/ANY/ALL

534cc8a

Enabled HAS ALL/ONLY/ANY tests for server

2437dad

Review suggestions from @CasperWA

1e35ec4

Co-Authored-By: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>

Fix defunct f-strings

cab0ec0

Co-Authored-By: Casper Welzel Andersen <43357585+CasperWA@users.noreply.github.com>

Restructured value_list to make the intention clearer

15a71f0

ml-evs added 2 commits March 4, 2020 17:03

Updated error detail string from previous change

2f77112

Added a couple of extra tests

9d153a2

ml-evs force-pushed the ml-evs/list_queries branch from 0134082 to 9d153a2 Compare March 4, 2020 17:04

ml-evs requested a review from CasperWA March 4, 2020 17:13

CasperWA reviewed Mar 4, 2020

View reviewed changes

tests/server/test_query_params.py Show resolved Hide resolved

Add note about mongomock bug

4257de4

ml-evs requested a review from CasperWA March 4, 2020 17:39

CasperWA approved these changes Mar 4, 2020

View reviewed changes

ml-evs merged commit f95f146 into master Mar 4, 2020

ml-evs deleted the ml-evs/list_queries branch March 4, 2020 17:45

ml-evs mentioned this pull request Mar 4, 2020

Skip HAS ONLY test if mongomock version <= 3.19.0 #206

Merged

CasperWA mentioned this pull request Mar 5, 2020

Up to v0.6.0 #210

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added default mongo implementations for HAS ALL/ANY/ONLY #173

Added default mongo implementations for HAS ALL/ANY/ONLY #173

ml-evs commented Feb 10, 2020 •

edited

ml-evs commented Feb 15, 2020

ml-evs commented Feb 24, 2020

CasperWA commented Feb 24, 2020

ml-evs commented Feb 24, 2020

CasperWA commented Feb 25, 2020

CasperWA left a comment

CasperWA commented Feb 27, 2020

ml-evs commented Feb 28, 2020

codecov bot commented Feb 28, 2020 •

edited

CasperWA commented Feb 28, 2020 •

edited

ml-evs commented Feb 28, 2020

CasperWA left a comment

CasperWA left a comment

Added default mongo implementations for HAS ALL/ANY/ONLY #173

Added default mongo implementations for HAS ALL/ANY/ONLY #173

Conversation

ml-evs commented Feb 10, 2020 • edited

ml-evs commented Feb 15, 2020

ml-evs commented Feb 24, 2020

CasperWA commented Feb 24, 2020

ml-evs commented Feb 24, 2020

CasperWA commented Feb 25, 2020

CasperWA left a comment

Choose a reason for hiding this comment

CasperWA commented Feb 27, 2020

ml-evs commented Feb 28, 2020

codecov bot commented Feb 28, 2020 • edited

Codecov Report

CasperWA commented Feb 28, 2020 • edited

ml-evs commented Feb 28, 2020

CasperWA left a comment

Choose a reason for hiding this comment

CasperWA left a comment

Choose a reason for hiding this comment

ml-evs commented Feb 10, 2020 •

edited

codecov bot commented Feb 28, 2020 •

edited

CasperWA commented Feb 28, 2020 •

edited