Generalising collections and adding ElasticsearchCollection

[ml-evs]

This PR is the spiritual successor of #339 and includes a lot of @markus1978's original work.

Closes #742.

To-do

• Create ElasticsearchCollection and move some mongo-specific stuff into the MongoCollection.
• Add the corresponding configuration options (closes Rename config variable use_real_mongo to something more general #742)
• Add tests and elasticsearch backend to CI
• Unify tests between mongo and elasticsearch collections

Known issues (to be raised pending review)

• Had to disable some of the more esoteric elasticsearch features that we already had, e.g. HAS ANY querying
• When querying a field not in the index, it returns an empty cursor, rather than an error
• Testing across all backends is slightly fragile, relying on some hardcoded checks for backend-dependent error messages

Some future to-do:

This PR opens up the opportunity to do some work, which we can track with new issues:

• Move all aliasing code into the base filtertransformer (see Move aliasing code to base transformer #743)
• Improve config to allow specification of compatible classes, not enums (see Custom configuration extensions & use standard pydantic way of loading config file #739)
  • Allow configurable elastic index without editing source file
  • Factory class for collections
• Reimplement some optional features for elasticsearch backend (e.g. HAS ONLY)

[markus1978]

I feel that much of the parameter handling (especially all the guard checks) could be done in the base class, as they are basically independent of the underlying db. I would suggest to implement find(params) in the base class and have an abstract method _run_db_query(filter_query, page_limit, page_offset, sort_field, order).

[codecov]

Codecov Report

Merging #660 (f2d97a8) into master (9306cb8) will decrease coverage by 0.10%.
The diff coverage is 93.71%.

@@            Coverage Diff             @@
##           master     #660      +/-   ##
==========================================
- Coverage   92.78%   92.67%   -0.11%     
==========================================
  Files          67       68       +1     
  Lines        3533     3645     +112     
==========================================
+ Hits         3278     3378     +100     
- Misses        255      267      +12     

Flag	Coverage Δ
project	92.67% <93.71%> (-0.11%)	⬇️
validator	92.67% <93.71%> (+29.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
optimade/filtertransformers/django.py	91.52% <ø> (ø)
optimade/filtertransformers/mongo.py	97.57% <25.00%> (ø)
optimade/filtertransformers/elasticsearch.py	83.78% <60.00%> (-5.44%)	⬇️
optimade/server/config.py	86.74% <81.25%> (-1.49%)	⬇️
...made/server/entry_collections/entry_collections.py	97.41% <94.11%> (-1.44%)	⬇️
optimade/server/entry_collections/elasticsearch.py	97.91% <97.91%> (ø)
optimade/server/data/__init__.py	75.00% <100.00%> (+5.00%)	⬆️
optimade/server/entry_collections/__init__.py	100.00% <100.00%> (ø)
optimade/server/entry_collections/mongo.py	96.15% <100.00%> (+0.84%)	⬆️
optimade/server/main.py	94.23% <100.00%> (ø)
... and 8 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9306cb8...f2d97a8. Read the comment docs.

[ml-evs]

I think this is now ready for you @markus1978... perhaps it makes sense for you to test your server with this backend as part of the review? We can then go through and improve the overall integration in future PRs (see e.g. future to-do list in top comment)

[ml-evs]

Requesting @CasperWA too in case he wants to look at the more general stuff.

[ml-evs]

Weird pydantic failures regarding the config with Python 3.9 that I can't shake off...

[CasperWA]

Several different things to touch on. But generally, I'm really optimistic about these changes and this implementation/extension to the entry collections module.

See all suggested changes and comments below.

[CasperWA]

I think we have issues when importing from the package outside specific test functions.
I see you're using it in decorators, but maybe we can get around this in another way?

[CasperWA]

Indeed, when reviewing this file, you're already using several different ways to handle the different backends. Choose one, and don't choose the one that uses decorators (because we then need to import from the package outside the test functions). And then import from the package where needed within each function.

[ml-evs]

Not sure what the problems that you're alluding to are; each pytest session is going to be testing one fixed backend so it shouldn't break anything. Importing inside each test will have the same side effect anyway, as the client will only be created once.

The different ways of handling it correspond to different outcomes: either needing to check for different error messages, or marking the tests as xfails.

[CasperWA]

I'm unsure about the import order, but I just remember there were issues with importing configuration settings and similar outside tests as opposed to inside. But if it works, it's fine, I guess.

[ml-evs]

It works fine with this current layout, i.e. testing each backend indepedently. It would be nice if we could wrap a lot of these tests with some kind of parameterized context manager that can switch between the backends (this will also be more extensible if we add more backends) but that is beyond the scope of this PR

[CasperWA]

Great job @ml-evs ! As well as the initial work @markus1978. Thanks !

[markus1978]

Thanks guys for all this work. I hope I can get around taking it for a spin soon.