Implementation of an OPTIMADE client

[ml-evs]

An implementation of a client that pulls all known databases from the providers (or uses a defined base URL) and asynchronously queries them with a nice rich interface.

Still to-do for this PR:

• Remove non-async version and replace with throttled async (i.e. one worker) Refactored such that keeping the non-async version is not a huge burden, which makes things easier to test in Jupyter etc.
• Enable Rich output inside Jupyter and use the Jupyter async event loop
• Return results in a sensible way:
  • big dictionary broken down by base URL
  • save to disk by default
  • read caches from disk for repeated queries by default
• Add way to limit the number of responses from any single provider (to avoid e.g. downloading 4m structures from NOMAD)
• Support for response fields
• Pre-flight filter validation using the filtertransformer
• Error handling and recovery (e.g., using unsupported pagination/sort)

Future design to-do:

• Make cache work with limits and enable resumable queries
• Add direct support for @CasperWA's gateway instead of directly querying providers
• Dynamic/provider-specific filter validation (supported keywords/provider fields/query params etc)
• For big queries, add async methods for paginating through a single provider
• Optional validation of incoming data with models
• Make async work with a running event loop in a better way

[codecov]

Codecov Report

Merging #1154 (697f294) into master (afce99c) will decrease coverage by 0.45%.
The diff coverage is 83.89%.

❗ Current head 697f294 differs from pull request most recent head 94c9b6a. Consider uploading reports for the commit 94c9b6a to get more accurate results

@@            Coverage Diff             @@
##           master    #1154      +/-   ##
==========================================
- Coverage   92.41%   91.95%   -0.46%     
==========================================
  Files          68       72       +4     
  Lines        3914     4265     +351     
==========================================
+ Hits         3617     3922     +305     
- Misses        297      343      +46     

Flag	Coverage Δ
project	91.95% <83.89%> (-0.46%)	⬇️
validator	91.20% <74.85%> (-1.21%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
optimade/client/client.py	82.05% <82.05%> (ø)
optimade/client/utils.py	84.21% <84.21%> (ø)
optimade/client/cli.py	87.75% <87.75%> (ø)
optimade/client/__init__.py	100.00% <100.00%> (ø)
optimade/utils.py	98.33% <100.00%> (+21.86%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update afce99c...94c9b6a. Read the comment docs.

[ml-evs]

There is probably enough here now for people to test... not sure if we want to split some of the other features mentioned above into separate PRs.

Would be helpful to have at least a minimal implementation of this working before the workshop.

[ml-evs]

My last changes will break the tests for now, will update them on the train home!

[JPBergsma]

Good work writing this in a short time.

[ml-evs]

Thanks for the ongoing reviews @JPBergsma, I think I have addressed everything and that this is now in reasonable shape for this PR. I have reworked the wishlist of new features above and we can raise some issues to track them, when we are happy with this.

I will rebase this PR into 3 commits: implementation, tests and docs.

Have you been able to get the client running yourself yet?

[ml-evs]

Final to-do(ish):

The client now falls back to synchronous mode when a running event loop is detected, which is the case in Jupyter notebooks and in pytest.

This is a slightly tricky feature that I would like to work on in a new PR, but it does mean the test coverage for this PR will be slightly lower until we can run the async stuff properly.

[ml-evs]

Final to-do(ish):

The client now falls back to synchronous mode when a running event loop is detected, which is the case in Jupyter notebooks and in pytest.

This is a slightly tricky feature that I would like to work on in a new PR, but it does mean the test coverage for this PR will be slightly lower until we can run the async stuff properly.

Just played around with @CasperWA's test setup for the gateway without much luck. The additional trouble is that this client sets up multiple httpx.AsyncClients and executes them with asyncio.run and asyncio.gather. I've tried a few tricks with the nest_asyncio library, without much luck.

[JPBergsma]

I have still have a problem

When I execute the line: optimade-get --filter 'elements HAS "Ag"'

I get the error message:
Traceback (most recent call last): File "/home/kwibus/miniconda3/envs/optimade/bin/optimade-get", line 33, in <module> sys.exit(load_entry_point('optimade', 'console_scripts', 'optimade-get')()) File "/home/kwibus/miniconda3/envs/optimade/bin/optimade-get", line 25, in importlib_load_entry_point return next(matches).load() StopIteration

[JPBergsma]

I think I am missing files that are in this PR. So it seems Git did something wrong. I'll try to delete my local branch and to fetch it again.

[JPBergsma]

Ok, that seems to have worked the command line client is now working.
I'll try to review the changes you made tomorrow.

[ml-evs]

Just removed the CI tests for assessing coverage for the validator only; I don't think the numbers are correct (the report seems to also include all other coverage tests) and I have never found myself looking at them (cuts our build time by about 20% too).

[JPBergsma]

I could not see the comments that I made before, so I will now check whether they still apply.

[ml-evs]

I could not see the comments that I made before, so I will now check whether they still apply.

They should all be listed as "resolved" above, you can double check them

[JPBergsma]

I could not find back the pending comments for the review, so I could not see whether they still applied with the latest patches.

[JPBergsma]

Ok everything seems to work now.
I had some issues with this PR and I noticed that the first commit is from 20 hours ago even though this PR was opened 17 days ago. Perhaps this caused some of the issues I experienced, and it is perhaps also the reason the codecov report seems to give a wrong percentage. Perhaps you can try to make a new branch and PR. Hopefully the code coverage is calculated correctly in that case.

[ml-evs]

Ok everything seems to work now. I had some issues with this PR and I noticed that the first commit is from 20 hours ago even though this PR was opened 17 days ago. Perhaps this caused some of the issues I experienced, and it is perhaps also the reason the codecov report seems to give a wrong percentage. Perhaps you can try to make a new branch and PR. Hopefully the code coverage is calculated correctly in that case.

Thanks for the review, glad it is working for you now.

I will actually reintroduce those codecov tests before merging and see if they fix themselves on master, we can always remove in a separate PR.

I will also hold off on releasing v0.18.0 until the weekend; I will write some tutorial notebooks with this development version and make sure everything is working.

[ml-evs]

I have just pushed one additional docs section on querying other endpoints. If the tests pass, I rebase this PR and merge, unless you wanted to have one last look @JPBergsma.

[JPBergsma]

I think you can leave out "immediately" I do not see what it adds to the sentence

[JPBergsma]

Maybee as a more general note, You could mention that not all the databases have implemented all the feature or strictly adhere to the standard. When people are going to use this client in it's current form they will see quite some error messages popping up, so it may be good to mention it is not their fault.

[JPBergsma]

I have just pushed one additional docs section on querying other endpoints. If the tests pass, I rebase this PR and merge, unless you wanted to have one last look @JPBergsma.

I have had a look and given my comments above.

[ml-evs]

Maybee as a more general note, You could mention that not all the databases have implemented all the feature or strictly adhere to the standard. When people are going to use this client in it's current form they will see quite some error messages popping up, so it may be good to mention it is not their fault.

This is a very good point, I will put that at the top of the client guide.