New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mock the erddap #252
Mock the erddap #252
Conversation
…rddap data fetcher
Implementation 1This add the following dependencies for CI environments: The mocked http server is implemented with aioresponses. This approach raises the questions:
RESULTS: 👎 |
Implementation 2I dig into the fsspec tests suite and gave a try to their implementation of a simple local server with http.server from here and used by the http store tests. This is different from the 1st implementation above because here, we don't catch the erddap requests to replace them, but we create a local http server. Therefore, to try this I had to make the erddap data fetcher configurable with regard to the server used. RESULTS: 👏 The mocked erddap server is used in:
|
- removed unused _dtype property
Possible option 'disable' for progress bar (used in CI tests)
- use mocked server in httpstore tests
- generalised the mocked erddap to other websites - replace mockec_erdda_ifremer
- simplified ftp option checking - refactoring of some tests - removed as much as possible safe_to_server_errors decorator - fix requirement of some tests
- a patch is on its way at ioos/erddapy#300 and conda-forge/conda-forge-repodata-patches-feedstock#426
- for unknown reasons this is way too long on GA !
- new ErddapPathError - proper erddap option validator - clean up some unused tests content
will merge since this has achieved the PR purpose |
The problem
As mentioned in the last release PR, CI tests suffer from very long execution delays that we suspect to come from remote server connections.
We already created a mocked FTP server in #249 but we have similar issues with all remote server connections.
NB: Dealing with remote servers in CI tests has been a difficulty since the very beginning of argopy (#26 ). Up to this point the strategy was to catch all the possible request errors that were not due to argopy but to the remote server or the connection with a specific fixture
The solution
Here, we extend the mocked server approach to the Ifremer erddap.
In other words, this PR will provide all the necessary fixture and data to mock all Ifremer erddap requests (with fsspec using the argopy httpstore). Hopefully (?) this will dramatically reduce the execution time of CI tests.