-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace Datacube Explorer Database Test Infrastructure #490
Conversation
Codecov ReportBase: 86.60% // Head: 86.04% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## develop #490 +/- ##
===========================================
- Coverage 86.60% 86.04% -0.57%
===========================================
Files 23 24 +1
Lines 3017 3188 +171
===========================================
+ Hits 2613 2743 +130
- Misses 404 445 +41
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
Switching back to Draft PR. I haven't explained the testing changes anywhere near well enough, or documented the new pytest fixtures I've added. I just got excited that all the tests were passing. |
Please update this too https://github.com/opendatacube/datacube-explorer#contributing-to-integration-test |
…ey can be cleaned up
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
- Use docker to run PostgreSQL - Create 1 DB per module, allowing parallelised testing with pytest-xdist - Pass ODC DB config via environment variables instead of explicitly - Re-remove dependency on digitalearthau
Remove the `populated_odc_db` fixture which attempted to load everything in the `data/` folder into the database.
Continuing on from work started by @Ariana-B in #479 and myself in opendatacube/odc-tools#524
This PR completely redoes the way database tests work for Datacube Explorer.
Temporary Docker based PostgreSQL server
postgresql_server
, scope = module)odc_db
of scopemodule
. It creates a new database within the postgres server, named after the test module, and configures ODC to access it via environment variables and temporary configuration files.odc_test_db
, ofmodule
scope, initialises the ODC database structure within the temporary database. It also removes the database after the module is finished.Test fixture data loading
Previously, test data was loaded a little bit all over the place, using a function which was copied into multiple test modules.
This has been replaced by a new fixture called
auto_odc_db
ofmodule
scope, which looks for module variables (namedMETADATA_TYPES
,PRODUCTS
, andDATASETS
), and loads them into the current database. This allows each test module to specify exactly which data it expects to be available.Benefits
In combination, this new setup:
docker
is available.pytest
simultaneously (due to unique docker postgres instances), and multiple test modules simultaneously usingpytest-xdist
, due to unique databases being created per module.These new fixtures have been implemented in a new
cubedash.testutils.database
module, which is loaded as apytest
plugin. My intention, once proven, is to move this test module intodatacube-core
to be available for all downstream ODC based software.📚 Documentation preview 📚: https://datacube-explorer--490.org.readthedocs.build/en/490/