Switch to Github Action and Docker based test running #841

Kirill888 · 2019-12-19T04:28:13Z

Docker for running tests

See ./docker directory

opendatacube/datacube-tests
Based on osgeo/gdal:ubuntu-small-3.0.2
Uses binary wheel for rasterio

ubuntu-full-3.0.2 + compiled rasterio is a safer combo, but ubuntu-full image is rather large and takes more than a minute to load, with ubuntu-small + binary rasterio, docker pull is quicker, less than 30s. Since ubuntu-small does not have netcdf support, binary rasterio is needed.

Contains python environment with all dependencies of datacube[s3,celery,performance,test] pre-installed, but no datacube itself. Also contains postgresql server that gets launched for integration tests.

Switch to Github Actions

Turned off travis
So far 2 work flows
- Build test docker
- Run tests using pre-built docker

Old Dockerfile

marked as deprecated
removed from docs
removed docker compose
switched to ubuntu gis ppa
don't use requirements-test.txt
don't use entry point to render config to a file that might not even be writable (datacube will read from env.vars directly so no need)

Other Changes

fix for No validation of config in ingestor #826 in ingestor
fix for Stack trace printout when system is not configured #517
fix for Read datacube configuration from a single Environment variable #829
some more tests

codecov · 2019-12-19T04:33:19Z

Codecov Report

Merging #841 into develop will increase coverage by 0.2%.
The diff coverage is 95%.

@@            Coverage Diff             @@
##           develop     #841     +/-   ##
==========================================
+ Coverage    88.66%   88.86%   +0.2%     
==========================================
  Files          112      112             
  Lines        10270    10331     +61     
==========================================
+ Hits          9106     9181     +75     
+ Misses        1164     1150     -14

Impacted Files	Coverage Δ
datacube/drivers/_tools.py	`100% <ø> (+6.66%)`	⬆️
datacube/ui/click.py	`85.43% <100%> (+0.49%)`	⬆️
datacube/scripts/ingest.py	`84.58% <80%> (+1.31%)`	⬆️
datacube/config.py	`98.26% <98.64%> (+3.17%)`	⬆️
datacube/utils/uris.py	`97.24% <0%> (-1.84%)`	⬇️
datacube/utils/documents.py	`93.96% <0%> (-0.87%)`	⬇️
datacube/api/core.py	`92.21% <0%> (+1.24%)`	⬆️
datacube/utils/dask.py	`100% <0%> (+2.27%)`	⬆️
datacube/api/_legacy/load.py	`95.37% <0%> (+6.48%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 15b8b85...fbb8894. Read the comment docs.

coveralls · 2019-12-19T05:00:44Z

Coverage increased (+0.1%) to 88.743% when pulling 85b7c3a on kk-gh-actions into 15b8b85 on develop.

- removed docker section from docs - don't user test requriements - install datacube with all features enabled - get rid of entrypoint

assembling parts from geobase, but basing off ubuntugis-unstable for now focus is running tests

This requires pinning to older pyproj since ubungis stable doesn't have PROJ6. In the unstable PPA Cascaded Union test triggers SEGFAULT inside libgeos.

Become user named odc with compatible UID/GID to src/datacube-core, remain as root if volume is owned by root.

There seems to be issue with cloudpickle, moving _echo into stand-alone function, this makes it work. Don't care about celery anyway as this whole "executor" nonsense needs to be deleted.

Also change /env ownership to odc:odc

Running chown on startup is way too costly, so make it writable by group and add odc user to that group.

or at least in python 3.5

- remove duplicate lines - catch morph product exceptions

- Check output for expected error strings - Make sure source product is present in db for src_varname checking test

One that doesn't inject config/env options

- print error message when no config file found - print error message when misspelled/missing env requested

- python3 style typing annotations - removed some python2 workarounds

Extract database configuration components from a url

Load config options from environment variables, compatible with old style docker config.

This will be used in dockers to render environment configuration to file.

Running `python -m datacube` will generated datacube config file from environment variables.

- test -> check - mark more phony targets as such

Rather then pulling down some older version from packages.dea{..} use one that is built from current checkout.

1. Check if DATACUBE_DB_URL is set, if yes use that 2. Check if any of DB_{HOSTNAME|USERNAME|PASSWORD|DATABASE|PORT} are set, then use them, this mimics current docker setup but is deprecated in favour of DATACUBE_DB_URL 3. Check files as per usual

Kirill888 · 2019-12-23T06:43:49Z

too many force pushes, will restart from fresh

Kirill888 force-pushed the kk-gh-actions branch from f011d39 to 69a6ec3 Compare December 19, 2019 04:53

Kirill888 marked this pull request as ready for review December 19, 2019 05:16

Kirill888 requested review from alexgleith and omad December 19, 2019 05:17

Kirill888 force-pushed the kk-gh-actions branch 3 times, most recently from 7ca6bbf to 6eca9f0 Compare December 23, 2019 04:51

Kirill888 added 21 commits December 23, 2019 17:27

Switch Dockerfile to use UbuntuGIS instead of NextGIS

562f14f

- removed docker section from docs - don't user test requriements - install datacube with all features enabled - get rid of entrypoint

Starting to add new docker infra

91daf5a

assembling parts from geobase, but basing off ubuntugis-unstable for now focus is running tests

Move scripts into assets + add volume, workdir + entrypoint

0f63e3f

Moved to ubuntugis stable

cebc04a

This requires pinning to older pyproj since ubungis stable doesn't have PROJ6. In the unstable PPA Cascaded Union test triggers SEGFAULT inside libgeos.

Add sudo to test runner docker

da992a2

Adding make file for docker

cca1ca9

Be more robust in shell

4d02403

Become non-root user in bootstrap script

dd9e492

Become user named odc with compatible UID/GID to src/datacube-core, remain as root if volume is owned by root.

Cleanup makefile

7117ddc

Launch db from bootstrap for running integration tests

1acbe6f

Switch to osgeo/gdal:ubuntu-full-3.0.2 as base image

259089d

Make celery tests "work"

4a10ea0

There seems to be issue with cloudpickle, moving _echo into stand-alone function, this makes it work. Don't care about celery anyway as this whole "executor" nonsense needs to be deleted.

Move tini install higher up

96a2c61

Make test docker two stage

019823a

Add github workflow for building test-runner docker

d3fe7a8

Add SKIP_DB=yes flag to bootstrap + report warning if root

cc8707a

Add github action for running tests inside docker

d22b72b

Whitelist paths for running tests

cd3546c

Install datacube in edit mode in bootstrap process

48a42c7

Also change /env ownership to odc:odc

Change /env to be root:users with rw for users

6355c65

Running chown on startup is way too costly, so make it writable by group and add odc user to that group.

Install fail drivers for testing

aeb12f9

Kirill888 added 25 commits December 23, 2019 17:27

Add --with-docker option to check-code script

80a4308

Rename workflow to 'build' update Build Status link and image

2e7583d

It's assert_called_once_with not assert_called_once

6c4cf2b

or at least in python 3.5

Validate ingestor config closer to where it's loaded

798a7ff

Verify src_varname when morphing product in ingestor

715eb09

Cleanup ingestor logic

9e4fca4

- remove duplicate lines - catch morph product exceptions

Fix validate ingestor test

dff5ef8

- Check output for expected error strings - Make sure source product is present in db for src_varname checking test

Check S3 creds

dbd25a3

Correct whitelist for main.yml workflow

99e7e67

Not testing 3.5 python anymore on travis

d961071

Add raw version of clirunner test fixture

c98cfca

One that doesn't inject config/env options

Deal with missing datacube config better

1e12d27

- print error message when no config file found - print error message when misspelled/missing env requested

Add more checks for "config missing" case

c2fca46

Cleanup in config module

13da60a

- python3 style typing annotations - removed some python2 workarounds

Add parse_connect_url method to config module

50e0dda

Extract database configuration components from a url

Cleanup more python2-isms in config.py

b3b87be

Add parse_env_params to datacube.config

a6c579c

Load config options from environment variables, compatible with old style docker config.

Add auto_config method to datacube.config

e6d1032

This will be used in dockers to render environment configuration to file.

Add datacube/__main__.py to render config from cli

9b0a259

Running `python -m datacube` will generated datacube config file from environment variables.

Remove dc_config_render from test runner docker

7ef3f7b

Cleanup docker make file

f5fc5a0

- test -> check - mark more phony targets as such

When building test runner docker use local datacube distribution

bf12351

Rather then pulling down some older version from packages.dea{..} use one that is built from current checkout.

Don't do --no-deps when installing datacube for testing

e0d5e0c

Push source/wheel dist to s3 bucket from develop branch

561f4d8

Kirill888 force-pushed the kk-gh-actions branch from de54c2d to 85b7c3a Compare December 23, 2019 06:29

Don't fail if code coverage upload fails

fbb8894

Kirill888 closed this Dec 23, 2019

mergify bot deleted the kk-gh-actions branch December 23, 2019 06:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to Github Action and Docker based test running #841

Switch to Github Action and Docker based test running #841

Kirill888 commented Dec 19, 2019 •

edited

Loading

codecov bot commented Dec 19, 2019 •

edited

Loading

coveralls commented Dec 19, 2019 •

edited

Loading

Kirill888 commented Dec 23, 2019

Switch to Github Action and Docker based test running #841

Switch to Github Action and Docker based test running #841

Conversation

Kirill888 commented Dec 19, 2019 • edited Loading

Docker for running tests

Switch to Github Actions

Old Dockerfile

Other Changes

codecov bot commented Dec 19, 2019 • edited Loading

Codecov Report

coveralls commented Dec 19, 2019 • edited Loading

Kirill888 commented Dec 23, 2019

Kirill888 commented Dec 19, 2019 •

edited

Loading

codecov bot commented Dec 19, 2019 •

edited

Loading

coveralls commented Dec 19, 2019 •

edited

Loading