Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More test tools for IO pipeline + cleanups #617

Merged
merged 29 commits into from
Jan 9, 2019
Merged

More test tools for IO pipeline + cleanups #617

merged 29 commits into from
Jan 9, 2019

Conversation

Kirill888
Copy link
Member

@Kirill888 Kirill888 commented Dec 19, 2018

Reason for this pull request

  • Add more test utilities that will be used for testing data loading pipeline
    • Generate images that encode X/Y spatial coordinate in the pixel value
    • X/Y coords are normalised and rotated to avoid symmetries along X/Y axis
  • Remove Python2 compat.py
  • Remove references to six library
  • Fix some type annotations

Didn't get to start using these images for tests yet as PR grew big already.

Proposed changes

  • Use test images that encode x, y coordinates of pixel location to test that "correct enough" source pixels end up in the destination image.
  • Add integration_tests directory to pytest config file (not sure why it was omitted)

- Rotating GeoBoxes around center
- Generate image pair that contains X,Y coords of image centers
Not sure why it was omitted, took me a while to figure out why VSCode wouldn't
see these tests.
@codecov
Copy link

codecov bot commented Dec 19, 2018

Codecov Report

Merging #617 into develop will increase coverage by 0.44%.
The diff coverage is 97.67%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #617      +/-   ##
===========================================
+ Coverage     87.1%   87.55%   +0.44%     
===========================================
  Files           94       93       -1     
  Lines         8385     8411      +26     
===========================================
+ Hits          7304     7364      +60     
+ Misses        1081     1047      -34
Impacted Files Coverage Δ
datacube/utils/geometry/__init__.py 100% <ø> (ø) ⬆️
datacube/drivers/s3/storage/s3aio/s3aio.py 86.06% <ø> (-0.66%) ⬇️
datacube/utils/dates.py 77.55% <0%> (-0.89%) ⬇️
datacube/storage/netcdf_writer.py 93.08% <0%> (ø) ⬆️
datacube/utils/geometry/_base.py 98.74% <100%> (+0.03%) ⬆️
datacube/drivers/s3/storage/s3aio/s3lio.py 90.9% <100%> (-0.37%) ⬇️
datacube/scripts/dataset.py 88.14% <100%> (+0.04%) ⬆️
datacube/drivers/postgres/_connections.py 88.88% <100%> (ø) ⬆️
datacube/drivers/datasource.py 83.33% <100%> (-5.56%) ⬇️
datacube/index/_metadata_types.py 94.89% <100%> (ø) ⬆️
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2386dc...f7ff0e0. Read the comment docs.

@coveralls
Copy link

coveralls commented Dec 19, 2018

Coverage Status

Coverage increased (+0.4%) to 87.504% when pulling f7ff0e0 on io-more-tests into e2386dc on develop.

For generating test images such that you can reconstruct X,Y coordinate from
pixel value. xy_norm takes output of xy_from_gbox and normalises it with a
reversible transform such that values are in (0,1) range and gradient is not
left/right, top/down but rather at an angle.
rotation when seen from the pixel plane depends on which axis are inverted, so
mentioning that.
rather than returning dictionaries return `SimpleNamespace` from `rio_slurp` and
`write_gtiff` functions in `testutils.io.` module.
Changes resolution of GeoBox covering the same region as original
If reading with shape change geobox for output is different than geobox for
whole image.
Geobox when supplied takes precedence over resolution,offset,crs triplet
be consistent with normal read when dealing with multi-band files, i.e.
reproject all bands not just first one.
Call to `denorm()` without arguments returns Affine matrix used to denormalise
images.
dtype argument wasn't passed on/tested
Transform a bunch of points stored as x,y ndarrays from one projection to
another.
- remove no longer used constants
- remove python2 imports
- remove unused imports
- remove duplicate constants
- remove some intermediate constants
- convert old style type annotations to py3
- clean up of stale type annotations
python2 is no longer supported so removing more of the left overs
Mypy fixes in base libs

- adding `# type: ignore` on overly dynamic things like
  memoized methods.
- converting some of old-style type annotations to py3
@Kirill888 Kirill888 changed the title WIP: more tests for IO pipeline More test tools for IO pipeline + cleanups Jan 9, 2019
@Kirill888 Kirill888 requested a review from omad January 9, 2019 02:51
Copy link
Member

@omad omad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome cleanup work Kirill. It's nice to say goodbye to Python 2.

Thanks 👍

@mergify mergify bot merged commit 2675a1d into develop Jan 9, 2019
@delete-merged-branch delete-merged-branch bot deleted the io-more-tests branch January 9, 2019 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants