Reprojection functionality #236

RubenImhoff · 2021-09-16T13:56:50Z

Thanks to @wdewettin for most of the work! This PR resolves issue #230.

Reprojection functionality for changing the projection and grid (up/downscaling possible) of input fields - generally NWP data for blending - with a destination field (the original radar field for instance). This functionality was not yet present in pysteps, but becomes necessary for blending two different data sources. Note that this implementation uses a new module, rasterio, which we may have to make an optional dependency in pysteps. It is worth reconsidering this, but at least the script is working and stable now with both the Belgian and Dutch data.

Reprojection script
Changes to KNMI radar and NWP importer to allow for correct reprojection
Changes to NWP importer and/or decorators to make sure the grid is specified for the cell centers, see issue Implement importer for NWP (netcdf) data #215.
Testing script
Documentation

… reprojection_wout_ruben

dnerini

Very nice work to all: this is an important utility functions when blending data from different sources!

I left few minor suggestions directly into the code.

rasterio: I don't mind the additional optional dependency (please remember to update the list of optional dependencies in the documentation), but since in V2 we anyway work with xarray, I would suggest to use rioxarray instead. This should simplify our code, most notably the need to loop through time, and the definition of the output DataArray at the very end. In fact, rioxarray implements a reproject_match method which looks very similar to what we are doing here.

pysteps/utils/reprojection.py

RubenImhoff · 2021-09-24T13:45:30Z

Thanks for your useful feedback, @dnerini! I didn't have time to work on it this week, but the changes will follow asap. :)

RubenImhoff · 2021-09-27T13:12:06Z

@cvelascof and @ladc, see also the description in issue #215, I have made some changes to the nwp importers to set the cell coordinates to the center of the cells. Could you check if this is also the center for the rmi and bom importers?

RubenImhoff · 2021-09-27T13:14:53Z

So far, I've sticked to rasterio instead of rioxarray. When trying rio, I kept running into projection issues. Maybe it is just me, or rio is not as stable yet as rasterio. Feel free to try it out and adjust the scripts, if you know how to do it properly!

I do agree with @dnerini that it would be nice to eventually move to rioxarray.

RubenImhoff · 2021-09-27T14:51:48Z

Tests still fail due to rasterio, I haven't figured out yet why..

cvelascof · 2021-10-01T07:14:12Z

the sample NWP datasets used for BoM are already regrided to the same projection of the BoM radar, so coordinates are at the center of the pixel

environment.yml

examples/test_steps_blending_rmi.py

dnerini · 2021-10-02T06:10:13Z

examples/test_steps_blending_rmi.py

+)
+
+importer = io.get_method(importer_name, "importer")
+R_input = io.read_timeseries(fns, importer, legacy=False, **importer_kwargs)


to import test data, you should use the helper function get_precipitation_fields
example here.

examples/test_steps_blending_rmi.py

pysteps/utils/reprojection.py

dnerini · 2021-10-02T06:22:01Z

pysteps/utils/reprojection.py

+    if src_array.attrs["yorigin"] != dst_array.attrs["yorigin"]:
+        src_array = src_array[:, ::-1, :]
+
+    for i in range(src_array["t"].shape[0]):


as already mentioned above, we could use xarray's apply_ufunc as an alternative

as reference, you can look at the new version of the utils.images.morph_opening() method:

pysteps/pysteps/utils/images.py

Line 71 in b67f8c0

field_bin_out = xr.apply_ufunc(

requirements.txt

setup.py

dnerini · 2021-10-02T06:25:46Z

So far, I've sticked to rasterio instead of rioxarray. When trying rio, I kept running into projection issues. Maybe it is just me, or rio is not as stable yet as rasterio.

Thanks for trying it out! I'm really fine with sticking to rasterio if it simplifies things to us. As you say, we can alwas decide to migrate to rioxarray later on.

RubenImhoff · 2021-10-02T07:19:06Z

Thanks a lot for looking into this, @dnerini! I'll make the changes coming Monday. :)

RubenImhoff · 2021-10-04T12:05:00Z

Hmm, I keep getting "pysteps.exceptions.MissingOptionalDependency: rasterio package is required for the reprojection tool, but it isnot installed". What am I doing wrong in the development dependencies? It is listed as an optional dependency.

dnerini · 2021-10-04T12:19:04Z

maybe there is a problem with the installation of rasterio in the test environment? Try to remove the try... except block to expose directly the original error message

RubenImhoff · 2021-10-04T13:13:00Z

Good call, that changes it in "ModuleNotFoundError: No module named 'rasterio'" unfortunately. I think I've missed an optional dependency or so for the tester, but can't find it :(

dnerini · 2021-10-04T13:16:15Z

Sorry I didn't notice that rasterio is missing from the test dependencies , look at the top of .github/workflows/test_pysteps.yaml

RubenImhoff · 2021-10-04T13:16:53Z

Timing, just found it myself too. It had to be somewhere. ;) Thanks for the help!

codecov · 2021-10-04T13:39:32Z

Codecov Report

Merging #236 (26e51a9) into pysteps-v2 (4d1fb51) will increase coverage by 0.16%.
The diff coverage is 96.38%.

@@              Coverage Diff               @@
##           pysteps-v2     #236      +/-   ##
==============================================
+ Coverage       73.67%   73.84%   +0.16%     
==============================================
  Files             146      148       +2     
  Lines           11056    11128      +72     
==============================================
+ Hits             8146     8217      +71     
- Misses           2910     2911       +1

Flag	Coverage Δ
unit_tests	`73.84% <96.38%> (+0.16%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pysteps/tests/test_io_bom_nwp.py	`100.00% <ø> (ø)`
pysteps/tests/test_io_knmi_hdf5.py	`100.00% <ø> (ø)`
pysteps/tests/test_io_knmi_nwp.py	`100.00% <ø> (ø)`
pysteps/tests/test_io_rmi_nwp.py	`100.00% <ø> (ø)`
pysteps/utils/reprojection.py	`91.89% <91.89%> (ø)`
pysteps/io/importers.py	`73.92% <100.00%> (ø)`
pysteps/io/nwp_importers.py	`90.04% <100.00%> (+0.41%)`	⬆️
pysteps/tests/test_utils_reprojection.py	`100.00% <100.00%> (ø)`
pysteps/utils/__init__.py	`100.00% <100.00%> (ø)`
pysteps/utils/interface.py	`87.69% <100.00%> (+0.39%)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d1fb51...26e51a9. Read the comment docs.

RubenImhoff · 2021-10-04T15:00:04Z

Tests succeed now! Only the tests for pysteps 3.6 fail due to rasterio looking for the gdal_version, which gives an error in only pysteps 3.6 for windows. Any ideas on how to fix this?
Other than that, the PR is probably good to go. A recommended change for later is to move from rasterio to rioxarray.

dnerini · 2021-10-05T08:22:51Z

Tests succeed now! Only the tests for pysteps 3.6 fail due to rasterio looking for the gdal_version, which gives an error in only pysteps 3.6 for windows. Any ideas on how to fix this? Other than that, the PR is probably good to go. A recommended change for later is to move from rasterio to rioxarray.

GDAL can be a real pain to install, but I thought conda would help in that sense. Maybe you can check that the right version of GDAL is installed?

Besides the error, your tests in test_utils_reprojection.py cannot pass if rasterio is not installed, which can happen as it is an optional dependency.
please add pytest.importorskip("rasterio") at the top of the test, and do the same for any other optional dependency in there.

This reverts commit bccb8fc.

* Update docstrings * More cleanup * Cleanup imports * Cleanup imports * More cleanup * Update docstrings * Update references Mention the work of Ravuri et al (2021, Nature) as an example of work using cGANs to generate ensembles * Clean up page * Reprojection functionality (#236) * Added Lesley's reprojection module to this branch * Added compatibility for three-dimensional xarrays * Add commentary to reprojection util * Changes to make reprojection of KNMI data possible * Changes after Daniele's review * Add dependencies * Changes to importers, see issue #215 * Add tests * Fix some issues * documentation * Fixes for tests * Set requirements again * Some fixes * Changes to nwp_importers after Carlos' response * Remove wrong example script * Remove rasterio dependencies from lists * First try to prevent testing error * Changes Daniele and fix knmi nwp importer * Add rasterio to tox.ini * Aesthetics * rasterio import test * Add rasterio to the test dependencies * Reset try-except functionality for rasterio import * Fix for failing test on windows python 3.6 * add importerskip rasterio Co-authored-by: Wout Dewettinck <wout.dewettinck@ugent.be> * Revert "Merge branch 'steps_blending' into pysteps-v2" (#239) This reverts commit 2c639f8, reversing changes made to bccb8fc. Co-authored-by: ned <daniele.nerini@meteoswiss.ch> Co-authored-by: dnerini <daniele.nerini@gmail.com> Co-authored-by: Wout Dewettinck <wout.dewettinck@ugent.be>

* First basic functions to implement STEPS blending * Add compute of blend means,sigmas and recompose * pysteps.io with xarray (#219) * Add xarray dependency * MCH importer returns an xarray Dataset * Remove plot lines * Remove import * Adapt readers to xarray format * Rewrite as more general decorator * Add missing metadata * Adapt io tests * Mrms bounding box (#222) * Fix bounding box coordinates * Add missing metadata * Import xarray DataArray Ignore quality field * Black * Do not hardcode metadata * Address review comments by ruben * Add a legacy option to the io functions A "legacy" options is added to revert back the importers and readers behavior to version 1. This is a temporary solution to allow the examples, and other functions, to run as usual (v1.*). Hopefully, this is will allow a easier transition into version 2 during the development process and will allow testing functions that were not updated to v2. * Fix compatibility problems with tests Many of the tests were updated to use the legacy data structures (v1). The tests that still contains issues, were tagged with a TODO message and they are skipped. This will allow incremental changes to be tested in the new v2. IMPORTANT: once the v2 branch is stable, we may remove the legacy compatibility from the code base and the tests. * Update dependencies * Ignore plugins test Co-authored-by: Andres Perez Hortal <16256571+aperezhortal@users.noreply.github.com> * Add blend_optical_flow * changes to steps blending procedure - weights according to adjusted BPS2006 method * changes to blending procedures - adjust weights from original BPS2006 method * Determine spatial correlation of NWP model forecast * First attempt to make correlations and thus weights lead time dependent (in progress..) * Change back to original BPS2006 blending formulation and add regression of skill values to climatological values for weights determination * Reformat code with Black * Skill score script imports climatological correlation-values from file now * Small changes to skill score script * Add skill score tests and an interface * Add skill score tests and an interface * Small change to docstring * Bom import xarray (#228) * Add import_bom_rf3 using xarray * Add tests to xarray version * Fix mrms importer tests * Pass **kwargs to internal functions * Add nwp_importers to read bom nwp sample data * Add bom nwp data to source file * Add tests for bom_nwp reader * Fix pystepsrc Co-authored-by: Andres Perez Hortal <16256571+aperezhortal@users.noreply.github.com> * Functions to store and compute climatological weights (#231) * Implement the functions get_default_weights, save_weights, calc_clim_weights. These functions are used to evolve the weights in the scale- and skill-dependent blending with NWP in the STEPS blending algorithm. The current weights, based on the correlations per cascade level, are regressed towards these climatological weights in the course of the forecast. These functions save the current and compute the climatological weights (a running mean of the weights of the past n days, where typically n=30). First daily averages are stored and these are then averaged over the running window of n days. * Add tests for pysteps climatological weight io and calculations. * Add path_workdir to outputs section in pystepsrc file and use it as a default path to store/retrieve blending weights. * Minor changes to docstrings, changes to skill scores and testing scripts * Completed documentation for blending clim module, cleanup. Co-authored-by: RubenImhoff <r.o.imhoff@live.nl> * Main blending module, first steps * Add simple tests * Minor changes to tester: velocity now based on rainfall field of NWP * Add utilities to decompose, store and load NWP cascades for use in blending (#232) * First version of NWP decomposition * Added saving to netCDF * Completed functions for saving and loading decomposed NWP data * Added example files for the decomposed NWP functions * Added compatibility with numpy datetime * Use default output path_workdir for tmp files in blending/utils.py. * Update documentation of NWP decomposition functions in utils.py Co-authored-by: Wout Dewettinck <wout.dewettinck@ugent.be> Co-authored-by: wdewettin <87696913+wdewettin@users.noreply.github.com> * Add importer for RMI NWP data (#234) Add importer for netcdf NWP data from RMI using xarrays. * Add test for RMI NWP data importer. * Add entry for RMI NWP data in pystepsrc. * Run black on everything: fix formatting. * Add KNMI Harmonie NWP netcdf importer and tests (#235) * Changes to v_models to make use of multiple timesteps. Changes in the velocity field over time in the NWP forecast will be taken into account now. * Fixes for KNMI importer: Add forgotten @postprocess_import() Don't call dropna on NWP data. * Avoid shadowing of pysteps.blending.utils by pysteps.utils * First attempt for probability matching and masking utility; part 1 * Changes to prob matching and masking methods; part 2 * Prob matching and masking changes; part 3. Ready for testing with real data from here on * Remove unnecessary print statements * Cleanup imports * More cleanup * Update docstrings * RMI importer for gallery example (will follow) * Reprojection functionality (#236) * Added Lesley's reprojection module to this branch * Added compatibility for three-dimensional xarrays * Add commentary to reprojection util * Changes to make reprojection of KNMI data possible * Changes after Daniele's review * Add dependencies * Changes to importers, see issue #215 * Add tests * Fix some issues * documentation * Fixes for tests * Set requirements again * Some fixes * Changes to nwp_importers after Carlos' response * Remove wrong example script * Remove rasterio dependencies from lists * First try to prevent testing error * Changes Daniele and fix knmi nwp importer * Add rasterio to tox.ini * Aesthetics * rasterio import test * Add rasterio to the test dependencies * Reset try-except functionality for rasterio import * Fix for failing test on windows python 3.6 * add importerskip rasterio Co-authored-by: Wout Dewettinck <wout.dewettinck@ugent.be> * Fixes in nwp importers * Revert "Merge branch 'steps_blending' into pysteps-v2" (#239) This reverts commit 2c639f8, reversing changes made to bccb8fc. * Merge latest version pysteps-v2 into steps_blending branch (#237) * Update docstrings * More cleanup * Cleanup imports * Cleanup imports * More cleanup * Update docstrings * Update references Mention the work of Ravuri et al (2021, Nature) as an example of work using cGANs to generate ensembles * Clean up page * Reprojection functionality (#236) * Added Lesley's reprojection module to this branch * Added compatibility for three-dimensional xarrays * Add commentary to reprojection util * Changes to make reprojection of KNMI data possible * Changes after Daniele's review * Add dependencies * Changes to importers, see issue #215 * Add tests * Fix some issues * documentation * Fixes for tests * Set requirements again * Some fixes * Changes to nwp_importers after Carlos' response * Remove wrong example script * Remove rasterio dependencies from lists * First try to prevent testing error * Changes Daniele and fix knmi nwp importer * Add rasterio to tox.ini * Aesthetics * rasterio import test * Add rasterio to the test dependencies * Reset try-except functionality for rasterio import * Fix for failing test on windows python 3.6 * add importerskip rasterio Co-authored-by: Wout Dewettinck <wout.dewettinck@ugent.be> * Revert "Merge branch 'steps_blending' into pysteps-v2" (#239) This reverts commit 2c639f8, reversing changes made to bccb8fc. Co-authored-by: ned <daniele.nerini@meteoswiss.ch> Co-authored-by: dnerini <daniele.nerini@gmail.com> Co-authored-by: Wout Dewettinck <wout.dewettinck@ugent.be> * NWP skill calculation only within radar domain * Update docs * Add example for gallery examples * Fix docstrings example * Remove additional normalization step * Fixes for the tests * update docs * changes to post-processing rainfall field and docstrings * Update contributing guidelines (#241) - Improve grammar. - Make the guide more concise. Remove unused/unnecessary rules. - Indicate more clearly which parts of the guidelines are inspired by other projects (before they were only mentioned at the end). - Change "Travis-CI" references by "GitHub Actions". * Advect noise cascade * Allow for moving domain mask of extrapolation component * minor fixes * Linear blending (#229) * Implemented linear blending function * Added example file and test * Added compatibility for NWP ensembles The PR is ready to go. Making the code xarray ready will be done in a separate PR. Co-authored-by: RubenImhoff <r.o.imhoff@live.nl> * weights calculation adjustment outside radar domain if only one model present * allow for mirroring of advected noise cascade * implementation of weights following Seed et al. (2013) * Allow for decomposed NWP precip and NWP velocity fields: part 2 * Store decomposed fields with compression * changes after first review Daniele * Remove unnecessary print statement * fixes to blending utils and implementation of blending utils tests * remove unnecessary lines * Fix one time step shift of extrapolation skill prior to blending * minor changes to blending climatology, blending weights and remove path_workdir from pystepsrc * Make NWP forecast decomposition prior to blending function optional * Use pathlib * Extract methods * Minor changes to docstrings * Access climatological skill file for multiple NWP model and date string changes to prevent errors in blending.utils Co-authored-by: Carlos Velasco <carlos.velasco@bom.gov.au> Co-authored-by: ned <daniele.nerini@meteoswiss.ch> Co-authored-by: Andres Perez Hortal <16256571+aperezhortal@users.noreply.github.com> Co-authored-by: Ruben Imhoff <Ruben.Imhoff@deltares.nl> Co-authored-by: Carlos Velasco <cvelascof@gmail.com> Co-authored-by: Lesley De Cruz <lesley.decruz+git@gmail.com> Co-authored-by: Wout Dewettinck <wout.dewettinck@ugent.be> Co-authored-by: wdewettin <87696913+wdewettin@users.noreply.github.com> Co-authored-by: Lesley De Cruz <lesley.decruz@meteo.be> Co-authored-by: dnerini <daniele.nerini@gmail.com>

wdewettin and others added 5 commits August 26, 2021 15:29

Added Lesley's reprojection module to this branch

156caa1

Added compatibility for three-dimensional xarrays

10d2856

Add commentary to reprojection util

7cde0a6

Merge remote-tracking branch 'pysteps_wout/reprojection_pySTEPS' into…

c38ecf4

… reprojection_wout_ruben

Changes to make reprojection of KNMI data possible

ae24cb7

RubenImhoff requested review from dnerini, aperezhortal, cvelascof and ladc September 16, 2021 13:57

RubenImhoff added this to In progress in pysteps-v2 via automation Sep 16, 2021

RubenImhoff added the enhancement New feature or request label Sep 16, 2021

RubenImhoff assigned ladc, RubenImhoff and wdewettin Sep 16, 2021

RubenImhoff linked an issue Sep 16, 2021 that may be closed by this pull request

Allow for reprojection and up/downscaling of DataArrays #230

Closed

dnerini reviewed Sep 19, 2021

View reviewed changes

pysteps/utils/reprojection.py Outdated Show resolved Hide resolved

pysteps/utils/reprojection.py Outdated Show resolved Hide resolved

pysteps/utils/reprojection.py Outdated Show resolved Hide resolved

pysteps/utils/reprojection.py Outdated Show resolved Hide resolved

RubenImhoff added 5 commits September 27, 2021 11:50

Changes after Daniele's review

c129944

Add dependencies

7ca46a0

Changes to importers, see issue pySTEPS#215

9f3ad66

Add tests

ea2b9f6

Fix some issues

8b3c8b2

RubenImhoff added 4 commits September 27, 2021 15:26

documentation

63706a1

Fixes for tests

d696aac

Set requirements again

f43bb04

Some fixes

0a03914

dnerini requested changes Oct 2, 2021

View reviewed changes

pysteps-v2 automation moved this from In progress to Review in progress Oct 2, 2021

RubenImhoff added 5 commits October 4, 2021 09:31

Remove wrong example script

2a27807

Remove rasterio dependencies from lists

9f8cdfa

First try to prevent testing error

b0bad80

Changes Daniele and fix knmi nwp importer

d0a93de

Add rasterio to tox.ini

019291c

Aesthetics

0c7206c

rasterio import test

3e95106

Add rasterio to the test dependencies

71d2c76

Reset try-except functionality for rasterio import

5979446

Fix for failing test on windows python 3.6

b1d5418

add importerskip rasterio

26e51a9

RubenImhoff merged commit bccb8fc into pySTEPS:pysteps-v2 Oct 5, 2021

pysteps-v2 automation moved this from Review in progress to Done Oct 5, 2021

RubenImhoff added a commit that referenced this pull request Oct 11, 2021

Revert "Reprojection functionality (#236)"

d6e819a

This reverts commit bccb8fc.

RubenImhoff deleted the reprojection_wout_ruben branch July 4, 2022 07:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reprojection functionality #236

Reprojection functionality #236

RubenImhoff commented Sep 16, 2021 •

edited

Loading

dnerini left a comment

RubenImhoff commented Sep 24, 2021

RubenImhoff commented Sep 27, 2021

RubenImhoff commented Sep 27, 2021

RubenImhoff commented Sep 27, 2021

cvelascof commented Oct 1, 2021

dnerini Oct 2, 2021

dnerini Oct 2, 2021

dnerini Oct 5, 2021

dnerini commented Oct 2, 2021

RubenImhoff commented Oct 2, 2021

RubenImhoff commented Oct 4, 2021

dnerini commented Oct 4, 2021

RubenImhoff commented Oct 4, 2021

dnerini commented Oct 4, 2021

RubenImhoff commented Oct 4, 2021

codecov bot commented Oct 4, 2021 •

edited

Loading

RubenImhoff commented Oct 4, 2021

dnerini commented Oct 5, 2021 •

edited

Loading

Reprojection functionality #236

Reprojection functionality #236

Conversation

RubenImhoff commented Sep 16, 2021 • edited Loading

dnerini left a comment

Choose a reason for hiding this comment

RubenImhoff commented Sep 24, 2021

RubenImhoff commented Sep 27, 2021

RubenImhoff commented Sep 27, 2021

RubenImhoff commented Sep 27, 2021

cvelascof commented Oct 1, 2021

dnerini Oct 2, 2021

Choose a reason for hiding this comment

dnerini Oct 2, 2021

Choose a reason for hiding this comment

dnerini Oct 5, 2021

Choose a reason for hiding this comment

dnerini commented Oct 2, 2021

RubenImhoff commented Oct 2, 2021

RubenImhoff commented Oct 4, 2021

dnerini commented Oct 4, 2021

RubenImhoff commented Oct 4, 2021

dnerini commented Oct 4, 2021

RubenImhoff commented Oct 4, 2021

codecov bot commented Oct 4, 2021 • edited Loading

Codecov Report

RubenImhoff commented Oct 4, 2021

dnerini commented Oct 5, 2021 • edited Loading

RubenImhoff commented Sep 16, 2021 •

edited

Loading

codecov bot commented Oct 4, 2021 •

edited

Loading

dnerini commented Oct 5, 2021 •

edited

Loading