Fix docs from #491

[dnerini]

See https://app.readthedocs.org/projects/pysteps/builds/29674718/

docs rendered here: https://pysteps.readthedocs.io/en/fix-docs-491/

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.14%. Comparing base (b12c420) to head (5898c18).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #525   +/-   ##
=======================================
  Coverage   84.14%   84.14%           
=======================================
  Files         168      168           
  Lines       14507    14507           
=======================================
  Hits        12207    12207           
  Misses       2300     2300           

Flag	Coverage Δ
unit_tests	84.14% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dnerini]

I'm getting now a

FileNotFoundError: [Errno 2] No such file or directory: './aux/grid_files/dwd/icon/R19B07/icon_grid_0047_R19B07_L.nc

[RubenImhoff]

Many thanks, @dnerini! And apologies that we overlooked these small issues. Can you check if my minor change fixes the issue? The file is there, so it should be able to find the path.

[dnerini]

hey @RubenImhoff thanks for having a look!

The error is till there unfortunately:
FileNotFoundError: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/pysteps/checkouts/fix-docs-491/examples/aux/grid_files/dwd/icon/R19B07/icon_grid_0047_R19B07_L.nc'

[RubenImhoff]

hey @RubenImhoff thanks for having a look!

The error is till there unfortunately: FileNotFoundError: [Errno 2] No such file or directory: '/home/docs/checkouts/readthedocs.org/user_builds/pysteps/checkouts/fix-docs-491/examples/aux/grid_files/dwd/icon/R19B07/icon_grid_0047_R19B07_L.nc'

I see - do you know if the data from ./aux/ is copied into that folder?

[RubenImhoff]

Does your last commit make the difference, @dnerini?

[dnerini]

@RubenImhoff I think I managed to fix it with b72804c, but now the build gets killed during the compilation of the new example because of memory, see https://app.readthedocs.org/projects/pysteps/builds/29712523/

can we try to make this example more lightweight? perhaps we could halve the spatial resolution of the input data?

[RubenImhoff]

Hi @dnerini, many thanks for your fixes so far! It indeed remains challenging to run this in the examples due to the high memory needs of the ensemble data from DWD. I have now upscaled the grid resolution of both NWP and radar with a factor two. Let's see if that does the trick.

[dnerini]

Unfortunately not, still same issue with memory: https://app.readthedocs.org/projects/pysteps/builds/29715456/

Do you know how much is currently needed to run the same example locally? Could you profile it?

[RubenImhoff]

Pity! The DWD data is heavy.. I just profiled it:

Current version (upscaled with a factor 2):
Forecast: 2.3 GB
During reading and aggregation of the data: 4 GB

After upscaling to 4x coarser resolution:
Forecast: 790 MB
During reading and aggregation of the data: 4 GB

So, the upscaling helps a lot, but reading the data (and processing it) is currently the bottleneck..
An option would be to only use a crop of the data or so.

[dnerini]

Uhm i see, thanks for the profiling!

Indeed the build fails even with factor 4... but I wonder, is the data imported with double precision? Maybe you can try to pass dtype="single" to the importer?

[RubenImhoff]

Uhm i see, thanks for the profiling!

Indeed the build fails even with factor 4... but I wonder, is the data imported with double precision? Maybe you can try to pass dtype="single" to the importer?

That seems to have lowered most of the memory needs, but the line nwp_precip_rprj, nwp_metadata_rprj = ( pysteps_nwp_importers.importer_dwd_nwp.unstructured2regular( nwp_precip, nwp_metadata, radar_metadata ) ) still requires 2.1 GB. I also just halved the number of ensemble members, which further reduces the memory needs. There is now one peak of 1.6 GB left, but generally it is well below 1 GB. What is the memory we have for building the docs?

[dnerini]

build succeeded! have a look: https://pysteps.readthedocs.io/en/fix-docs-491/

now it would be a good moment to have a close look at the docs about the EnKF blending.

@m-rempel would you have time to have a close look too?

[RubenImhoff]

Thanks for the additions, @dnerini! This looks great. :)

[m-rempel]

Thank you @dnerini and @RubenImhoff for your efforts regarding the docs and especially the tutorial. I have to apologize that we missed that part of the documentation issue and also for the immense memory consumption.

I have now taken a closer look at the tutorial section and it looks really good to me! Maybe, if one wants to change something, then the description of n_tapering. Currently, even for me it seems difficult to understand. Perhaps "Number of covariance pairs (i, i +/- n_tapering) used within the covariance matrix. With n_tapering=0, only the variance of each principal component is used." or something like that is more meaningful.