feat(wind): Bias correct wind speeds based on scaling to a known average #405

coroa · 2024-11-09T17:19:15Z

Superseeds #403 .

This PR adds several functions to support bias correcting wind speeds at a given height by scaling them to a known average.

A new keyword argument windspeed_bias_correction is introduced, ie.
cutout.wind(..., windspeed_bias_correction=...) , which accepts an explicit scaling factor for wind speeds, if it is a DataArray with a height meta data attribute. This scaling factor can also be part of the cutout (and called wnd_bias_correction there).

This opens up the possibility for three usage modi with the average wind speed dataset by GWA 3.1 (Global Wind Atlas), which can be retrieved using their API https://globalwindatlas.info/en/download/gis-files (only loosely called gwa_path here)

The bias correction factor can be calculated manually for an ERA5 cutout by calling atlite.wind.calculate_windspeed_bias_correction(cutout, gwa_path). This retrieves the hourly 100m wind speeds for the years 2008-2017 (which are the GWA 3.1 support years) for the cutout from CDS to calculate a data_average, then re-projects the raster dataset that gwa_path is pointing to the cutout coordinates as real_average and then returns real_average / data_average.

We can calculate a constant global bias correction factor that we should distribute through zenodo with:

data_average = atlite.datasets.era5.retrieve_windspeed_average()
windspeed_bias_correction = atlite.wind.calculate_windspeed_bias_correction(cutout=None, real_average=gwa_path, data_average=data_average, data_crs=atlite.datasets.era5.crs)

This can be used with any cutout (with default resolution 0.25/0.25) simply by:

cutout.wind(..., windspeed_bias_correction=xr.open_dataarray("windspeed_bias_correction.nc"))

Result is small:
era5-gwa34-windspeed_bias_correction.zip (2.7MB). Should probably be added to zenodo, once the PR lands.

During cutout generation (or feature preparation) one can specify the parameter windspeed_real_average_path to add the new windspeed_bias_correction feature to the cutout:
```
cutout = atlite.Cutout(
    path="western-europe-2011.nc",
    module="era5",
    x=slice(-13.6913, 1.7712),
    y=slice(49.9096, 60.8479),
    time="2011",
    windspeed_real_average_path=gwa_path
)
cutout.prepare()
```
If windspeed_real_average_path is not provided the new feature is skipped with a warning. So this change is cleanly backwards-compatible. The default value of windspeed_bias_correction of cutout.wind is None, which is interpreted as apply the bias correction factor if it is exists in the cutout. The correction can still be disabled or required with an explicit value of cutout.wind(..., windspeed_bias_correction=False/True).

I think that is the most flexible and consistent set of functionality possible. Still open to suggestions, of course.

Closes #373 .

Bias correction factor examples

For our default western europe (ie. UK+Ireland) cutout, this means moderate onshore windspeed scalings between 10% and 70%, offshore the changes are small.

You can see here also that GWA only provides offshore data within 300km of the shoreline. Others are nan and are returned as nan by atlite as well. Is this problematic for floating offshore in pypsa-eur?

Open questions

Distribute the global windspeed bias correction factor next to the cutouts? Can someone take care of generating it centrally? It's unfortunately a massive time drain, since one needs to generate and download 10 years of global hourly wind speeds from CDS at about 36GB per year. Generated it at ZIB.
Should we add the bias correction factor to the distributed cutouts? (at https://zenodo.org/records/14936211)
Unit tests?
Documentation?

Checklist

Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
Newly introduced dependencies are added to environment.yaml, environment_docs.yaml and setup.py (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

coroa · 2025-05-02T06:56:13Z

@lkstrp Do you understand the missing schema errors in the macos test runs for the data download?

lkstrp · 2025-05-02T07:05:51Z

Ahh, we had that here in #433. Which was a big mystery to me as the PR doesn't even touch anything related

coroa · 2025-05-02T09:34:22Z

Damn, the way i retrieve the long-run average of the era5 data does not work, since the monthly means are of the vector components. rather than the wind speed. which is necessarily lower.

there is probably no way around retrieving the full hourly 100m windspeeds for several years. i think we have to provide this as an external input.

Adds a new function :py:func:`atlite.datasets.era5.retrieve_average_windspeed` to retrieve average windspeeds. This dataset function is called by :py:func:`atlite.wind.calculate_windspeed_bias_correction` to derive a bias correction which can be passed to the default wind generation. Example usage: ```python windspeed_bias_correction = atlite.wind.calculate_windspeed_bias_correction( cutout, real_average="gwa3_250_windspeed_100m.tif", height=100, ) cutout.wind( atlite.windturbines.Enercon_E101_3000kW, windspeed_bias_correction=windspeed_bias_correction, windspeed_height=100, ) ```

coroa · 2025-05-06T09:48:18Z

Ok, updated to use the hourly windspeeds and new usage modi which will allow us to distribute a global correction factor, but also embed it into the cutout.

lkstrp

I only had a quick look and I have a couple of style things. But some are probably a general issue of the architecture of atlite, to which this PR fits well, and nothing that is introduced here. But I would like to keep it in mind for the history books/resolve at some point.

Can we add unit tests to all the new methods? I know the code cov diff is a pain, but every new method should have unit tests.

And how did you solve the MacOS runner problem?

lkstrp · 2025-05-06T11:41:16Z

atlite/convert.py

 ) -> xr.DataArray:
    """
    Convert wind speeds for turbine to wind energy generation.
    """
    V, POW, hub_height, P = itemgetter("V", "POW", "hub_height", "P")(turbine)

+    ds, from_height = windm.apply_windspeed_bias_correction(


The check if apply_windspeed_bias_correction actually does something should be done here and not within the method. If it's called it should do/ raise something or should not even be called.

lkstrp · 2025-05-06T11:41:29Z

atlite/data.py

@@ -31,14 +31,15 @@ def get_features(
    tmpdir=None,
    monthly_requests=False,
    concurrent_requests=False,
+    **parameter_updates,


We can use this for internal methods, but it's much more readable and gives a better docs/api reference if we avoid this general kwargs handling

Well, we don't know at cutout creation time or in the prepare doc-string, which of these parameters exist, since they are datasets module specific. So, I don't think there is much i can do about this here.

lkstrp · 2025-05-06T11:41:40Z

atlite/datasets/era5.py

+    """
+    real_average_path = creation_parameters.get("windspeed_real_average_path")
+    if real_average_path is None:
+        logger.warning(


This warning is going to appear for every single cutout creation where the path is not specified, isn't it? Shouldn't we make it less verbose and more optional? The information should go in the docstring for the argument in Cutout() instead

Cutout creation is not happening often and it is easy to miss this new feature. Further, it is new that the preparation of an available feature is skipped, when a data argument is missing, previously the implicit dataset generation rules meant that it should raise, but raising would be to invasive here.

Therefore, I decided for it to be verbose.

Same as above, it cannot go into the Cutout, since this is an era5 specific argument.

It is already in the cutout init, so the info should be added there as well.

I still disagree that this should be a default warning, but I see your points. If you want to go with it, mention in the warning that this is a new feature from v0.5.0 and all docs would need to be updated

Ok, I am not married to the warning. Down-graded it to a debug message and copied the GWA reference also into the Cutout.init docstring.

lkstrp · 2025-05-06T11:41:45Z

atlite/datasets/era5.py

@@ -552,8 +673,10 @@ def get_data(
        If True, the monthly data requests are posted concurrently.
        Only has an effect if `monthly_requests` is True.
    **creation_parameters :
-        Additional keyword arguments. The only effective argument is 'sanitize'
-        (default True) which sets sanitization of the data on or off.
+        Additional keyword arguments.


As mentioned above

Inherent to the data preparation routines.

lkstrp · 2025-05-06T11:42:08Z

atlite/wind.py

+        Dataset containing wind speed time-series (at one or multiple heights)
+        with keys like 'wnd{height:d}m' and potentially a pre-calculated scaling
+        factor at key 'wnd_bias_correction'
+    windspeed_bias_correction : bool or DataArray, optional


I am also not a fan of these "could be anything" arguments.
False is not needed, you just don't execute the function.
None uses a default from ds if it exists, otherwise does nothing is not clean either. This logic can be done before. You either use it or you don't by not calling this method. None can be used, but if you rely on a default it should raise an Error if the data for that does not exist. Same goes for True.

There are four different possibilities, i would like to support:

Explicit externally built scaling_factor as a dataarray

Must have bias correction factor in cutout (or raise) -> True

Do not apply bias correction factor -> False

Apply bias correction factor if it is there -> None (that would be the default setting)

That is why these arguments exists, i had the impression that treating them in the convert_wind function made that too noisy, but i can move some of the code around.

The default has been chosen to not introduce any problems for existing uses.

But 2-4 can be handled before calling all those apply_... methods that may do nothing.
It just makes it much harder to follow the code. Mypy will also be a pain to implement and less helpful with these many typed variables. I am not saying that this needs to be changed here, but in an ideal world we would have most of the variables in a single type. For sure internally

Ok, moved the argument treatment up into the cutout.wind method. Does this make your mypy life less painful?

coroa · 2025-05-06T13:06:47Z

Hi @lkstrp , thanks for the comments. I think most of your issues point at us having to overhaul the cutout generation and disentangle it from Cutout's __init__ method.

I did not solve the MacOS runner failure. When i force-pushed my new changes (which were rebased on the latest master), they did not reappear. No idea.

…ut.__init__ docstring

coroa · 2025-05-07T08:10:39Z

@lkstrp I think I addressed your code style comments where adequate. Additional changes would require us to come up with a better cutout preparation routine.

I still need to add tests and update docs.

euronion · 2025-05-07T10:02:19Z

@coroa I haven't looked at the new version yet - let me know when I should review.

Thanks for the great effort! :)

coroa · 2025-05-07T10:19:05Z

Re: @euronion

@coroa I haven't looked at the new version yet - let me know when I should review.

Thanks for the great effort! :)

I expect the code to be stable now for review.

Further required efforts are unit tests and docs, that i will start tackling next week.

coroa force-pushed the wind-add-simpler-bias-correction branch from 4412bcd to 0346ceb Compare March 12, 2025 23:53

coroa mentioned this pull request Mar 13, 2025

feat(wind): Bias correct wind speeds based on scaling to a known average #403

Closed

7 tasks

coroa force-pushed the wind-add-simpler-bias-correction branch from 0346ceb to 545ac65 Compare May 2, 2025 06:50

coroa changed the title ~~feat(wind): Bias correct wind speeds based on scaling to a known average (alternative)~~ feat(wind): Bias correct wind speeds based on scaling to a known average May 2, 2025

coroa marked this pull request as ready for review May 2, 2025 06:52

coroa marked this pull request as draft May 2, 2025 09:35

coroa added 8 commits May 6, 2025 11:45

refactor: move bias correction into wind module

1d4442d

feat: add bias correction as feature to era5

daa5877

fix wind bias correction

4b020d7

Allow skipping features during cutout preparation

4abf434

fix: make data_format an explicit argument with grib as default

f9ecafa

use hourly wind speeds again and allow global download

606850e

make calculate_windspeed_bias_correction work with global

f1e154e

coroa force-pushed the wind-add-simpler-bias-correction branch from 24e94dc to f1e154e Compare May 6, 2025 09:46

coroa marked this pull request as ready for review May 6, 2025 09:46

lkstrp reviewed May 6, 2025

View reviewed changes

coroa added 4 commits May 6, 2025 15:39

Move windspeed_bias_correction argument treatment into wind method

e28eba1

Update Cutout.__init__ and Cutout.prepare docstrings

1405b16

Remove warning of skipping feature creation and add more details Cuto…

de38bec

…ut.__init__ docstring

Improve cdsapi compliance

bdb4f8e

fix bias_correction reindexing

f9f4d77

Merge branch 'master' into wind-add-simpler-bias-correction

fc33325

feat(wind): Bias correct wind speeds based on scaling to a known average #405

Are you sure you want to change the base?

feat(wind): Bias correct wind speeds based on scaling to a known average #405

Uh oh!

Conversation

coroa commented Nov 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bias correction factor examples

Open questions

Checklist

Uh oh!

coroa commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lkstrp commented May 2, 2025

Uh oh!

coroa commented May 2, 2025

Uh oh!

coroa commented May 6, 2025

Uh oh!

lkstrp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coroa May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coroa commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coroa commented May 7, 2025

Uh oh!

euronion commented May 7, 2025

Uh oh!

coroa commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

coroa commented Nov 9, 2024 •

edited

Loading

coroa commented May 2, 2025 •

edited

Loading

coroa May 6, 2025 •

edited

Loading

coroa commented May 6, 2025 •

edited

Loading

coroa commented May 7, 2025 •

edited

Loading