Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Fix "RuntimeError: Example is wrong shape" #412

Merged
merged 6 commits into from
Nov 16, 2021
Merged

Fix "RuntimeError: Example is wrong shape" #412

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bcb2e60
Merge branch 'bug/manger-sun' into jack/temporary-merge-with-manager-sun
JackKelly Nov 16, 2021
5ff3fe6
Merge branch 'bug/400-pandas-loc' into jack/bug/nwp-example-is-wrong-…
JackKelly Nov 16, 2021
c97cbab
reverse y index of NWPs
JackKelly Nov 16, 2021
2c4ce32
Only reverse y axis if necessary. Unittests pass.
JackKelly Nov 16, 2021
33aedec
Merge branch 'main' into jack/bug/nwp-example-is-wrong-shape
JackKelly Nov 16, 2021
b55a74e
adding comment about UKV
JackKelly Nov 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion nowcasting_dataset/data_sources/nwp/nwp_data_source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ class NWPDataSource(ZarrDataSource):
Attributes:
_data: xr.DataArray of Numerical Weather Predictions, opened by open().
x is left-to-right.
y is bottom-to-top.
y is top-to-bottom (after reversing the `y` index in open_nwp()).
consolidated: Whether or not the Zarr store is consolidated.
channels: The NWP forecast parameters to load. If None then don't filter.
See: http://cedadocs.ceda.ac.uk/1334/1/uk_model_data_sheet_lores1.pdf
Expand Down Expand Up @@ -179,8 +179,23 @@ def open_nwp(zarr_path: str, consolidated: bool) -> xr.DataArray:
zarr_path, engine="zarr", consolidated=consolidated, mode="r", chunks=None
)

# Select the "UKV" DataArray from the "nwp" Dataset.
# "UKV" is the one and only DataArray in the Zarr Dataset.
# "UKV" stands for "United Kingdom Variable", and it the UK Met Office's high-res deterministic
# NWP for the UK. All the NWP variables are represented in the `variable` dimension within
# the UKV DataArray.
ukv = nwp["UKV"]
Copy link
Contributor

@peterdudfield peterdudfield Nov 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the reversing needed for other channels?

Copy link
Contributor

@peterdudfield peterdudfield Nov 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may have miss understood here - UKV might not be a channel. Perhaps add a comment why "UKV" is selected

Copy link
Contributor Author

@JackKelly JackKelly Nov 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes, good point! UKV is the name of the (one and only) DataArray in the Zarr Dataset. You're right, UKV isn't a channel. UKV is the name of the DataArray which holds all the channels (the channels are represented as a dimension in the DataArray). I'll add a comment! Thanks!


# Reverse `y` so it's top-to-bottom (so ZarrDataSource.get_example() works correctly!)
# if necessary. Adapted from:
# https://stackoverflow.com/questions/54677161/xarray-reverse-an-array-along-one-coordinate
if ukv.y[0] < ukv.y[1]:
_LOG.warning(
"NWP y axis runs from bottom-to-top. Will reverse y axis so it runs top-to-bottom."
)
y_reversed = ukv.y[::-1]
ukv = ukv.reindex(y=y_reversed)

# Sanity checks.
# If there are any duplicated init_times then drop the duplicated init_times:
init_time = pd.DatetimeIndex(ukv["init_time"])
Expand Down