NWPDataSource RuntimeError: Example is wrong shape!

[JackKelly]

2021-11-16 18:54:25,868 DEBUG processID=1365921 Opening NWP data: /mnt/storage_ssd_4tb/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/UK_Met_Office/UKV/zarr/UKV_intermediate_version_2.zarr | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/nwp/nwp_data_source.py#L169
2021-11-16 18:54:28,338 WARNING processID=1365921 NWP Zarr has 6 duplicated init_times.  Fixing... | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/nwp/nwp_data_source.py#L182
2021-11-16 18:54:28,340 WARNING processID=1365921 NWP Zarr init_time is not monotonic_increasing.  Fixing... | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/nwp/nwp_data_source.py#L189
2021-11-16 18:54:28,352 INFO processID=1365921 Fixed 12 out of order init_times. | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/nwp/nwp_data_source.py#L197
2021-11-16 18:54:28,359 DEBUG processID=1365921 NWPDataSource creating batch 1! | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/data_source.py#L199
2021-11-16 18:54:28,688 ERROR processID=1365509 Worker process nwp raised exception! | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/manager.py#L449
NoneType: None
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/jack/miniconda3/envs/nowcasting_dataset/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/data_source.py", line 202, in create_batches
    batch = self.get_batch(
  File "/home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/data_source.py", line 260, in get_batch
    examples = [future_example.result() for future_example in future_examples]
  File "/home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/data_source.py", line 260, in <listcomp>
    examples = [future_example.result() for future_example in future_examples]
  File "/home/jack/miniconda3/envs/nowcasting_dataset/lib/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/home/jack/miniconda3/envs/nowcasting_dataset/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/home/jack/miniconda3/envs/nowcasting_dataset/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/data_source.py", line 419, in get_example
    raise RuntimeError(
RuntimeError: Example is wrong shape! x_meters_center=241537.6109145787
y_meters_center=639792.3593665359
t0_dt=2020-10-09 12:30:00
times are <xarray.DataArray 'time' (time: 3)>
array(['2020-10-09T12:00:00.000000000', '2020-10-09T13:00:00.000000000',
       '2020-10-09T14:00:00.000000000'], dtype='datetime64[ns]')
Coordinates:
    init_time  datetime64[ns] 2020-10-09T09:00:00
    step       (time) timedelta64[ns] 03:00:00 04:00:00 05:00:00
  * time       (time) datetime64[ns] 2020-10-09T12:00:00 ... 2020-10-09T14:00:00
Attributes:
    long_name:      time since forecast_reference_time
    standard_name:  forecast_period
expected shape=(10, 4, 64, 64)
actual shape (10, 3, 0, 64)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jack/dev/ocf/nowcasting_dataset/scripts/prepare_ml_data.py", line 79, in <module>
    main()
  File "/home/jack/miniconda3/envs/nowcasting_dataset/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/jack/miniconda3/envs/nowcasting_dataset/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/jack/miniconda3/envs/nowcasting_dataset/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jack/miniconda3/envs/nowcasting_dataset/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/utils.py", line 155, in inner_func
    return func(*args, **kwargs)
  File "/home/jack/dev/ocf/nowcasting_dataset/scripts/prepare_ml_data.py", line 72, in main
    manager.create_batches(overwrite_batches)
  File "/home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/manager.py", line 450, in create_batches
    raise exception
RuntimeError: Example is wrong shape! x_meters_center=241537.6109145787
y_meters_center=639792.3593665359
t0_dt=2020-10-09 12:30:00
times are <xarray.DataArray 'time' (time: 3)>
array(['2020-10-09T12:00:00.000000000', '2020-10-09T13:00:00.000000000',
       '2020-10-09T14:00:00.000000000'], dtype='datetime64[ns]')
Coordinates:
    init_time  datetime64[ns] 2020-10-09T09:00:00
    step       (time) timedelta64[ns] 03:00:00 04:00:00 05:00:00
  * time       (time) datetime64[ns] 2020-10-09T12:00:00 ... 2020-10-09T14:00:00
Attributes:
    long_name:      time since forecast_reference_time
    standard_name:  forecast_period
expected shape=(10, 4, 64, 64)
actual shape (10, 3, 0, 64)

[JackKelly]

hmm, I wonder if the lack of y data is because the y coords are reversed in the new zarr?