Indice mismatch in VectorWind when using daily data but not seasonally averaged data

[MicrMc]

Hi Andrew
I am using VectorWind to calculate divergence and absolute vorticity in model runs. Both work fine when using seasonal U and V data however when I switched to daily data extracting season of interest, DJF, it cannot calculate the VectorWind, crashing at u.transpose(apiorder) in source code and producing the error IndexError: Indice mismatch. Indices must have the same length.

My data is:

In [45]: print ctl_uwnd
x_wind / (m s-1)                    (time: 5220; latitude: 144; longitude: 192)
     Dimension coordinates:
          time                           x               -               -
          latitude                       -               x               -
          longitude                      -               -               x
     Auxiliary coordinates:
          clim_season                    x               -               -
          forecast_period                x               -               -
          month                          x               -               -
          year                           x               -               -
     Scalar coordinates:
          forecast_reference_time: 1981-09-01 00:00:00
          pressure: 200.0 hPa
     Attributes:
          Conventions: CF-1.5
          STASH: m01s15i243
          source: Data from Met Office Unified Model
          um_version: 8.4
     Cell methods:
          mean: time (1 hour)

In [46]: print ctl_vwnd
y_wind / (m s-1)                    (time: 5220; latitude: 144; longitude: 192)
     Dimension coordinates:
          time                           x               -               -
          latitude                       -               x               -
          longitude                      -               -               x
     Auxiliary coordinates:
          clim_season                    x               -               -
          forecast_period                x               -               -
          month                          x               -               -
          year                           x               -               -
     Scalar coordinates:
          forecast_reference_time: 1981-09-01 00:00:00
          pressure: 200.0 hPa
     Attributes:
          Conventions: CF-1.5
          STASH: m01s15i244
          source: Data from Met Office Unified Model
          um_version: 8.4
     Cell methods:
          mean: time (1 hour)

With the seasonal data the same lat and lon dimensions but only 58 timesteps.
What would the error be in using daily data as opposed to seasonally averaged data for VectorWind?

[ajdawson]

Please provide the full traceback for the error you are seeing.

[MicrMc]

IndexError Traceback (most recent call last) /home/users/mmccrystall/model_analysis/output/60year_model_run/wave_analysis/RWS/vort_div.py in <module>() 105 ## warnings are caught here as the files are not completely CF compliant. 106 ################################################ --> 107 ctl_w = VectorWind(ctl_uwnd, ctl_vwnd) 108 trop_w = VectorWind(trop_uwnd, trop_vwnd) 109 /usr/lib/python2.7/site-packages/windspharm/iris.pyc in __init__(self, u, v) 82 v = v.copy() 83 u.transpose(apiorder) ---> 84 v.transpose(apiorder) 85 # Records the current shape and dimension coordinates of the inputs. 86 self._ishape = u.shape /usr/lib/python2.7/site-packages/iris/cube.pyc in transpose(self, new_order) 2805 if new_order is None: 2806 new_order = np.arange(self.data.ndim)[::-1] -> 2807 elif len(new_order) != self.data.ndim: 2808 raise ValueError('Incorrect number of dimensions.') 2809 /usr/lib/python2.7/site-packages/iris/cube.pyc in data(self) 1681 if not isinstance(data, np.ndarray): 1682 try: -> 1683 data = data.masked_array() 1684 except MemoryError: 1685 msg = "Failed to create the cube's data as there was not" \ /usr/lib/python2.7/site-packages/biggus/__init__.pyc in masked_array(self) 1445 1446 def masked_array(self): -> 1447 array = self._apply_keys() 1448 # We want the shape of the result to match the shape of the 1449 # Array, so where we've ended up with an array-scalar, /usr/lib/python2.7/site-packages/biggus/__init__.pyc in _apply_keys(self) 1531 """ 1532 def _apply_keys(self): -> 1533 array = self.concrete.__getitem__(self._keys) 1534 return array 1535 /usr/lib/python2.7/site-packages/iris/fileformats/netcdf.pyc in __getitem__(self, keys) 390 variable = dataset.variables[self.variable_name] 391 # Get the NetCDF variable data and slice. --> 392 data = variable[keys] 393 finally: 394 dataset.close() /usr/lib/python2.7/site-packages/netCDF4.so in netCDF4.Variable.__getitem__ (netCDF4.c:33881)() /usr/lib/python2.7/site-packages/netCDF4_utils.pyc in _StartCountStride(elem, shape, dimensions, grp, datashape) 233 sdim.append(1) 234 else: --> 235 raise IndexError("Indice mismatch. Indices must have the same length.") 236 # Scalar 237 else: IndexError: Indice mismatch. Indices must have the same length. From: Andrew Dawson [mailto:notifications@github.com] Sent: 21 March 2017 08:19 To: ajdawson/windspharm <windspharm@noreply.github.com> Cc: McCrystall, Michelle R. <michmcr@bas.ac.uk>; Author <author@noreply.github.com> Subject: Re: [ajdawson/windspharm] Indice mismatch in VectorWind when using daily data but not seasonally averaged data (#83) Please provide the full traceback for the error you are seeing. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#83 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AJICeoPtq6EHqf_MpHkKPRYuuRZwf-3oks5rn4gIgaJpZM4MjA9V>.

…

________________________________ This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system.

________________________________

[ajdawson]

Weird. Seems to be having a problem actually loading the data within iris rather than being a problem in windspharm. Can you try putting the following line before the call to windspharm (line 107 in your script):

ctl_vwnd.data
ctl_w = VectorWind(ctl_uwnd, ctl_vwnd)

This will force iris to load the data for the ctl_vwind cube, and I'm wondering if this will trigger the same error.

[MicrMc]

This seems to have worked, thanks Adam. Just to clarify, this was a problem with iris loading the data? This is odd as it has never happened before and when looking at the cube it appears it has been loaded in. How can one conduct more robust checks to ensure that iris and actually loading the data? From: Andrew Dawson [mailto:notifications@github.com] Sent: 21 March 2017 09:22 To: ajdawson/windspharm <windspharm@noreply.github.com> Cc: McCrystall, Michelle R. <michmcr@bas.ac.uk>; Author <author@noreply.github.com> Subject: Re: [ajdawson/windspharm] Indice mismatch in VectorWind when using daily data but not seasonally averaged data (#83) Weird. Seems to be having a problem actually loading the data within iris rather than being a problem in windspharm. Can you try putting the following line before the call to windspharm (line 107 in your script): ctl_vwnd.data ctl_w = VectorWind(ctl_uwnd, ctl_vwnd) This will force iris to load the data for the ctl_vwind cube, and I'm wondering if this will trigger the same error. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#83 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AJICel6NSsQOHFZ3lK_uR_OenFwWuXt2ks5rn5akgaJpZM4MjA9V>.

…

________________________________ This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system.

________________________________

[ajdawson]

Its Andrew, and yes I think this is a problem with iris, but may be raised because windspharm makes a copy of a cube with deferred data then in-place transposes it (however you seem to be running an older version of iris, I'm not sure this would happen in the most recent version).

You should read up on the deferred loading in iris, but basically when you load a netcdf file (or other format) iris reads the metadata, but doesn't load any data values into memory, it only does this when it needs it. You can force iris to load the data by accessing the .data attribute as we did here. This makes sense for windspharm, which needs to load all the data anyway so it can be passed to a Fortran library.

[ajdawson]

I forgot, iris cubes have a method has_lazy_data() that you can call to see if data is still deferred.

[MicrMc]

Thanks Andrew (sorry for name mix-up) and thanks for the information and the help. From: Andrew Dawson [mailto:notifications@github.com] Sent: 21 March 2017 10:01 To: ajdawson/windspharm <windspharm@noreply.github.com> Cc: McCrystall, Michelle R. <michmcr@bas.ac.uk>; Author <author@noreply.github.com> Subject: Re: [ajdawson/windspharm] Indice mismatch in VectorWind when using daily data but not seasonally averaged data (#83) Its Andrew, and yes I think this is a problem with iris, but may be raised because windspharm makes a copy of a cube with deferred data then in-place transposes it (however you seem to be running an older version of iris, I'm not sure this would happen in the most recent version). You should read up on the deferred loading in iris, but basically when you load a netcdf file (or other format) iris reads the metadata, but doesn't load any data values into memory, it only does this when it needs it. You can force iris to load the data by accessing the .data attribute as we did here. This makes sense for windspharm, which needs to load all the data anyway so it can be passed to a Fortran library. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#83 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AJICevdYupNbcV0VIEVo6cYa25PXpKO8ks5rn5_ZgaJpZM4MjA9V>.

…

________________________________ This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system.

________________________________