Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with XArray master #1283

Closed
dopplershift opened this issue Jan 10, 2020 · 2 comments · Fixed by #1285
Closed

Issues with XArray master #1283

dopplershift opened this issue Jan 10, 2020 · 2 comments · Fixed by #1285
Labels
Area: Xarray Pertains to xarray integration Type: Bug Something is not working like it should
Milestone

Comments

@dopplershift
Copy link
Member

We're failing with XArray master at the moment:

=================================== FAILURES ===================================
1302_______________ test_narr_example_variable_without_grid_mapping ________________
1303
1304test_ds = <xarray.Dataset>
1305Dimensions:              (isobaric: 29, time: 1, x: 292, y: 118)
1306Coordinates:
1307  * time               ...ateModelRunDate:  1987-04-04T18:00:00Z
1308    History:                  Translated to CF-1.0 Conventions by Netcdf-Java...
1309
1310    def test_narr_example_variable_without_grid_mapping(test_ds):
1311        """Test that NARR example is parsed correctly, with x/y coordinates scaled the same."""
1312        data = test_ds.metpy.parse_cf()
1313        # Make sure that x and y coordinates are parsed correctly, rather than having unequal
1314        # scaling based on whether that variable has the grid_mapping attribute. This would
1315        # otherwise double the coordinates's shapes since xarray tries to combine the coordinates
1316        # with different scaling from differing units.
1317>       assert test_ds['x'].shape == data['lon'].metpy.x.shape
1318E       assert (292,) == (584,)
1319E         At index 0 diff: 292 != 584
1320E         Use -v to get the full diff
1321
1322tests/test_xarray.py:442: AssertionError
1323______________________ test_data_array_loc_with_ellipsis _______________________
1324
1325self = Float64Index([-3086.8056464496035, -3054.3426464496033, -3021.8796464496036,
1326              -2989.4166464496034,  -2956...535503968,   678.9023535503966,
1327                711.3653535503963],
1328             dtype='float64', name='y', length=118)
1329key = 711365.3535503963, method = None, tolerance = None
1330
1331    @Appender(_index_shared_docs["get_loc"])
1332    def get_loc(self, key, method=None, tolerance=None):
1333        if method is None:
1334            if tolerance is not None:
1335                raise ValueError(
1336                    "tolerance argument only valid if using pad, "
1337                    "backfill or nearest lookups"
1338                )
1339            try:
1340>               return self._engine.get_loc(key)
1341
1342../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/pandas/core/indexes/base.py:2897: 
1343_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
1344
1345>   ???
1346
1347pandas/_libs/index.pyx:107: 
1348_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
1349
1350>   ???
1351
1352pandas/_libs/index.pyx:131: 
1353_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
1354
1355>   ???
1356
1357pandas/_libs/hashtable_class_helper.pxi:384: 
1358_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
1359
1360>   ???
1361E   KeyError: 711365.3535503963
1362
1363pandas/_libs/hashtable_class_helper.pxi:390: KeyError
1364
1365During handling of the above exception, another exception occurred:
1366
1367test_var = <xarray.DataArray 'Temperature' (time: 1, isobaric: 29, y: 118, x: 292)>
1368[999224 values with dtype=float32]
1369Coordinate...e:  Initialized analysis product
1370    GRIB_level_type:               100
1371    GRIB_VectorComponentFlag:      gridRelative
1372
1373    def test_data_array_loc_with_ellipsis(test_var):
1374        """Test the .loc indexer using multiple Ellipses to verify expansion behavior."""
1375        truth = test_var[:, :, -1, :]
1376>       assert truth.identical(test_var.metpy.loc[..., 711.3653535503963 * units.km, ...])
1377
1378tests/test_xarray.py:545: 
1379_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
1380../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/metpy/xarray.py:440: in __getitem__
1381    return self.data_array.loc[key]
1382../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/xarray/core/dataarray.py:197: in __getitem__
1383    return self.data_array.sel(**key)
1384../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/xarray/core/dataarray.py:1069: in sel
1385    ds = self._to_temp_dataset().sel(
1386../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/xarray/core/dataset.py:2064: in sel
1387    pos_indexers, new_indexes = remap_label_indexers(
1388../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/xarray/core/coordinates.py:391: in remap_label_indexers
1389    pos_indexers, new_indexes = indexing.remap_label_indexers(
1390../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/xarray/core/indexing.py:260: in remap_label_indexers
1391    idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance)
1392../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/xarray/core/indexing.py:179: in convert_label_indexer
1393    indexer = index.get_loc(
1394../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/pandas/core/indexes/numeric.py:479: in get_loc
1395    return super().get_loc(key, method=method, tolerance=tolerance)
1396../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/pandas/core/indexes/base.py:2899: in get_loc
1397    return self._engine.get_loc(self._maybe_cast_indexer(key))
1398pandas/_libs/index.pyx:107: in pandas._libs.index.IndexEngine.get_loc
1399    ???
1400pandas/_libs/index.pyx:131: in pandas._libs.index.IndexEngine.get_loc
1401    ???
1402pandas/_libs/hashtable_class_helper.pxi:384: in pandas._libs.hashtable.Float64HashTable.get_item
1403    ???
1404_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
1405
1406>   ???
1407E   KeyError: 711365.3535503963
1408
1409pandas/_libs/hashtable_class_helper.pxi:390: KeyError
1410____________________ test_data_array_sel_kwargs_with_units _____________________
1411
1412test_var = <xarray.DataArray 'Temperature' (time: 1, isobaric: 29, y: 118, x: 292)>
1413[999224 values with dtype=float32]
1414Coordinate...e:  Initialized analysis product
1415    GRIB_level_type:               100
1416    GRIB_VectorComponentFlag:      gridRelative
1417
1418    def test_data_array_sel_kwargs_with_units(test_var):
1419        """Test .sel on the metpy accessor with kwargs and axis type."""
1420        truth = test_var.loc[:, 500.][..., 122]
1421>       selection = test_var.metpy.sel(vertical=5e4 * units.Pa, x=-16.569 * units.km,
1422                                       tolerance=1., method='nearest')
1423
1424tests/test_xarray.py:570: 
1425_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
1426../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/metpy/xarray.py:455: in sel
1427    return self._data_array.sel(indexers, method=method, tolerance=tolerance, drop=drop)
1428../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/xarray/core/dataarray.py:1069: in sel
1429    ds = self._to_temp_dataset().sel(
1430../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/xarray/core/dataset.py:2064: in sel
1431    pos_indexers, new_indexes = remap_label_indexers(
1432../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/xarray/core/coordinates.py:391: in remap_label_indexers
1433    pos_indexers, new_indexes = indexing.remap_label_indexers(
1434../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/xarray/core/indexing.py:260: in remap_label_indexers
1435    idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance)
1436../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/xarray/core/indexing.py:179: in convert_label_indexer
1437    indexer = index.get_loc(
1438../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/pandas/core/indexes/numeric.py:479: in get_loc
1439    return super().get_loc(key, method=method, tolerance=tolerance)
1440_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
1441
1442self = Float64Index([ -3977.054863757302, -3944.5918637573022, -3912.1288637573025,
1443              -3879.6658637573023,  -3847...136242698,   5437.215136242698,
1444                5469.678136242697],
1445             dtype='float64', name='x', length=292)
1446key = -16569.0, method = 'nearest', tolerance = 1.0
1447
1448    @Appender(_index_shared_docs["get_loc"])
1449    def get_loc(self, key, method=None, tolerance=None):
1450        if method is None:
1451            if tolerance is not None:
1452                raise ValueError(
1453                    "tolerance argument only valid if using pad, "
1454                    "backfill or nearest lookups"
1455                )
1456            try:
1457                return self._engine.get_loc(key)
1458            except KeyError:
1459                return self._engine.get_loc(self._maybe_cast_indexer(key))
1460        indexer = self.get_indexer([key], method=method, tolerance=tolerance)
1461        if indexer.ndim > 1 or indexer.size > 1:
1462            raise TypeError("get_loc requires scalar valued input")
1463        loc = indexer.item()
1464        if loc == -1:
1465>           raise KeyError(key)
1466E           KeyError: -16569.0
1467
1468../../../virtualenv/python3.8-dev/lib/python3.8/site-packages/pandas/core/indexes/base.py:2905: KeyError
1469______________________ test_dataset_parse_cf_varname_list ______________________
1470
1471test_ds = <xarray.Dataset>
1472Dimensions:              (isobaric: 29, time: 1, x: 292, y: 118)
1473Coordinates:
1474  * time               ...ateModelRunDate:  1987-04-04T18:00:00Z
1475    History:                  Translated to CF-1.0 Conventions by Netcdf-Java...
1476
1477    def test_dataset_parse_cf_varname_list(test_ds):
1478        """Test that .parse_cf() returns correct subset of dataset when given list of vars."""
1479        full_ds = test_ds.copy().metpy.parse_cf()
1480        partial_ds = test_ds.metpy.parse_cf(['u_wind', 'v_wind'])
1481    
1482>       assert full_ds[['u_wind', 'v_wind']].identical(partial_ds)
1483E       assert False
1484E        +  where False = <bound method Dataset.identical of <xarray.Dataset>\nDimensions:   (isobaric: 29, time: 1, x: 584, y: 236)\nCoordinates:...teModelRunDate:  1987-04-04T18:00:00Z\n    History:                  Translated to CF-1.0 Conventions by Netcdf-Java...>(<xarray.Dataset>\nDimensions:   (isobaric: 29, time: 1, x: 292, y: 118)\nCoordinates:\n  * time      (time) datetime64[ns...ateModelRunDate:  1987-04-04T18:00:00Z\n    History:                  Translated to CF-1.0 Conventions by Netcdf-Java...)
1485E        +    where <bound method Dataset.identical of <xarray.Dataset>\nDimensions:   (isobaric: 29, time: 1, x: 584, y: 236)\nCoordinates:...teModelRunDate:  1987-04-04T18:00:00Z\n    History:                  Translated to CF-1.0 Conventions by Netcdf-Java...> = <xarray.Dataset>\nDimensions:   (isobaric: 29, time: 1, x: 584, y: 236)\nCoordinates:\n  * isobaric  (isobaric) float64 1...ateModelRunDate:  1987-04-04T18:00:00Z\n    History:                  Translated to CF-1.0 Conventions by Netcdf-Java....identical
1486
1487tests/test_xarray.py:632: AssertionError
@jthielen
Copy link
Collaborator

After digging into this a bit, these failures arose with pydata/xarray#3519, which appears to have made indexes immutable. This, combined with the sentiment expressed here, makes MetPy's inplace unit conversion of coordinates

MetPy/src/metpy/xarray.py

Lines 543 to 556 in ce7b25f

def _fixup_coords(self, var):
"""Clean up the units on the coordinate variables."""
for coord_name, data_array in var.coords.items():
if (check_axis(data_array, 'x', 'y')
and not check_axis(data_array, 'longitude', 'latitude')):
try:
var.coords[coord_name].metpy.convert_units('meters')
except DimensionalityError: # Radians!
if 'crs' in var.coords:
new_data_array = data_array.copy()
height = var.coords['crs'].item()['perspective_point_height']
scaled_vals = new_data_array.metpy.unit_array * (height * units.meters)
new_data_array.metpy.unit_array = scaled_vals.to('meters')
var.coords[coord_name] = new_data_array

a bad idea. However, I don't have any idea of how to fix this except rewriting parse_cf?

@dopplershift dopplershift added Area: Xarray Pertains to xarray integration Type: Bug Something is not working like it should labels Jan 10, 2020
@dopplershift dopplershift added this to the 1.0 milestone Jan 10, 2020
@dopplershift
Copy link
Member Author

Yeah, I agree that we probably need to rework parse_cf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Xarray Pertains to xarray integration Type: Bug Something is not working like it should
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants