-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_cross_engine_read_write_netcdf3 is now failing on master #2050
Comments
This was referenced Apr 10, 2018
Same problem here. Full log:
With python-netCDF4 == 1.3.1, scipy == 1.0.0, netcdf == 4.4.1 Probably a conditional skip is a (short term) solution? |
The test is marked as xfail now, so if you merge in master tests should
pass.
…On Sat, Apr 14, 2018 at 5:48 AM Sebastian ***@***.***> wrote:
Same problem here. Full log:
[ 69s] =================================== FAILURES ===================================
[ 69s] ______________ GenericNetCDFDataTest.test_append_overwrite_values ______________
[ 69s]
[ 69s] self = <xarray.tests.test_backends.GenericNetCDFDataTest testMethod=test_append_overwrite_values>
[ 69s]
[ 69s] def test_append_overwrite_values(self):
[ 69s] # regression for GH1215
[ 69s] data = create_test_data()
[ 69s] with create_tmp_file(allow_cleanup_failure=False) as tmp_file:
[ 69s] self.save(data, tmp_file, mode='w')
[ 69s] data['var2'][:] = -999
[ 69s] data['var9'] = data['var2'] * 3
[ 69s] > self.save(data[['var2', 'var9']], tmp_file, mode='a')
[ 69s]
[ 69s] xarray/tests/test_backends.py:796:
[ 69s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 69s] xarray/tests/test_backends.py:162: in save
[ 69s] **kwargs)
[ 69s] xarray/core/dataset.py:1137: in to_netcdf
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/api.py:657: in to_netcdf
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/core/dataset.py:1074: in dump_to_store
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/common.py:363: in store
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/common.py:402: in set_variables
[ 69s] self.writer.add(source, target)
[ 69s] xarray/backends/common.py:265: in add
[ 69s] target[...] = source
[ 69s] xarray/backends/scipy_.py:61: in __setitem__
[ 69s] data[key] = value
[ 69s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 69s]
[ 69s] self = <scipy.io.netcdf.netcdf_variable object at 0x7f0f9970d090>
[ 69s] index = Ellipsis
[ 69s] data = array([[-999., -999., -999., -999., -999., -999., -999., -999., -999.],
[ 69s] ...999.],
[ 69s] [-999., -999., -999., -999., -999., -999., -999., -999., -999.]])
[ 69s]
[ 69s] def __setitem__(self, index, data):
[ 69s] if self.maskandscale:
[ 69s] missing_value = (
[ 69s] self._get_missing_value() or
[ 69s] getattr(data, 'fill_value', 999999))
[ 69s] self._attributes.setdefault('missing_value', missing_value)
[ 69s] self._attributes.setdefault('_FillValue', missing_value)
[ 69s] data = ((data - self._attributes.get('add_offset', 0.0)) /
[ 69s] self._attributes.get('scale_factor', 1.0))
[ 69s] data = np.ma.asarray(data).filled(missing_value)
[ 69s] if self._typecode not in 'fd' and data.dtype.kind == 'f':
[ 69s] data = np.round(data)
[ 69s]
[ 69s] # Expand data for record vars?
[ 69s] if self.isrec:
[ 69s] if isinstance(index, tuple):
[ 69s] rec_index = index[0]
[ 69s] else:
[ 69s] rec_index = index
[ 69s] if isinstance(rec_index, slice):
[ 69s] recs = (rec_index.start or 0) + len(data)
[ 69s] else:
[ 69s] recs = rec_index + 1
[ 69s] if recs > len(self.data):
[ 69s] shape = (recs,) + self._shape[1:]
[ 69s] # Resize in-place does not always work since
[ 69s] # the array might not be single-segment
[ 69s] try:
[ 69s] self.data.resize(shape)
[ 69s] except ValueError:
[ 69s] self.__dict__['data'] = np.resize(self.data, shape).astype(self.data.dtype)
[ 69s] > self.data[index] = data
[ 69s] E ValueError: assignment destination is read-only
[ 69s]
[ 69s] /usr/lib64/python2.7/site-packages/scipy/io/netcdf.py:996: ValueError
[ 69s] ___________________ GenericNetCDFDataTest.test_append_write ____________________
[ 69s]
[ 69s] self = <xarray.tests.test_backends.GenericNetCDFDataTest testMethod=test_append_write>
[ 69s]
[ 69s] def test_append_write(self):
[ 69s] # regression for GH1215
[ 69s] data = create_test_data()
[ 69s] > with self.roundtrip_append(data) as actual:
[ 69s]
[ 69s] xarray/tests/test_backends.py:786:
[ 69s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 69s] /usr/lib64/python2.7/contextlib.py:17: in __enter__
[ 69s] return self.gen.next()
[ 69s] xarray/tests/test_backends.py:155: in roundtrip_append
[ 69s] self.save(data[[key]], path, mode=mode, **save_kwargs)
[ 69s] xarray/tests/test_backends.py:162: in save
[ 69s] **kwargs)
[ 69s] xarray/core/dataset.py:1137: in to_netcdf
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/api.py:657: in to_netcdf
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/core/dataset.py:1074: in dump_to_store
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/common.py:363: in store
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/common.py:402: in set_variables
[ 69s] self.writer.add(source, target)
[ 69s] xarray/backends/common.py:265: in add
[ 69s] target[...] = source
[ 69s] xarray/backends/scipy_.py:61: in __setitem__
[ 69s] data[key] = value
[ 69s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 69s]
[ 69s] self = <scipy.io.netcdf.netcdf_variable object at 0x7f0f991df910>
[ 69s] index = Ellipsis, data = array([0. , 0.5, 1. , 1.5, 2. , 2.5, 3. , 3.5, 4. ])
[ 69s]
[ 69s] def __setitem__(self, index, data):
[ 69s] if self.maskandscale:
[ 69s] missing_value = (
[ 69s] self._get_missing_value() or
[ 69s] getattr(data, 'fill_value', 999999))
[ 69s] self._attributes.setdefault('missing_value', missing_value)
[ 69s] self._attributes.setdefault('_FillValue', missing_value)
[ 69s] data = ((data - self._attributes.get('add_offset', 0.0)) /
[ 69s] self._attributes.get('scale_factor', 1.0))
[ 69s] data = np.ma.asarray(data).filled(missing_value)
[ 69s] if self._typecode not in 'fd' and data.dtype.kind == 'f':
[ 69s] data = np.round(data)
[ 69s]
[ 69s] # Expand data for record vars?
[ 69s] if self.isrec:
[ 69s] if isinstance(index, tuple):
[ 69s] rec_index = index[0]
[ 69s] else:
[ 69s] rec_index = index
[ 69s] if isinstance(rec_index, slice):
[ 69s] recs = (rec_index.start or 0) + len(data)
[ 69s] else:
[ 69s] recs = rec_index + 1
[ 69s] if recs > len(self.data):
[ 69s] shape = (recs,) + self._shape[1:]
[ 69s] # Resize in-place does not always work since
[ 69s] # the array might not be single-segment
[ 69s] try:
[ 69s] self.data.resize(shape)
[ 69s] except ValueError:
[ 69s] self.__dict__['data'] = np.resize(self.data, shape).astype(self.data.dtype)
[ 69s] > self.data[index] = data
[ 69s] E ValueError: assignment destination is read-only
[ 69s]
[ 69s] /usr/lib64/python2.7/site-packages/scipy/io/netcdf.py:996: ValueError
[ 69s] _______ GenericNetCDFDataTestAutocloseTrue.test_append_overwrite_values ________
[ 69s]
[ 69s] self = <xarray.tests.test_backends.GenericNetCDFDataTestAutocloseTrue testMethod=test_append_overwrite_values>
[ 69s]
[ 69s] def test_append_overwrite_values(self):
[ 69s] # regression for GH1215
[ 69s] data = create_test_data()
[ 69s] with create_tmp_file(allow_cleanup_failure=False) as tmp_file:
[ 69s] self.save(data, tmp_file, mode='w')
[ 69s] data['var2'][:] = -999
[ 69s] data['var9'] = data['var2'] * 3
[ 69s] > self.save(data[['var2', 'var9']], tmp_file, mode='a')
[ 69s]
[ 69s] xarray/tests/test_backends.py:796:
[ 69s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 69s] xarray/tests/test_backends.py:162: in save
[ 69s] **kwargs)
[ 69s] xarray/core/dataset.py:1137: in to_netcdf
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/api.py:657: in to_netcdf
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/core/dataset.py:1074: in dump_to_store
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/common.py:363: in store
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/common.py:402: in set_variables
[ 69s] self.writer.add(source, target)
[ 69s] xarray/backends/common.py:265: in add
[ 69s] target[...] = source
[ 69s] xarray/backends/scipy_.py:61: in __setitem__
[ 69s] data[key] = value
[ 69s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 69s]
[ 69s] self = <scipy.io.netcdf.netcdf_variable object at 0x7f0f9aca4fd0>
[ 69s] index = Ellipsis
[ 69s] data = array([[-999., -999., -999., -999., -999., -999., -999., -999., -999.],
[ 69s] ...999.],
[ 69s] [-999., -999., -999., -999., -999., -999., -999., -999., -999.]])
[ 69s]
[ 69s] def __setitem__(self, index, data):
[ 69s] if self.maskandscale:
[ 69s] missing_value = (
[ 69s] self._get_missing_value() or
[ 69s] getattr(data, 'fill_value', 9[ 64.291734] serial8250: too much work for irq4
[ 69s] 99999))
[ 69s] self._attributes.setdefault('missing_value', missing_value)
[ 69s] self._attributes.setdefault('_FillValue', missing_value)
[ 69s] data = ((data - self._attributes.get('add_offset', 0.0)) /
[ 69s] self._attributes.get('scale_factor', 1.0))
[ 69s] data = np.ma.asarray(data).filled(missing_value)
[ 69s] if self._typecode not in 'fd' and data.dtype.kind == 'f':
[ 69s] data = np.round(data)
[ 69s]
[ 69s] # Expand data for record vars?
[ 69s] if self.isrec:
[ 69s] if isinstance(index, tuple):
[ 69s] rec_index = index[0]
[ 69s] else:
[ 69s] rec_index = index
[ 69s] if isinstance(rec_index, slice):
[ 69s] recs = (rec_index.start or 0) + len(data)
[ 69s] else:
[ 69s] recs = rec_index + 1
[ 69s] if recs > len(self.data):
[ 69s] shape = (recs,) + self._shape[1:]
[ 69s] # Resize in-place does not always work since
[ 69s] # the array might not be single-segment
[ 69s] try:
[ 69s] self.data.resize(shape)
[ 69s] except ValueError:
[ 69s] self.__dict__['data'] = np.resize(self.data, shape).astype(self.data.dtype)
[ 69s] > self.data[index] = data
[ 69s] E ValueError: assignment destination is read-only
[ 69s]
[ 69s] /usr/lib64/python2.7/site-packages/scipy/io/netcdf.py:996: ValueError
[ 69s] _____________ GenericNetCDFDataTestAutocloseTrue.test_append_write _____________
[ 69s]
[ 69s] self = <xarray.tests.test_backends.GenericNetCDFDataTestAutocloseTrue testMethod=test_append_write>
[ 69s]
[ 69s] def test_append_write(self):
[ 69s] # regression for GH1215
[ 69s] data = create_test_data()
[ 69s] > with self.roundtrip_append(data) as actual:
[ 69s]
[ 69s] xarray/tests/test_backends.py:786:
[ 69s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 69s] /usr/lib64/python2.7/contextlib.py:17: in __enter__
[ 69s] return self.gen.next()
[ 69s] xarray/tests/test_backends.py:155: in roundtrip_append
[ 69s] self.save(data[[key]], path, mode=mode, **save_kwargs)
[ 69s] xarray/tests/test_backends.py:162: in save
[ 69s] **kwargs)
[ 69s] xarray/core/dataset.py:1137: in to_netcdf
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/api.py:657: in to_netcdf
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/core/dataset.py:1074: in dump_to_store
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/common.py:363: in store
[ 69s] unlimited_dims=unlimited_dims)
[ 69s] xarray/backends/common.py:402: in set_variables
[ 69s] self.writer.add(source, target)
[ 69s] xarray/backends/common.py:265: in add
[ 69s] target[...] = source
[ 69s] xarray/backends/scipy_.py:61: in __setitem__
[ 69s] data[key] = value
[ 69s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 69s]
[ 69s] self = <scipy.io.netcdf.netcdf_variable object at 0x7f0f992c80d0>
[ 69s] index = Ellipsis, data = array([0. , 0.5, 1. , 1.5, 2. , 2.5, 3. , 3.5, 4. ])
[ 69s]
[ 69s] def __setitem__(self, index, data):
[ 69s] if self.maskandscale:
[ 69s] missing_value = (
[ 69s] self._get_missing_value() or
[ 69s] getattr(data, 'fill_value', 999999))
[ 69s] self._attributes.setdefault('missing_value', missing_value)
[ 69s] self._attributes.setdefault('_FillValue', missing_value)
[ 69s] data = ((data - self._attributes.get('add_offset', 0.0)) /
[ 69s] self._attributes.get('scale_factor', 1.0))
[ 69s] data = np.ma.asarray(data).filled(missing_value)
[ 69s] if self._typecode not in 'fd' and data.dtype.kind == 'f':
[ 69s] data = np.round(data)
[ 69s]
[ 69s] # Expand data for record vars?
[ 69s] if self.isrec:
[ 69s] if isinstance(index, tuple):
[ 69s] rec_index = index[0]
[ 69s] else:
[ 69s] rec_index = index
[ 69s] if isinstance(rec_index, slice):
[ 69s] recs = (rec_index.start or 0) + len(data)
[ 69s] else:
[ 69s] recs = rec_index + 1
[ 69s] if recs > len(self.data):
[ 69s] shape = (recs,) + self._shape[1:]
[ 69s] # Resize in-place does not always work since
[ 69s] # the array might not be single-segment
[ 69s] try:
[ 69s] self.data.resize(shape)
[ 69s] except ValueError:
[ 69s] self.__dict__['data'] = np.resize(self.data, shape).astype(self.data.dtype)
[ 69s] > self.data[index] = data
[ 69s] E ValueError: assignment destination is read-only
[ 69s]
[ 69s] /usr/lib64/python2.7/site-packages/scipy/io/netcdf.py:996: ValueError
[ 69s] =============================== warnings summary ===============================
[ 69s] xarray/tests/test_backends.py::ScipyInMemoryDataTest::test_default_fill_value
[ 69s] /home/abuild/rpmbuild/BUILD/xarray-0.10.3/xarray/conventions.py:748: SerializationWarning: saving variable x with floating point data as an integer dtype without any _FillValue to use for NaNs
[ 69s] for k, v in iteritems(variables))
[ 69s]
[ 69s] xarray/tests/test_backends.py::ScipyInMemoryDataTest::test_pickle
[ 69s] /usr/lib64/python2.7/site-packages/scipy/io/netcdf.py:299: RuntimeWarning: Cannot close a netcdf_file opened with mmap=True, when netcdf_variables or arrays referring to its data still exist. All data arrays obtained from such files refer directly to data on disk, and must be copied before the file can be cleanly closed. (See netcdf_file docstring for more information on mmap.)
[ 69s] ), category=RuntimeWarning)
[ 69s]
[ 69s] xarray/tests/test_dataarray.py::TestDataArray::test_reindex_regressions
[ 69s] /home/abuild/rpmbuild/BUILD/xarray-0.10.3/xarray/core/dataarray.py:882: FutureWarning: Indexer has dimensions ('time2',) that are different from that to be indexed along time. This will behave differently in the future.
[ 69s] method=method, tolerance=tolerance, copy=copy, **indexers)
[ 69s]
[ 69s] xarray/tests/test_missing.py::test_scipy_methods_function
[ 69s] /usr/lib64/python2.7/site-packages/scipy/interpolate/polyint.py:511: RuntimeWarning: overflow encountered in multiply
[ 69s] self.wi[:j] *= (self.xi[j]-self.xi[:j])
[ 69s] /usr/lib64/python2.7/site-packages/scipy/interpolate/polyint.py:512: RuntimeWarning: overflow encountered in reduce
[ 69s] self.wi[j] = np.multiply.reduce(self.xi[:j]-self.xi[j])
[ 69s] /usr/lib64/python2.7/site-packages/scipy/interpolate/polyint.py:609: RuntimeWarning: invalid value encountered in true_divide
[ 69s] p = np.dot(c,self.yi)/np.sum(c,axis=-1)[...,np.newaxis]
[ 69s] /usr/lib64/python2.7/site-packages/scipy/interpolate/polyint.py:324: RuntimeWarning: overflow encountered in multiply
[ 69s] pi = w*pi
[ 69s] /usr/lib64/python2.7/site-packages/scipy/interpolate/polyint.py:325: RuntimeWarning: invalid value encountered in multiply
[ 69s] p += pi[:,np.newaxis] * self.c[k]
[ 69s] /usr/lib64/python2.7/site-packages/scipy/interpolate/polyint.py:325: RuntimeWarning: invalid value encountered in add
[ 69s] p += pi[:,np.newaxis] * self.c[k]
[ 69s]
[ 69s] xarray/tests/test_variable.py::TestVariable::test_index_0d_not_a_time
[ 69s] /home/abuild/rpmbuild/BUILD/xarray-0.10.3/xarray/core/duck_array_ops.py:137: FutureWarning: In the future, 'NAT == x' and 'x == NAT' will always be False.
[ 69s] flag_array = (arr1 == arr2)
[ 69s] /home/abuild/rpmbuild/BUILD/xarray-0.10.3/xarray/tests/test_variable.py:141: FutureWarning: In the future, 'NAT == x' and 'x == NAT' will always be False.
[ 69s] assert variable.values[0] == expected_value0
[ 69s] /home/abuild/rpmbuild/BUILD/xarray-0.10.3/xarray/tests/test_variable.py:142: FutureWarning: In the future, 'NAT == x' and 'x == NAT' will always be False.
[ 69s] assert variable[0].values == expected_value0
[ 69s]
[ 69s] xarray/tests/test_variable.py::TestVariableWithDask::test_index_0d_not_a_time
[ 69s] /usr/lib/python2.7/site-packages/dask/local.py:271: FutureWarning: In the future, 'NAT == x' and 'x == NAT' will always be False.
[ 69s] return func(*args2)
[ 69s]
[ 69s] -- Docs: http://doc.pytest.org/en/latest/warnings.html
[ 69s] 4 failed, 2621 passed, 1443 skipped, 19 xfailed, 4 xpassed, 13 warnings in 44.85 seconds
With python-netCDF4 == 1.3.1, scipy == 1.0.0, netcdf == 4.4.1
Probably a conditional skip is a (short term) solution?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#2050 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABKS1j4chjZUTWaRf8LvOzCctSmtBQC2ks5tofAkgaJpZM4TOx5c>
.
|
On 2018-04-14 18:48, Stephan Hoyer wrote:
The test is marked as xfail now, so if you merge in master tests should
pass.
No, commit 9b76f21 sets the xfail for
test_cross_engine_read_write_netcdf3, but not for
|GenericNetCDFDataTest.test_append_overwrite_values
GenericNetCDFDataTest.test_append_write ||GenericNetCDFDataTestAutocloseTrue.test_append_overwrite_values|
||
|
shoyer
added a commit
to shoyer/xarray
that referenced
this issue
Feb 3, 2019
Fixes pydata#2050 I'm not quite sure what was going on, but it passes now.
1 task
jhamman
pushed a commit
that referenced
this issue
Feb 4, 2019
Fixes #2050 I'm not quite sure what was going on, but it passes now.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Only on Python 3.5 and 3.6 for now:
Here's the diff of conda packages:
The culprit is almost certainly libnetcdf 4.4.1.1 -> 4.5.0
It looks like it's basically this issue again: Unidata/netcdf-c#657
We could fix this either by skipping the tests in xarray's CI or upgrading netCDF-C on conda forge to 4.6.0 or 4.6.1.
The text was updated successfully, but these errors were encountered: