Xarray test suite failing with dask-master

[shoyer]

There are a wide variety of failures, mostly related to backends and indexing, e.g., AttributeError: 'tuple' object has no attribute 'tuple'. By the looks of it, something is going wrong with xarray's internal ExplicitIndexer objects, which are getting converted into something else.

I'm pretty sure this is due to the recent merge of the Array._meta pull request: dask/dask#4543

There are 81 test failures, but my guess is that there that probably only a handful (at most) of underlying causes.

[mrocklin]

cc @pentschev

[mrocklin]

I believe that this is now resolved. Please let me know

[max-sixty]

@mrocklin unfortunately I think the tests are still failing with dask master: https://travis-ci.org/pydata/xarray/jobs/546491564

I'm not that familiar with this part of the codebase; though let me know if I can be helpful at all

[mrocklin]

OK, reproduced. I'll take a look later today. Thanks for pointing me to that @max-sixty .

[max-sixty]

We're down to 1 failing test!

        # Invalid cases
        if not use_dask or LooseVersion(dask.__version__) > LooseVersion('0.17.4'):
            with pytest.raises(TypeError):
>               xr.dot(da_a, dims='a', invalid=None)
E               Failed: DID NOT RAISE <class 'TypeError'>

https://travis-ci.org/pydata/xarray/jobs/546491564

[shoyer]

Xarray definitely had a bug with ImplicitToExplicitIndexingAdapter. See #3027 for the fix.

Unfortunately, this triggers new failures from inside dask's blockwise_meta helper:

>       return meta.astype(dtype)
E       AttributeError: 'ImplicitToExplicitIndexingAdapter' object has no attribute 'astype'

The bigger issue this surfaces is that I'm not sure whether ImplicitToExplicitIndexingAdapter (and xarray's other custom array types used for data loading) should be in meta or not. These array types overload indexing, but they don't implement the rest of NumPy's ndarray API. So it's a little strange to see them in meta (and of course methods like astype don't work on them).

On the other hand, they do need to be preserved rather than always cast to NumPy arrays. Otherwise repeated indexing like x[i][j] doesn't get optimized by dask, which turns out to be a really important optimization.

[shoyer]

We're down to 1 failing test!

        # Invalid cases
        if not use_dask or LooseVersion(dask.__version__) > LooseVersion('0.17.4'):
            with pytest.raises(TypeError):
>               xr.dot(da_a, dims='a', invalid=None)
E               Failed: DID NOT RAISE <class 'TypeError'>

I think this was an intentional change -- see dask/dask#4914.

[pentschev]

Indeed it was. Regardless, dask/dask#4954 will solve that.