Feature request: dask.dataframe.to_xarray()

[raybellwaves]

See https://stackoverflow.com/questions/60896303/dask-convert-a-dask-dataframe-to-an-xarray-dataset/60904272#60904272

There is an equivalent method in pandas (pandas.DataFrame.to_xarray) and xarray has the method to convert to a dask.dataframe. (xarray.Dataset.to_dask_dataframe)

My motivation is I was showing how a package I developed xskillscore can be dropped in to your work flow if you are using pandas.DataFrame's (https://github.com/raybellwaves/xskillscore-tutorial/blob/master/01_Determinisitic.ipynb). Because xskillscore is an extension of xarray it works with dask. I was looking to repeat this workflow using a dask.dataframe. In the end I went with the dask.array approach (https://github.com/raybellwaves/xskillscore-tutorial/blob/master/03_Big_Data.ipynb).

@mrocklin @TomAugspurger apologies for the tagging. you may know if there are xarray devs here who may be interested.

[jrbourbeau]

Thanks for raising an issue @raybellwaves. In theory a .to_xarray() method seems in scope as there's an equivalent pandas method.

cc @shoyer @jhamman who may have thoughts on this topic

[shoyer]

We have xarray.Dataset.to_dask_dataframe, but not from_dask_dataframe yet. I would suggest implementing that method in xarray first, which dask could call. That's the equivalent of what we do for pandas's to_xarray.

I think this could probably be done, but some care would need to be taken to do so in an efficient manner.

[TomAugspurger]

@shoyer in general, how well does xarray handle arrays with unknown chunk sizes? We can optionally compute the chunk sizes if desired, but the default would result in unknown sizes.

[shoyer]

I'm not sure. In theory it could be OK if there are no coordinate labels, but I don't think we have any builtin support for unsized dimensions currently.

…

On Thu, Apr 2, 2020 at 10:49 AM Tom Augspurger ***@***.***> wrote: @shoyer <https://github.com/shoyer> in general, how well does xarray handle arrays with unknown chunk sizes? We can optionally compute the chunk sizes if desired, but the default would result in unknown sizes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6058 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJJFVTSQNOYLQ7HPUEHWRTRKTF2FANCNFSM4L2KRCWA> .

[martindurant]

XREF PR in xarray: pydata/xarray#4659