Implement DataArray.to_dask_dataframe() #7635

dsgreen2 · 2023-03-16T11:42:06Z

Adds a method to_dask_dataframe() to convert a dataarray to a dask dataframe.

Closes Implement DataArray.to_dask_dataframe() #7409
Tests added
User visible changes (including notable bug fixes) are documented in whats-new.rst
New functions/methods are listed in api.rst

I have added the function to_dask_dataframe() in dataarray.py . This implementation is as suggested in issue #7409 . The function first converts the data array to a temporary dataset and then calls Dataset.to_dask_dataframe() method.

Could you please review it . Thank you.

…ssue pydata#7409

headtr1ck · 2023-03-16T14:18:12Z

xarray/core/dataarray.py

+        Examples
+        --------
+
+        da=xr.DataArray(np.random.rand(4,3,2),


This example is not correctly formatted, see other functions as reference.

Also, don't use random values in examples, simply use np.ones(...)

np.ones can hide errors when dealing with tricky shapes so something like np.arange(4*3*2).reshape(4,3,2) is a little better.

headtr1ck · 2023-03-16T14:18:44Z

xarray/tests/test_dataarray.py

@@ -3205,6 +3205,39 @@ def test_to_dataframe_0length(self) -> None:
        assert len(actual) == 0
        assert_array_equal(actual.index.names, list("ABC"))

+    def test_to_dask_dataframe(self) -> None:
+        arr_np = np.random.randn(3, 4)


Even though it doesn't really matter in most cases, we try to avoid random values in tests.

Maybe use np.arange(3*4).reshape(3,4).

headtr1ck · 2023-03-16T14:20:46Z

xarray/core/dataarray.py

+        if self.ndim == 0:
+            raise ValueError("Cannot convert a scalar to a dataframe")
+
+        tmp_dataset = Dataset({name: self})


Normally we use the to tmp dataset method here, but since we only use it to construct the data frame and don't roundtrip it doesn't actually matter?

xarray/tests/test_dataarray.py

xarray/core/dataarray.py

Illviljan · 2023-03-16T15:34:25Z

xarray/core/dataarray.py

+        dim_order: Sequence of Hashable or None , optional
+        Hierarchical dimension order for the resulting dataframe.


Suggested change

dim_order: Sequence of Hashable or None , optional

Hierarchical dimension order for the resulting dataframe.

dim_order: Sequence of Hashable or None , optional

Hierarchical dimension order for the resulting dataframe.

Follow numpys docstring conventions. More errors above and below.

xarray/core/dataarray.py

Illviljan · 2023-03-16T15:36:12Z

xarray/core/dataarray.py

+        if name is None:
+            name = self.name
+
+        if name is None:
+            raise ValueError(
+                "Cannot convert an unnamed DataArray to a "
+                "dask dataframe : use the ``name`` parameter"
+            )


Suggested change

if name is None:

name = self.name

if name is None:

raise ValueError(

"Cannot convert an unnamed DataArray to a "

"dask dataframe : use the ``name`` parameter"

)

Not needed when using self._to_dataset_whole.

Can it be better to keep this error message ? When I removed it, The error shown was ' unable to convert unnamed DataArray to a Dataset without providing an explicit name ' . Keeping these lines can show the error message specific to dataarray to daskdataframe conversion.

xarray/core/dataarray.py

Illviljan · 2023-03-16T15:54:08Z

xarray/core/dataarray.py

+        Examples
+        --------
+
+        da=xr.DataArray(np.random.rand(4,3,2),


np.ones can hide errors when dealing with tricky shapes so something like np.arange(4*3*2).reshape(4,3,2) is a little better.

dsgreen2 · 2023-03-19T16:38:32Z

I have made the changes as suggested. Please review them .Thanks

Illviljan · 2023-03-19T17:14:35Z

xarray/core/dataarray.py

+        if name is None:
+            name = self.name
+
+        if name is None:
+            raise ValueError(
+                "Cannot convert an unnamed DataArray to a "
+                "dask dataframe : use the ``name`` parameter ."
+            )
+        ds = self._to_dataset_whole(name)
+        return ds.to_dask_dataframe(dim_order, set_index)


Suggested change

if name is None:

name = self.name

if name is None:

raise ValueError(

"Cannot convert an unnamed DataArray to a "

"dask dataframe : use the ``name`` parameter ."

)

ds = self._to_dataset_whole(name)

return ds.to_dask_dataframe(dim_order, set_index)

name = self.name if self.name is not None else _THIS_ARRAY

ds = self._to_dataset_whole(name, shallow_copy=False)

return ds.to_dask_dataframe(dim_order, set_index)

I think we go with this. I don't think it should be necessary to name the dataarray which is more in line with how self._to_temp_dataset works and setting dataarray.name = "new_name" is easy enough.

Illviljan · 2023-03-19T17:16:30Z

xarray/core/dataarray.py

+
+        name : Hashable or None, optional
+            Name given to this array(required if unnamed).
+            It is a keyword-only argument. A keyword-only argument can only be passed
+            to the function using its name as a keyword argument , and not as a
+            positional argument.
+


Suggested change

name : Hashable or None, optional

Name given to this array(required if unnamed).

It is a keyword-only argument. A keyword-only argument can only be passed

to the function using its name as a keyword argument , and not as a

positional argument.

Illviljan · 2023-03-19T17:16:56Z

xarray/core/dataarray.py

+        *,
+        name: Hashable | None = None,


Suggested change

*,

name: Hashable | None = None,

I have made the changes .Please review them.

…ssue pydata#7409

…n2/xarray into method-dataarray-to-daskdataframe Updating branch doc/whats-new.rst

Illviljan

There's still an issue with the docstring example, probably some whitespace mismatch somewhere. It should just be copy/pasting the results from the ipython console.

Illviljan · 2023-03-27T18:03:06Z

doc/whats-new.rst

@@ -70,6 +71,7 @@ New Features
  By `Deepak Cherian <https://github.com/dcherian>`_.
 - Improved performance in ``open_dataset`` for datasets with large object arrays (:issue:`7484`, :pull:`7494`).
  By `Alex Goodman <https://github.com/agoodm>`_ and `Deepak Cherian <https://github.com/dcherian>`_.
+- Added new method :py:meth:`DataArray.to_dask_dataframe`,convert a dataarray into a dask dataframe (:issue:`7409`).


xarray/core/dataarray.py

Illviljan · 2023-03-27T18:05:28Z

xarray/core/dataarray.py

+            vectors in contiguous order , so the last dimension in this list
+            will be contiguous in the resulting DataFrame. This has a major influence
+            on which operations are efficient on the resulting dask dataframe.
+


Suggested change

I have made the changes .Please review them. Thanks

dcherian · 2023-04-18T22:38:19Z

xarray/core/dataarray.py

+        if self.name is None:
+            raise ValueError(
+                "Cannot convert an unnamed DataArray to a "
+                "dask dataframe : use the ``name`` parameter ."


Suggested change

"dask dataframe : use the ``name`` parameter ."

"dask dataframe : use the ``.rename`` method to assign a name."

xarray/tests/test_dataarray.py

for more information, see https://pre-commit.ci

xarray/tests/test_dataarray.py

doc/whats-new.rst

dcherian · 2023-04-28T14:29:52Z

Thanks for your patience here @dsgreen2 . This is a nice contribution. Welcome to Xarray!

for more information, see https://pre-commit.ci

welcome · 2023-04-28T15:09:31Z

Congratulations on completing your first pull request! Welcome to Xarray! We are proud of you, and hope to see you again!

* main: Introduce Grouper objects internally (pydata#7561) [skip-ci] Add cftime groupby, resample benchmarks (pydata#7795) Fix groupby binary ops when grouped array is subset relative to other (pydata#7798) adjust the deprecation policy for python (pydata#7793) [pre-commit.ci] pre-commit autoupdate (pydata#7803) Allow the label run-upstream to run upstream CI (pydata#7787) Update asv links in contributing guide (pydata#7801) Implement DataArray.to_dask_dataframe() (pydata#7635) `ds.to_dict` with data as arrays, not lists (pydata#7739) Add lshift and rshift operators (pydata#7741) Use canonical name for set_horizonalalignment over alias set_ha (pydata#7786) Remove pandas<2 pin (pydata#7785) [pre-commit.ci] pre-commit autoupdate (pydata#7783)

dsgreen2 and others added 3 commits March 14, 2023 20:34

Add feature to convert dataarray to dask dataframe. This is for the i…

fe5e7ad

…ssue pydata#7409

Add test for new method dataarray.to_dask_dataframe()

4174d50

Merge branch 'main' into method-dataarray-to-daskdataframe

ae32e83

headtr1ck reviewed Mar 16, 2023

View reviewed changes

xarray/tests/test_dataarray.py Show resolved Hide resolved

Illviljan reviewed Mar 16, 2023

View reviewed changes

dsgreen2 and others added 5 commits March 18, 2023 01:10

Changes after review

ba47c16

Merge branch 'main' into method-dataarray-to-daskdataframe

924d2dd

Updating method-dataarray-to-daskdataframe branch with remote branch

3b9165c

Corrections in docstring and import

ed559d3

docstring correction

65f9fab

Illviljan reviewed Mar 19, 2023

View reviewed changes

dsgreen2 and others added 15 commits March 21, 2023 22:03

Remove name parameter

bffe3cb

Add feature to convert dataarray to dask dataframe. This is for the i…

a7e423c

…ssue pydata#7409

Add test for new method dataarray.to_dask_dataframe()

5c7a48f

Changes after review

43a8b9b

Corrections in docstring and import

aff8eca

docstring correction

9e4f816

Remove name parameter

bcd06c8

Corrected doc/whats-new.rst

4cca0a8

Merge branch 'method-dataarray-to-daskdataframe' of github.com:dsgree…

a2ee962

…n2/xarray into method-dataarray-to-daskdataframe Updating branch doc/whats-new.rst

Merge branch 'main' into method-dataarray-to-daskdataframe

34a2ac5

Merge branch 'main' into method-dataarray-to-daskdataframe

8c50d1e

Update whats-new.rst

2c9ffb3

Merge branch 'pydata:main' into method-dataarray-to-daskdataframe

88f10bc

update

b19cd6f

Merge branch 'main' into method-dataarray-to-daskdataframe

1db928f

Illviljan reviewed Mar 27, 2023

View reviewed changes

Merge branch 'main' into method-dataarray-to-daskdataframe

aa6c123

Merge branch 'main' into method-dataarray-to-daskdataframe

e577a1c

dcherian reviewed Apr 18, 2023

View reviewed changes

xarray/tests/test_dataarray.py Show resolved Hide resolved

dcherian reviewed Apr 18, 2023

View reviewed changes

xarray/tests/test_dataarray.py Outdated Show resolved Hide resolved

dsgreen2 and others added 10 commits April 19, 2023 10:19

Merge branch 'main' into method-dataarray-to-daskdataframe

d07a3c6

Add test for unnamed dataarray.

694ba77

Update remote branch

53e9b4d

[pre-commit.ci] auto fixes from pre-commit.com hooks

4866d8b

for more information, see https://pre-commit.ci

Remove scalar array test

c563989

Remove scalar test

c2acb6f

[pre-commit.ci] auto fixes from pre-commit.com hooks

87e993d

for more information, see https://pre-commit.ci

Change error message

b60a226

Change unnamed array error

4552d18

[pre-commit.ci] auto fixes from pre-commit.com hooks

2466903

for more information, see https://pre-commit.ci

dcherian reviewed Apr 20, 2023

View reviewed changes

xarray/tests/test_dataarray.py Outdated Show resolved Hide resolved

dcherian and others added 6 commits April 20, 2023 11:58

Update xarray/tests/test_dataarray.py

8bfbf3f

Merge branch 'main' into method-dataarray-to-daskdataframe

6eb67b3

Update whats-new.rst

1f67c6f

Update whats-new.rst

8b5c607

Merge branch 'main' into method-dataarray-to-daskdataframe

7c2e50c

Merge branch 'main' into method-dataarray-to-daskdataframe

8116bb1

dcherian reviewed Apr 28, 2023

View reviewed changes

doc/whats-new.rst Show resolved Hide resolved

dcherian added 2 commits April 28, 2023 08:28

Update doc/whats-new.rst

e80ece0

Merge branch 'main' into method-dataarray-to-daskdataframe

dd58648

dcherian enabled auto-merge (squash) April 28, 2023 14:29

[pre-commit.ci] auto fixes from pre-commit.com hooks

7ed49ad

for more information, see https://pre-commit.ci

dcherian merged commit 25d9a28 into pydata:main Apr 28, 2023
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement DataArray.to_dask_dataframe() #7635

Implement DataArray.to_dask_dataframe() #7635

dsgreen2 commented Mar 16, 2023 •

edited

headtr1ck Mar 16, 2023

Illviljan Mar 16, 2023

headtr1ck Mar 16, 2023

headtr1ck Mar 16, 2023

Illviljan Mar 16, 2023

Illviljan Mar 16, 2023

dsgreen2 Mar 19, 2023

Illviljan Mar 16, 2023

dsgreen2 commented Mar 19, 2023

Illviljan Mar 19, 2023

Illviljan Mar 19, 2023

Illviljan Mar 19, 2023

dsgreen2 Mar 25, 2023

Illviljan left a comment

Illviljan Mar 27, 2023

Illviljan Mar 27, 2023

dsgreen2 Apr 1, 2023 •

edited

dcherian Apr 18, 2023

dcherian commented Apr 28, 2023 •

edited

welcome bot commented Apr 28, 2023

		dim_order: Sequence of Hashable or None , optional
		Hierarchical dimension order for the resulting dataframe.

	"dask dataframe : use the ``name`` parameter ."
	"dask dataframe : use the ``.rename`` method to assign a name."

Implement DataArray.to_dask_dataframe() #7635

Implement DataArray.to_dask_dataframe() #7635

Conversation

dsgreen2 commented Mar 16, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsgreen2 commented Mar 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Illviljan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsgreen2 Apr 1, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcherian commented Apr 28, 2023 • edited

welcome bot commented Apr 28, 2023

dsgreen2 commented Mar 16, 2023 •

edited

dsgreen2 Apr 1, 2023 •

edited

dcherian commented Apr 28, 2023 •

edited