Add `DataFrame.from_dict` classmethod #9017

MrPowers · 2022-05-03T17:03:30Z

Closes Make it easier to manually create DataFrames #9009
Tests added / passed
Passes pre-commit run --all-files

I'm not sure this was added in the right files / properly, so feel free to provide feedback! Thank you!

GPUtester · 2022-05-03T17:03:32Z

Can one of the admins verify this patch?

quasiben · 2022-05-03T17:06:27Z

add to allowlist

douglasdavis

This indeed looks like a nice convenience (and covers more area of the pandas API). I had a few comments.

dask/dataframe/io/io.py

pavithraes · 2022-05-03T18:38:53Z

@MrPowers Thanks for this PR, it looks great! I think we can also add this function to the docs here:

dask/dataframe/io/io.py

MrPowers · 2022-05-04T01:08:40Z

@pavithraes - good catch about adding this method to the docs. I think I added it properly. Can you take a look and confirm? Thanks!

pavithraes · 2022-05-04T12:33:58Z

docs/source/dataframe-create.rst

@@ -33,6 +33,7 @@ File Formats:
    read_fwf
    from_bcolz
    from_array
+    from_dict


Just a thought, maybe from_dict might be better suited under a different heading "Python Collections"?

This works as-is too because from_array also doesn't perfectly fit under "file formats". I'll leave it to you. :)

@pavithraes - this is a good point. I'm actually not entirely sure how from_array works, so not sure the best categorization. I'm guessing it's not referring to a Dask Array cause there is a separate from_dask_array method. I guess I'd argue to keep this as-is, mainly cause I don't know enough to have an opinion.

pavithraes · 2022-05-05T12:33:30Z

@MrPowers Looking at the CI failure, I think there might be some merge conflicts -- would you mind merging main into this branch and checking for any conflicts?

MrPowers · 2022-05-05T12:49:47Z

@pavithraes - I rebased on top of the latest version of main, let's see if that does the trick ;)

pavithraes · 2022-05-05T14:33:34Z

@MrPowers Thanks!

@jrbourbeau The failing windows-py3.10 test seems unrelated to this PR, but it's not a known flaky test. Could you please help confirm? Other than that, we might be good to merge :)

jrbourbeau

Thanks @MrPowers!

dask/dataframe/io/io.py

jrbourbeau · 2022-05-05T14:56:28Z

dask/dataframe/io/tests/test_io.py

+        {"num1": [1, 2, 3, 4], "num2": [7, 8, 9, 10]},
+    )
+    expected = dd.from_pandas(pandas_df, npartitions=2)
+    assert_eq(actual, expected)


This is comparing that dd.from_dict and dd.from_pandas are giving a similar result. This is true, but an implementation detail. The check that we want here is that dd.from_dict and the corresponding pandas method (i.e. pd.DataFrame.from_dict) give the same result.

jrbourbeau · 2022-05-05T14:57:43Z

dask/dataframe/io/tests/test_io.py

@@ -241,6 +241,15 @@ def test_from_bcolz_column_order():
    assert list(df.loc[0].compute().columns) == ["x", "y", "a"]


+def test_from_dict_dataframe():


We're not currently testing that the dtype, orient, etc. parameters are working as expected. Looking at the code, they are getting forwarded properly, but it would still be good to add test coverage for those cases

How about something like:

@pytest.mark.parametrize("dtype", [int, float]) @pytest.mark.parametrize("orient", ["columns", "index"]) @pytest.mark.parametrize("npartitions", [2, 5]) def test_from_dict(dtype, orient, npartitions): data = {"a": range(10), "b": range(10)} expected = pd.DataFrame.from_dict(data, dtype=dtype, orient=orient) result = dd.from_dict(data, npartitions=npartitions, dtype=dtype, orient=orient) if orient == "index": # DataFrame only has two rows with this orientation assert result.npartitions == 1 else: assert result.npartitions == npartitions assert_eq(result, expected)

jrbourbeau · 2022-05-05T15:30:50Z

dask/dataframe/io/io.py

@@ -146,6 +146,40 @@ def from_array(x, chunksize=50000, columns=None, meta=None):
    return new_dd_object(dsk, name, meta, divisions)


+def from_dict(data, npartitions, orient="columns", dtype=None, columns=None):


I see we're adding this as a top-level dd.from_dict method, but the corresponding pandas method is actually a classmethod -- so pd.DataFrame.from_dict instead of pd.from_dict. My default is usually to match the existing DataFrame API if one exists. Thoughts of making this dd.DataFrame.from_dict?

jrbourbeau · 2022-05-05T15:45:36Z

The failing windows-py3.10 test seems unrelated to this PR

Yep, I agree this looks unrelated. Just opened up #9035

Co-authored-by: James Bourbeau <jrbourbeau@users.noreply.github.com>

MrPowers · 2022-05-07T23:25:19Z

Updated from_dict to be a class method based on a suggestion from @jrbourbeau. dd.DataFrame.from_dict is more consistent with pd.DataFrame.from_dict.

@pavithraes - can you please take another look and check to make sure that I properly updated the docs to reflect that this is a class method 🙏

Do you think this method should have the @derived_from(pd.DataFrame) annotation? Thanks for all the help!!

pavithraes · 2022-05-09T11:36:24Z

@pavithraes - can you please take another look and check to make sure that I properly updated the docs to reflect that this is a class method 🙏

@MrPowers Thank you! The docs updates look good to me.

Do you think this method should have the @derived_from(pd.DataFrame) annotation?

I don't think we need it here because you've added a nice and comprehensive docstring. :)

jrbourbeau

Thanks for the updates @MrPowers! Didn't get a chance to look at them today. Will plan to review / merge tomorrow though

jrbourbeau

Thanks @MrPowers -- will merge after CI passes

github-actions bot added dataframe io labels May 3, 2022

douglasdavis requested changes May 3, 2022

View reviewed changes

dask/dataframe/io/io.py Outdated Show resolved Hide resolved

dask/dataframe/io/io.py Outdated Show resolved Hide resolved

dask/dataframe/io/io.py Outdated Show resolved Hide resolved

pavithraes reviewed May 3, 2022

View reviewed changes

dask/dataframe/io/io.py Outdated Show resolved Hide resolved

github-actions bot added the documentation Improve or add to documentation label May 4, 2022

pavithraes reviewed May 4, 2022

View reviewed changes

Add from_dict method to create Dask DataFrame from dictionary

0b4264b

MrPowers force-pushed the add_from_dict branch from a1baf24 to 0b4264b Compare May 5, 2022 12:47

Alphabetical for linter

c08fdfc

jrbourbeau reviewed May 5, 2022

View reviewed changes

jrbourbeau mentioned this pull request May 5, 2022

test_filter_nonpartition_columns flaky #9035

Closed

MrPowers and others added 5 commits May 5, 2022 13:55

Update dask/dataframe/io/io.py

e1c1fb5

Co-authored-by: James Bourbeau <jrbourbeau@users.noreply.github.com>

Update dask/dataframe/io/io.py

1f566a8

Co-authored-by: James Bourbeau <jrbourbeau@users.noreply.github.com>

Refactor from_dict tests

8540c23

Make from_dict a DataFrame class method

cae7948

from_dict cleanup

fa26dfe

github-actions bot removed the io label May 7, 2022

pavithraes requested a review from jrbourbeau May 9, 2022 11:36

jrbourbeau reviewed May 10, 2022

View reviewed changes

Small updates

587163b

jrbourbeau approved these changes May 11, 2022

View reviewed changes

jrbourbeau changed the title ~~Add from_dict method to create Dask DataFrame from dictionary~~ Add DataFrame.from_dict classmethod May 11, 2022

jrbourbeau merged commit dffc957 into dask:main May 11, 2022

erayaslan pushed a commit to erayaslan/dask that referenced this pull request May 12, 2022

Add DataFrame.from_dict classmethod (dask#9017)

3095519

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `DataFrame.from_dict` classmethod #9017

Add `DataFrame.from_dict` classmethod #9017

MrPowers commented May 3, 2022

GPUtester commented May 3, 2022

quasiben commented May 3, 2022

douglasdavis left a comment

pavithraes commented May 3, 2022

MrPowers commented May 4, 2022

pavithraes May 4, 2022

MrPowers May 4, 2022

pavithraes commented May 5, 2022

MrPowers commented May 5, 2022

pavithraes commented May 5, 2022

jrbourbeau left a comment

jrbourbeau May 5, 2022

jrbourbeau May 5, 2022

jrbourbeau May 5, 2022

jrbourbeau commented May 5, 2022

MrPowers commented May 7, 2022

pavithraes commented May 9, 2022

jrbourbeau left a comment

jrbourbeau left a comment

		@@ -241,6 +241,15 @@ def test_from_bcolz_column_order():
		assert list(df.loc[0].compute().columns) == ["x", "y", "a"]


		def test_from_dict_dataframe():

		@@ -146,6 +146,40 @@ def from_array(x, chunksize=50000, columns=None, meta=None):
		return new_dd_object(dsk, name, meta, divisions)


		def from_dict(data, npartitions, orient="columns", dtype=None, columns=None):

Add DataFrame.from_dict classmethod #9017

Add DataFrame.from_dict classmethod #9017

Conversation

MrPowers commented May 3, 2022

GPUtester commented May 3, 2022

quasiben commented May 3, 2022

douglasdavis left a comment

Choose a reason for hiding this comment

pavithraes commented May 3, 2022

MrPowers commented May 4, 2022

pavithraes May 4, 2022

Choose a reason for hiding this comment

MrPowers May 4, 2022

Choose a reason for hiding this comment

pavithraes commented May 5, 2022

MrPowers commented May 5, 2022

pavithraes commented May 5, 2022

jrbourbeau left a comment

Choose a reason for hiding this comment

jrbourbeau May 5, 2022

Choose a reason for hiding this comment

jrbourbeau May 5, 2022

Choose a reason for hiding this comment

jrbourbeau May 5, 2022

Choose a reason for hiding this comment

jrbourbeau commented May 5, 2022

MrPowers commented May 7, 2022

pavithraes commented May 9, 2022

jrbourbeau left a comment

Choose a reason for hiding this comment

jrbourbeau left a comment

Choose a reason for hiding this comment

Add `DataFrame.from_dict` classmethod #9017

Add `DataFrame.from_dict` classmethod #9017