Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Delay long list arguments #4735

Merged
merged 2 commits into from Apr 26, 2019
Merged

Delay long list arguments #4735

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9ba0203
Delay long list arguments
mrocklin Apr 25, 2019
ce80f35
use normalize_arg in map_partitions
mrocklin Apr 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions dask/array/core.py
View file
Open in desktop
Expand Up @@ -299,6 +299,8 @@ def normalize_arg(x):
return x
elif isinstance(x, str) and re.match(r'_\d+', x):
return delayed(x)
elif isinstance(x, list) and len(x) >= 10:
return delayed(x)
mrocklin marked this conversation as resolved.
Show resolved Hide resolved
elif sizeof(x) > 1e6:
return delayed(x)
else:
Expand Down
10 changes: 10 additions & 0 deletions dask/dataframe/tests/test_dataframe.py
View file
Open in desktop
Expand Up @@ -3544,3 +3544,13 @@ def test_replace():
assert_eq(df.replace({1: 10, 2: 20}), ddf.replace({1: 10, 2: 20}))
assert_eq(df.x.replace(1, 10), ddf.x.replace(1, 10))
assert_eq(df.x.replace({1: 10, 2: 20}), ddf.x.replace({1: 10, 2: 20}))


def test_map_partitions_delays_lists():
df = pd.DataFrame({'x': [1, 2, 3, 4, 5]})
ddf = dd.from_pandas(df, npartitions=2)

L = list(range(100))
out = ddf.map_partitions(lambda x, y: x + sum(y), y=L)

assert any(str(L) == str(v) for v in out.__dask_graph__().values())