Implement AxisPartition for Dask #281

pcahyna · 2018-12-02T12:04:29Z

No description provided.

+ using dask.delayed + simple case works

(mostly copy of pandas_on_python.py)

…sPartition The latter fails the test_var test.

AmplabJenkins · 2018-12-02T12:09:37Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-02T12:09:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-Performance-PRB/73/
Test PASSed.

AmplabJenkins · 2018-12-02T12:12:13Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-02T12:12:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/135/
Test PASSed.

devin-petersohn · 2018-12-03T16:20:25Z

Thanks @pcahyna and @coobas! I will resolve the travis failures and get this merged today.

AmplabJenkins · 2018-12-03T16:28:58Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T16:29:00Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-Performance-PRB/78/
Test PASSed.

AmplabJenkins · 2018-12-03T16:31:35Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T16:31:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/140/
Test PASSed.

AmplabJenkins · 2018-12-03T16:56:34Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T16:56:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-Performance-PRB/79/
Test PASSed.

AmplabJenkins · 2018-12-03T17:00:39Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T17:00:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/141/
Test PASSed.

AmplabJenkins · 2018-12-03T17:11:32Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T17:11:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-Performance-PRB/80/
Test PASSed.

AmplabJenkins · 2018-12-03T17:14:08Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T17:14:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/142/
Test PASSed.

AmplabJenkins · 2018-12-03T17:25:49Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T17:25:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-Performance-PRB/81/
Test PASSed.

AmplabJenkins · 2018-12-03T17:28:41Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T17:28:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/143/
Test PASSed.

AmplabJenkins · 2018-12-03T17:34:17Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T17:34:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-Performance-PRB/82/
Test PASSed.

AmplabJenkins · 2018-12-03T17:36:26Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T17:36:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-Performance-PRB/83/
Test PASSed.

AmplabJenkins · 2018-12-03T17:37:13Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T17:37:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/144/
Test PASSed.

AmplabJenkins · 2018-12-03T17:37:40Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T17:37:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/145/
Test PASSed.

AmplabJenkins · 2018-12-03T17:41:59Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T17:42:00Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-Performance-PRB/84/
Test PASSed.

AmplabJenkins · 2018-12-03T17:44:23Z

Merged build finished. Test PASSed.

AmplabJenkins · 2018-12-03T17:44:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/146/
Test PASSed.

devin-petersohn

Great addition thanks!

coobas · 2018-12-03T18:18:24Z

@devin-petersohn thanks for bringing us on board and sorry for being silent for a while. It was a nice project for our weekend hackathon - the skeleton you provided helped a lot. There may be a bug fix that @pcahyna has not pushed yet (he will do it today evening, maybe for a separate MR). I also wanted to go through the code once more before making an upstream pull request -- you were just too efficient 😄

devin-petersohn · 2018-12-03T19:27:16Z

No problem! Further pull requests are welcome! I tested the code locally and it works great for a first pass.

coobas and others added 6 commits December 2, 2018 09:59

dask remote partition first implementations:

935684b

+ using dask.delayed + simple case works

introduce queue of delayed calls and use it in add_to_apply_call

89a5ba8

add the function to apply stack in apply()

b039723

redo apply and get properly

f1fc07c

Implement AxisPartition for Dask

57e9f67

(mostly copy of pandas_on_python.py)

Use pandas_on_ray not pandas_on_python as template for PandasOnRayAxi…

07c5b4e

…sPartition The latter fails the test_var test.

Merge branch 'master' into AxisPartition

9675d59

devin-petersohn added 2 commits December 3, 2018 08:34

Lint

a1a1f6c

Fixing python2 travis test failure

2d3d68f

Lint

97dec45

Removing lru_cache temporarily

89def49

devin-petersohn added 2 commits December 3, 2018 09:29

Removing unused import

8690854

Lint

a71c3f8

Lint

ef4dbdb

devin-petersohn approved these changes Dec 3, 2018

View reviewed changes

devin-petersohn merged commit e0a8d6a into modin-project:master Dec 3, 2018

devin-petersohn mentioned this pull request Dec 4, 2018

Dask runtime for Modin takes longer than expected #289

Closed

devin-petersohn mentioned this pull request Mar 28, 2019

Query: What is the difference between Dask and Modin? #515

Closed

abdulelahsm mentioned this pull request Nov 14, 2020

[DOCS] Update ReadMe.md with Modin vs Dask DataFrame explanation #2433

Closed

Implement AxisPartition for Dask #281

Implement AxisPartition for Dask #281

Conversation

pcahyna commented Dec 2, 2018

AmplabJenkins commented Dec 2, 2018

AmplabJenkins commented Dec 2, 2018

AmplabJenkins commented Dec 2, 2018

AmplabJenkins commented Dec 2, 2018

devin-petersohn commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

AmplabJenkins commented Dec 3, 2018

devin-petersohn left a comment

Choose a reason for hiding this comment

coobas commented Dec 3, 2018

devin-petersohn commented Dec 3, 2018