New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply the culling optimization to ColumnAccessor. #521

Merged
merged 1 commit into from Sep 11, 2018

Conversation

Projects
None yet
1 participant
@rainwoodman
Member

rainwoodman commented Sep 11, 2018

Since dask 0.19.0 we start to get OOM errors, because
dask stopped applying culling to the default compute method when
optimize_graph is False.

We still do not want optimize_graph because it permutes the order
ing of column and row access in a unfavorable way; but we do
want the culling. So implement the culling in the new
graph protocal and see if it helps.

Apply the culling optimization to ColumnAccessor.
Since dask 0.19.0 we start to get OOM errors, because
dask stopped applying culling to the default compute method when
optimize_graph is False.

We still do not want optimize_graph because it permutes the order
ing of column and row access in a unfavorable way; but we do
want the culling. So implement the culling in the new
graph protocal and see if it helps.
@rainwoodman

This comment has been minimized.

Member

rainwoodman commented Sep 11, 2018

This is the source of the OOM problem. @ybh0822

@rainwoodman

This comment has been minimized.

Member

rainwoodman commented Sep 11, 2018

My local test shows the problem is fixed. Will merge after Travis is green.

@rainwoodman rainwoodman merged commit 967fe22 into bccp:master Sep 11, 2018

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.002%) to 95.069%
Details

@rainwoodman rainwoodman deleted the rainwoodman:dask-0.19.0-oom branch Sep 11, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment