Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BlockMatrixFilterRowsRDD to decrease transposing #9373

Merged

Conversation

johnc1231
Copy link
Contributor

@johnc1231 johnc1231 commented Aug 28, 2020

The current filter_rows is implemented as a transpose. Nik and Konrad are running into driver memory issues when doing tree_matmul, and one theory of mine is that making lots of BlockMatrixTransposeRDD's is contributing to this. I'd like to make this change to avoid that in the future.

@johnc1231
Copy link
Contributor Author

Assigning @tpoterba since this is quite similar to #9374 that you reviewed. It's basically just a copy paste of the column oriented one, but adjusted to do rows instead.

@johnc1231 johnc1231 force-pushed the johnc-block-matrix-filter-rows-no-transpose branch from 2735c64 to f418b68 Compare August 31, 2020 19:48
@johnc1231 johnc1231 mentioned this pull request Aug 31, 2020
@danking danking merged commit 8049d1b into hail-is:main Aug 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants