Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BlockMatrixFilterRowsRDD to decrease transposing #9373



Copy link

@johnc1231 johnc1231 commented Aug 28, 2020

The current filter_rows is implemented as a transpose. Nik and Konrad are running into driver memory issues when doing tree_matmul, and one theory of mine is that making lots of BlockMatrixTransposeRDD's is contributing to this. I'd like to make this change to avoid that in the future.

Copy link
Contributor Author

Assigning @tpoterba since this is quite similar to #9374 that you reviewed. It's basically just a copy paste of the column oriented one, but adjusted to do rows instead.

@johnc1231 johnc1231 force-pushed the johnc-block-matrix-filter-rows-no-transpose branch from 2735c64 to f418b68 Compare August 31, 2020 19:48
@johnc1231 johnc1231 mentioned this pull request Aug 31, 2020
@danking danking merged commit 8049d1b into hail-is:main Aug 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

3 participants