Features/31 outer #596
Merged
Features/31 outer #596
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #596 +/- ##
==========================================
+ Coverage 97.41% 97.43% +0.01%
==========================================
Files 75 75
Lines 15030 15155 +125
==========================================
+ Hits 14642 14766 +124
- Misses 388 389 +1
Continue to review full report at Codecov.
|
heat/core/linalg/basics.py
Outdated
| # blocking send and recv | ||
| if split == 0: | ||
| b.comm.Send(t_b, dest_rank) | ||
| t_b = torch.empty(lshape_map[actual_origin], dtype=t_outer_dtype, device=t_a.device) |
There was a problem hiding this comment.
Can we initialize this buffer once outside the loop and then reuse it? Would save up some initialization time (and then slice it via the chunks below)
There was a problem hiding this comment.
That's a good point. I'm initializing a buffer for the "traveling" data now, t_b_run and t_a_run, and slicing them to the correct size afterwards.
heat/core/linalg/basics.py
Outdated
| t_outer_slice[1] = remote_slice[0] | ||
| elif split == 1: | ||
| a.comm.Send(t_a, dest_rank) | ||
| t_a = torch.empty(lshape_map[actual_origin], dtype=t_outer_dtype, device=t_a.device) |
Markus-Goetz
approved these changes
Jun 18, 2020
too bad... |
Markus-Goetz
previously approved these changes
Jun 24, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Introducing the outer product of two DNDarrays as per numpy.outer(), i.e.:
ht.outer(a, b)aandbmust be 1-dimensional, or will be flattened by default.In linear algebra, if
ais a vector of n elements, andbis a vector of m elements, the outer productouter(a,b)is a matrix of size (n, m) and elements:Implementation
Locally, the outer product is calculated via PyTorch's Einstein summation, as in:
torch.einsum("i,j->ij", a, b)The memory-distributed implementation works under the assumption that the vectors / DNDarrays are dense (implementation for sparse case will be addressed at a later point (Implement sparse matrices #384). Procedure:
aandbare distributeda_i * b_jmultiplications have taken place.torch.einsum) is calculated locally, written into the appropriate slice ofout = ht.outer(a,b), and never needs to be communicated among ranksoutis by definition split along 0 ifbis sent around andastays put. Conversely, it is split along 1 ifais sent around andbstays put. As a consequence...splitto allow users to choose the split direction of the output matrix.splitdecides which DNDarray stays and which one goes round the ranks. If split is None in the distributed case,ht.outer(a,b).splitwill be the same asa.splitandb.split, i.e. 0.Note that the distributed case requires that at least one among
a.split,b.splitandout.splitbenot None. Otherwise we fall back to local.Issue/s resolved: #31
Changes proposed:
splitto define the split dimension of the output matrix, e.g.:out = ht.outer(a, b, split=1)Type of change
Due Diligence
Does this change modify the behaviour of other functions? If so, which?
no