svd_backward: more memory and computationally efficient. #50109
Conversation
💊 CI failures summary and remediationsAs of commit 8b34f0b (more details on the Dr. CI page): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions to the (internal) Dr. CI Users group. This comment has been revised 11 times. |
| L = L - L.conj(); | ||
| Tensor imag_term = 0.5 * at::matmul(u * (L * sigma_inv).unsqueeze(-2), vh); |
There was a problem hiding this comment.
nit: Could it be better to replace L with just 2 * L.imag() and use the float-float mul in L * sigma_inv instead of the float-complex one, then multiply by the imaginary unit in the most efficient way?..
There was a problem hiding this comment.
That's a simple change that might give better performance, so I think it's worth adding it to this PR.
There was a problem hiding this comment.
Also you don't need to do the 2* if you remove the 0.5* right? :D
There was a problem hiding this comment.
Yes, so one more reason to follow the nit.
smessmer
added
the
triaged
|
Resolves #38353. |
| L = L - L.conj(); | ||
| Tensor imag_term = 0.5 * at::matmul(u * (L * sigma_inv).unsqueeze(-2), vh); |
There was a problem hiding this comment.
That's a simple change that might give better performance, so I think it's worth adding it to this PR.
| @@ -1947,9 +1944,12 @@ Tensor svd_backward(const std::vector<torch::autograd::Variable> &grads, const T | |||
|
|
|||
| if (gu.defined()) { | |||
| auto guh = gu.conj().transpose(-2, -1); | |||
| u_term = at::matmul(u, at::matmul(F.mul(at::matmul(uh, gu) - at::matmul(guh, u)), sigma_mat)); | |||
| u_term = at::matmul(u, (F.mul(at::matmul(uh, gu) - at::matmul(guh, u)) * sigma.unsqueeze(-2))); | |||
There was a problem hiding this comment.
nit there is an extra parenthesis here that is not needed for the second argument of the outer most matmul:
u_term = at::matmul(u, F.mul(at::matmul(uh, gu) - at::matmul(guh, u)) * sigma.unsqueeze(-2));
| L = L - L.conj(); | ||
| Tensor imag_term = 0.5 * at::matmul(u * (L * sigma_inv).unsqueeze(-2), vh); |
There was a problem hiding this comment.
Also you don't need to do the 2* if you remove the 0.5* right? :D
…ed/svd_complex_backward_minor_fix
…ed/svd_complex_backward_minor_fix
There was a problem hiding this comment.
@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
…ed/svd_complex_backward_minor_fix
There was a problem hiding this comment.
@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
As per title.
CC @IvanYashchuk (unfortunately I cannot add you as a reviewer for some reason).