Document properties of computed gradients in cholesky and eigh

[hvy]

An alternative to #8306, keeping the current gradient behavior but document it instead.

[toslunar]

I suggest:

The computed gradient is a symmetric matrix, where the mathematical definition of Chelesky decomposition supersedes the definition of ``*potrf`` in LAPACK. More specifically, the gradient is computed as if the domain of the function is restricted to a Riemannian submanifold of the Euclidean space :math:`R^{n \times n}` (an open subset of the subspace of symmetric matrices, whose inner product structure is compatible with the Frobenius norm).

(cholesky needs "an open subset of" for the positive-definiteness but eigh does not.)

[hvy]

How would you rephrase the documentation for eigh similar to this?

[toslunar]

@masomatics helped me to improve the above.

The computed gradient is a symmetric matrix. Our implementation of backpropagation is faithful to the mathematical definition of Cholesky decomposition. The forward computation with LAPACK does not really check if the input matrix is symmetric. We, however, do not really backpropagate this very process. Instead, the gradient is computed as if the function is restricted to a Riemannian submanifold of R_{n \times n} consisting just of positive-definite symmetric matrices.

How would you rephrase the documentation for eigh similar to this?

s/Cholesky decomposition/eigenvalue decomposition of symmetric matrices/
s/positive-definite//

I admit that it is superfluous to call the linear subspace Sym(n) of R^{n×n} a Riemannian submanifold of R^{n×n}.

[hvy]

Thanks for the suggestion. Could you just elaborate what you mean with We, however, do not really backpropagate this very process.? It sounded a bit unnatural to me.

[toslunar]

A sentence like "LAPACK reconstructs a symmetric matrix from a triangular part of the input" should be inserted. However the word "reconstruct" there does not mean copying on the memory.

[hvy]

Updated the notes, please take a look at them.

[toslunar]

undefined sounds ValueError from the math perspective or undefined behavior from the C++ perspective. I suggest:

This functions assumes that ``a`` is symmetric. The output of the forward/backward computation is unspecified otherwise.

IMO, the backward of eigh is also returning an inconsistent value unless the forward input is symmetric.

[hvy]

Addressed this too.

[hvy]

PTAL.

[hvy]

PTAl.

[toslunar]

LGTM

[toslunar]

Jenkins & flexCI, test this please.

[chainer-ci]

Jenkins CI test (for commit d078f3e, target branch master) failed with status FAILURE.

[toslunar]

The Jenkins failure is unrelated to the PR.