[discussion] Refactor spectral_norm to use the newly merged lowrank solvers and proposal for Linear Algebra Cookbook page

[vadimkantorov]

LOBPCG variants were merged in #29488

However, spectral_norm implementation contains an older power iteration implementation to get an estimate for leading eigenvalue.

I propose (also in #8049 (comment)) to either:

1. surface the power iteration / implement randomized power iteration to the user - it's useful reference code (for debugging linear layer properties), but it's one another solver to maintain, although not a big amount of code
2. replace existing power iteration by the new LOBPCG call (discussed in the end of [feature request] [pytorch] Truncated SVD #8049 and deemed quite practical by @lobpcg)

In general I think it'd be nice to have some dedicated page "Linear Algebra PyTorch Cookbook" and detail for typical linear algebra tasks (linear system solvers, eigenproblem solvers, ...) what methods are currently supported (wrt sparsity, symmetric, p.s.d, problem sizes, differentiability, space/time complexity, cpu/gpu support, batchability, what external libraries are used, is parallelization across batches used, numerical stability considerations, etc)

cc @vincentqb @vishwakftw @ssnl @jianyuh @nikitaved

[nikitaved]

It also makes sense to investigation the effects of the overestimation of the Lipschitz constant. There is a suspicion that the current implementation might actually underestimate it.
@vadimkantorov, would you be interested in helping with the experiments involving GANs?