[feature request] Singular values and spectral norm for convolutional layers

[vadimkantorov]

Google Brain open-sourced a simple impl based on FFT: https://github.com/brain-research/conv-sv/blob/master/conv2d_singular_values.py (paper)

It could be a nice addition to SpectralNorm or for debugging in general

cc @mruberry @peterbell10 @vincentqb @vishwakftw @jianyuh @nikitaved @pearu @ssnl

[soumith]

seems simple enough, I looked at the code yesterday. However, is spectral_norm API the best suited place for this?

[vadimkantorov]

@soumith On a broader side, I think SpectralNorm power iteration code and this new Conv layer treatment should be factored out from the current SpectralNorm place to a more accessible generic torch namespace (or maybe some new torch.spectral namespace, but svd lives in torch, so not sure).

[vadimkantorov]

There was also https://github.com/google/svcca (fast feature correlation computations), so may be some torch.interpretability, torch.nn.diagnostics can be a good addition as well (maybe within contrib?)

[jensbehrmann]

Would also be interested in this.

One issue here is that the algorithm needs to compute SVD of a complex matrix (output of FFT). Is that even supported in PyTorch?

[redwrasse]

I see this issues is a few years old, but I'd be interested in this as well- following the expressions in Szegedy's original paper https://arxiv.org/pdf/1312.6199.pdf

[vadimkantorov]

One issue here is that the algorithm needs to compute SVD of a complex matrix (output of FFT). Is that even supported in PyTorch?

I think now it is!

[redwrasse]

I'm not up to speed with research papers but as I understand the broad idea is singular values (either computed directly or in frequency space) characterize robustness properties. The documentation around SpectralNorm and possible additions need some more general discussion/use cases? Don't need operators to be convolutions; but in such cases singular values computed in freq. space, etc. It seems confusing as it is restrict operator norms to an application to GANs.

[skmda37]

Would also still be very interested in a pytorch implementation of spectral norm of conv layers as in https://github.com/brain-research/conv-sv/blob/master/conv2d_singular_values.py (paper).

[redwrasse]

Any progress on this?

Maybe related, there is also an open issue on correctness of current spectral norm implementation for conv2d operators: #99149

On a broader level, I'm having a hard time understanding the realm of applicability of the currrent spectral norm implementation: which operators does it work on, what guarantees of convergence are there, and does it work on complex-valued networks (I'm finding it doesn't).

Briefly perusing the research the more important concern is effectively computing the Lipshitz constant, which is a hard problem. The spectral norm is one tool in the toolkit. The pytorch code and docs on the other hand seem tied to the spectral norm in the context of a particular research paper around use in GANs.