[docs] torch.set_flush_denormal(...) to mention default mode

[vadimkantorov]

Currently the torch.set_flush_denormal docs do not mention what is the default mode.

It would also be nice if they mentioned the FTZ / DAZ flags and a link to some reference page (like intel's?) - like that this method would show up for google queries like pytorch ftz.

Also, a note on whether denormals are flushed on GPU would be helpful.

[colesbury]

These seem like good things to mention in the docs.

Currently the torch.set_flush_denormal docs do not mention what is the default mode.

Usually denormals are not flushed-to-zero, but it's dependent on the compiler (and potentially OS and CPU). For example, Intel's compiler reportedly sets FTZ when optimization is turned on.

It would also be nice if they mentioned the FTZ / DAZ flags and a link to some reference page (like intel's?)

The linked page is for Intel's compiler (ICC). I don't think we should link to that since we don't generally use it. I don't know of a great reference, but wikipedia is OK and Intel's manual is the authoritative reference for Intel CPUs.

https://en.wikipedia.org/wiki/Denormal_number
Intel® 64 and IA-32 Architectures Software Developer’s Manuals - authoritative, but dense

Also, a note on whether denormals are flushed on GPU would be helpful.

No - they're not flushed to zero. The immediate reasons is that it would require recompilation instead of something we could change at runtime. It also tends not to be as much of an issue. I think FMA (fused multiply-add) operate at full throughput with denormals on recent GPUs (this blog post hints at it).

Other operations (division, sqrt, etc.) might be slower but we tend not to use them in the super compute-heavy ops like convolution/gemm. They're more likely to be part of an element-wise computation, which tends to be memory bound.