Something wrong with FP16

[samecos]

@lightvector @y-ich
I used TITAN X (PASCAL) and automatically turned on the FP16 under the 1.3.2 program, but the performance is very low, which is far less efficient than the normal FP32.

[y-ich]

@samecos san,

In my experience, you need 20 threads to get max. performance on P100. If lower threads, for example 1, you would get much lower performance than fp32.

Try setting threads and batchSize to around 20.

[lightvector]

Also, shortly after release I updated the v1.3.2 release to no longer automatically choose FP16. It will choose only FP32 by default, if you re-download v1.3.2 again.

If you would like to try to get FP16 working, you can still try to, by manually setting cudaUseFP16 = true instead of cudaUseFP16 = auto. I don't have any experience in Pascal-generation GPUs, so probably y-ich can help you more if you still find problems.

[samecos]

@samecos san,

In my experience, you need 20 threads to get max. performance on P100. If lower threads, for example 1, you would get much lower performance than fp32.

Try setting threads and batchSize to around 20.
@y-ich
Thanks .
uem I try to change paramater numSearchThreads = 24 ; nnMaxBatchSize = 24
cudaUseFP16 = true ; cudaUseNHWC = auto ;
But the engine efficiency is always low.

[y-ich]

It seems no usage of GPU.

What would happen if you set “ cudaUseFP16 = true; cudaUseNHWC = false”?

[samecos]

@y-ich
Sure , I made a simple table

| UseFP16 = false; UseNHWC = false Power 256/250 GPU-Util 95% rxpci_max 1800 txpci_max 1800 |
| UseFP16 = false; UseNHWC = true Power 256/250 GPU-Util 97% rxpci_max 1800 txpci_max 30 |
| UseFP16 = true; UseNHWC = false Power 120/250 GPU-Util 100% rxpci_max 2200 txpci_max 200 |
| UseFP16 = true; UseNHWC = true Power 221/250 GPU-Util 100% rxpci_max 1800 txpci_max 210 |

[y-ich]

@samecos san,

Thank you for additional information.
Now I have no idea.

I tried to confirm my experience on Google Colab, but got T4 for today's GPU.

[samecos]

@y-ich Hi,
I got some info about fp16 support.The only GPUs with full-rate FP16 performance are Tesla P100, Quadro GP100, and Jetson TX1/TX2.
All GPUs with compute capability 6.1 (e.g. GTX 1050, 1060, 1070, 1080, Pascal Titan X, Titan Xp, Tesla P40, etc.) have low-rate FP16 performance. It's not the fast path on these GPUs.
But all of these GPUs should support "full rate" INT8 performance.
So we Katago should consider supporting INT8? uem may be precision not good enough?

[merlinpan]

The only "reasonable" way to use FP16 on Pascal is to use FP16 storage but FP32 computation I think. That's what leela does as well? There is supposedly some improvement because the convolutions needed for DNNs can be memory bandwidth bound (fetching weights).

[y-ich]

As samecos san pointed out, there are two kinds of Pascal architecture, supporting packed fp16 or not.
cudaUseFP16 = true with large threads certainly has performance boost on P100 but not on TITAN Xp.