TPU utilization could be improved further?

[sayakpaul]

Training details are in #2

I think the TPU utilization is a bit lower than expected:

Is this expected?

I understand there might be other network access factors that can contribute to this but wanted to know.

[akolesnikoff]

TPU utilization should be close to 100%. I think your dashboard is showing something else.

My guess it shows percentage of time when TPUs do FLOP-heavy operation, like matrix multiplications. The rest is various data reshapes, weight synchronization and so on. IIUC it is hard to do substantially better than what we have now.

[sayakpaul]

Here's what I am doing.

• Go to https://console.cloud.google.com/compute/tpus/detail.
• Then on the monitoring panel, I am getting:

I'm probably wrong in selecting that. But you've suggested clues which are enough for now.