GPU utilization becomes lower with 16-bit floating point

[Lollipop321]

When I tried to implement the paper "mixed precision training" in the tensor2tensor open source code, I found that I could not achieve the training speed that I expected. After checking, I found that the GPU utilization of my modified code was significantly reduced. However, I did not modify the underlying code that allocates GPU resources. I suspect that the problem occurs where the GPU and CPU interact.

Has anyone encountered the same/similar problems? Many thanks in advance!

[liutengbo]

I have encourtered the same problem . Anybody has a solution?

[Mr-wang2016]

@Lollipop321 @liiutengbo
Hi.
My t2t version is 1.13.2, may i ask how to make use of Mixed precision in t2t?
Thanks a lot!

[lkluo]

I used --hparams_set=transformer_fairseq_fp16_activation_big. The training does speed up, but warnings keep showing: "It seems that global step (tf.train.get_global_step) has not been increased." The loss seems decreasing slower than full precision at the early stage.

update
The loss increased after 8k steps.

[Mr-wang2016]

@lkluo
Hi
What type of GPU are you using?I have 1080 GPU. Can 1080 GPU speed up the training?

[lkluo]

@Mr-wang2016 V100

[Lollipop321]

@Lollipop321 @liiutengbo
Hi.
My t2t version is 1.13.2, may i ask how to make use of Mixed precision in t2t?
Thanks a lot!

Yes, V100 and you need to make some changes to the code.

[mengdong]

Any GPU with Tensor Core (V100/T4) can benefit from mixed-precision training, in tensorflow 1.14, the automatic mixed-precision training can be enabled by https://www.tensorflow.org/api_docs/python/tf/train/experimental/enable_mixed_precision_graph_rewrite, which is a wrapper over the optimizer. And config.graph_options.rewrite_options.auto_mixed_precision = rewriter_config_pb2.RewriterConfig.ON
need be passed into session config if config is used. And to use Tensor Core, the tensor need to meet certain shape requirement(multiple of 8). there is usually good speedup when GPU is already fully saturated at FP32.