ignore amp.scale_loss()

[jizongFox]

Hi all,
Thanks for this wonderful work, it is very useful for my current project.
I am using apex with only one GPU, I am wondering if in this case,
with amp.scale_loss(loss, optimizer) as loss: loss.barckward()
can be replaced by the conventional one:
optimizer.zero() loss.backward() optimizer.step()
Thanks by advance for your reply.

[ptrblck]

Hi @jizongFox,

good to hear apex is useful for your project! :)

The loss scaling should also be performed on a single GPU and is not related to a distributed / multi-GPU setup.

[jizongFox]

Hi @ptrblck ,
Thanks for your kind reply!
I am still very confused about this component despite having consulted its API.
On context manager entrance, creates scaled_loss = (loss.float())*current loss scale. scaled_loss is yielded so that the user can call scaled_loss.backward():
I assume here the loss.float() is the same as FP32 case, but why we must multiply a current loss scale?
I saw the examples below:
model, optim = amp.initialize(model, optim, opt_level="O1") # uses "loss_scale="dynamic" default model, optim = amp.initialize(model, optim, opt_level="O1", loss_scale=128.0|"128.0")
Why the loss_scale is set to be 2**7 = 128 ?
Some documents mentioned gradient clipping, why it is necessary?
I will be appreciated if you can offer detailed explanations.

[jizongFox]

I also noticed that using O0 level, the convergence can be extremely slow, and the gradient norm of traditional training is about 1e-3 while at O0 level, it is about 1e-5.

[ptrblck]

The loss is being scaled, since the gradient updates might result in an underflow due to limited dynamic range in FP16. Without scaling loss.backward() might create gradients which are that small in magnitude that they cannot be represented in FP16. Loss scaling prevents this by using the dynamic range of FP16 better.

You could set the loss scaling to a static value, but I would recommend using the dynamic approach and let amp adjust the scale factor.

Gradient clipping is used e.g. to prevent gradient blowups. Which documents mention it, as this doesn't seem to be directly related to mixed precision training.

O0 is the vanilla FP32 training. Do you see any difference between using amp in O0 and not using amp at all?

[jizongFox]

I noticed a big difference between using O0 and not to use Apex. I will check if there's a bug in my code.

[ptrblck]

If you cannot find any bug, would it be possible to post a code snippet so that we can have a look?

[mcarilli]

Hi all,
Thanks for this wonderful work, it is very useful for my current project.
I am using apex with only one GPU, I am wondering if in this case,
with amp.scale_loss(loss, optimizer) as loss: loss.barckward()
can be replaced by the conventional one:
optimizer.zero() loss.backward() optimizer.step()
Thanks by advance for your reply.

Running the backward pass under the with amp.scale_loss context manager is an essential part of the API and cannot be omitted. Also, I don't recommend the exact syntax you used.

with amp.scale_loss(loss, optimizer) as loss: loss.backward()
                                         ^      ^ don't name these "loss", call them "scaled_loss" instead.

Instead, name things the way the documentation shows:

with amp.scale_loss(loss, optimizer) as scaled_loss: 
    scaled_loss.backward()

so that the scaled loss yielded by the context manager does not overwrite your original loss.

You only need to use gradient clipping if your model originally required it (without Amp). Make sure to apply gradient clipping AFTER the context manager exit, as the documentation shows.

Loss scaling is intended to prevent FP16 gradients from underflowing in the backward pass. See our GTC talk (video, slides) for more detail.
Amp can either automatically choose a loss scale and adjust it over time. This is called "dynamic loss scaling." Dynamic loss scaling is the default (and recommended) behavior for opt_level O1and O2. However, static loss scaling can be selected instead. The static loss scale value of 128 was just an example to show the syntax. It is a "reasonable" value, but other than that there is nothing special or unique about it numerically.

O0 is not mixed precision, O0 is intended to be ordinary FP32 training. For mixed precision training, we recommend O1 with other options left to default:

model, optimizer = amp.initialize(model, optimizer, opt_level="O1")

[jizongFox]

Hi @ptrblck and @mcarilli,
Thanks for your gracious explanations. The previously reported issue was due to my code. Now it works very consistently with FP32 case and reduces memory usage.
Thanks a lot for your work. I close the issue.