In QAT (Quantization-Aware Training), why does the model fail to converge if a pre-trained model is not loaded?

[huangqiu15444]

In QAT (Quantization-Aware Training), why does the model fail to converge if a pre-trained model is not loaded?

[alexsu52]

Hello @huangqiu15444,

The best practice is to initialize the model with pre-trained weights, because:

1. If the model starts with random weights (without pre-training), the small changes introduced by quantization-aware operations can cause large deviations from an optimal path, making it much harder for the model to learn from scratch.
2. Without pre-trained weights, the learning process starts with highly inaccurate predictions and noisy gradients, making it difficult for the optimizer to find a good direction for improvement.