Questions about the squeeze-and-exicitation module in PPHGNet series model

[thsno02]

Q: Is there any justification for using the ESE module instead of the SE module in PPHGNetv1?

The design of PPHGNetv1 strikes a balance between computational efficiency and model performance. However, the ESE module has more parameters than the SE module. The ESE module consists of a fully connected layer without dimension reduction, resulting in a parameter size of $C * C$. In contrast, the SE module includes two fully connected layers with a dimension reduction ratio of $r$, leading to a parameter size of $C * C / r + C/r * C$, which simplifies to $2 * C * C / r$. If $r$ is greater than 2, the SE module has fewer parameters. The original paper recommends setting $r$ to 16¹.

When considering the model's inference performance, it appears that using the ES module would be faster. So, why opt for the ESE² module in v1? Additionally, v2 employs an ES module with a ratio of 2 (if I understand correctly³). Would the performance of the ES module be superior while maintaining a consistent number of parameters?

From my perspective, the SE module may have an advantage in capturing inter-channel dependencies, given its utilization of dimension reduction as a form of compression. On the other hand, the ESE module might not be as effective in this regard.

Footnotes

1. https://arxiv.org/abs/1709.01507 ↩

2. https://github.com/PaddlePaddle/PaddleClas/blob/e35611697a3514f24289f170e115a12399f6a342/ppcls/arch/backbone/legendary_models/pp_hgnet.py#L74-L84 ↩

3. https://github.com/PaddlePaddle/PaddleClas/blob/2bc2fc0348e5778291362d64781830b142e9d85e/ppcls/arch/backbone/legendary_models/pp_hgnet_v2.py#L299-L312 ↩

[cuicheng01]

Hello, thank you for your interest in PP-HGNet. In PP-HGNetV1, we did consider that ESE module takes less inference time than SE module without reducing accuracy, so we adopted the ESE module. However, we further found that the ESE module or SE module did not significantly improve object detection, semantic segmentation, and other tasks. Therefore, in PP-HGNetV2, we removed this module. In summary, there is neither SE module nor ESE module in PP-HGNetV2.