【PaddlePaddle Hackathon 4 No.53】：为 Paddle label_smooth 支持 float16 数据类型 #50921
Conversation
|
你的PR提交成功，感谢你对开源项目的贡献! |
thunder95
changed the title
| np.testing.assert_allclose(y_np_1, y_np_2, rtol=1e-03) | ||
| np.testing.assert_allclose(x_g_np_1, x_g_np_2, rtol=1e-03) | ||
|
|
||
|
|
There was a problem hiding this comment.
需要按照低精度单测规范，添加Op的单测。另外，添加完OpTest后，这里的case可以去掉了。https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/dev_guides/amp_precision/amp_test_dev_guide_cn.html#step2
| @@ -28,7 +30,8 @@ struct LabelSmoothGradFunctor { | |||
| } | |||
|
|
|||
| __device__ __forceinline__ T operator()(const T x) const { | |||
| return static_cast<T>(1 - epsilon) * x; | |||
| using MT = typename phi::dtype::MPTypeTrait<T>::Type; | |||
| return static_cast<T>((1 - static_cast<MT>(epsilon)) * static_cast<MT>(x)); | |||
There was a problem hiding this comment.
结合上面的代码。这里的epsilon在上面29行已经被cast到了FP16，这里又cast回FP32。
| using MT = typename phi::dtype::MPTypeTrait<T>::Type; | ||
| return static_cast<T>((1 - static_cast<MT>(epsilon)) * static_cast<MT>(x) + | ||
| static_cast<MT>(epsilon) / | ||
| static_cast<MT>(label_dim)); |
paddle/phi/kernels/gpu/prelu_funcs.h
Outdated
| @@ -137,6 +137,9 @@ void PreluChannelWiseDirectCUDAFunctor<T>::operator()(gpuStream_t stream, | |||
| stream>>>( | |||
| input, alpha, output, channel, numel); | |||
| } else { | |||
| printf("debug: spatial: %d, ch_num: %d\n", | |||
| static_cast<int>(numel / batch_size / channel), | |||
| static_cast<int>(channel)); | |||
|
记得提交中文文档的修改，并且在这个PR的描述中引用。 |
@zhangting2020 中文文档以修改，docs的PR链接已在描述中。 |
| } | ||
|
|
||
| __device__ __forceinline__ T operator()(const T x) const { | ||
| return static_cast<T>(1 - epsilon) * x; | ||
| using MT = typename phi::dtype::MPTypeTrait<T>::Type; |
| if core.is_compiled_with_cuda(): | ||
| place = core.CUDAPlace(0) | ||
| if core.is_float16_supported(place): | ||
| self.check_output_with_place(place, atol=1e-3) |
There was a problem hiding this comment.
OpTest Class中，为FP16设置了默认的atol为1e-3。另外在不支持的设备会自动跳过单测，所以test_check_output可以直接删掉，使用基类实现即可。
下面的test_check_grad可以去掉关于fp16支持情况的判断，简化代码
… labelsmooth_fp16
|
close due to the following PR is merged: |
PR types
Performance optimization
PR changes
OPs
Describe
为label_smooth 新增float16 数据类型
测试设备：RTX 2070s
目前label_smooth前向和反向推理性能测试：
中文API文档更新支持fp16数据类型: PaddlePaddle/docs#5642