Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize batchnorm1d using 2D kernel #43530

Merged
merged 24 commits into from
Jul 14, 2022
Merged

Optimize batchnorm1d using 2D kernel #43530

Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
441da36
refactor code structure
EsdeathYZH May 29, 2022
c48e076
add native kernel usage
EsdeathYZH May 29, 2022
0a68ba3
add wellford impl
EsdeathYZH Jun 3, 2022
b3248c9
add shmem impl
EsdeathYZH Jun 4, 2022
78349a2
add dispatch logic
EsdeathYZH Jun 4, 2022
98c66f0
add channel_last impl
EsdeathYZH Jun 4, 2022
570dc55
refine the global space init
EsdeathYZH Jun 6, 2022
aaca04a
impl 2d kernel
EsdeathYZH Jun 7, 2022
29ef723
Merge remote-tracking branch 'paddle/develop' into optim_batchnorm1d
EsdeathYZH Jun 11, 2022
74b792b
rm wellford
EsdeathYZH Jun 11, 2022
a0bd5b6
fix backward
EsdeathYZH Jun 11, 2022
2433ebf
add unit test for batchnorm1d
EsdeathYZH Jun 11, 2022
90c27a6
fix bug
EsdeathYZH Jun 11, 2022
91d83e5
impl channel last 2d
EsdeathYZH Jun 11, 2022
6871dbf
refine
EsdeathYZH Jun 15, 2022
0571ecc
fix memory thpt
EsdeathYZH Jun 15, 2022
804ba03
fix threshold
EsdeathYZH Jun 16, 2022
48c6344
fix backward threshold
EsdeathYZH Jun 16, 2022
6785f6f
refine unit test
EsdeathYZH Jun 16, 2022
e46ef54
refine test
EsdeathYZH Jun 16, 2022
938cde3
delete pragma unroll
EsdeathYZH Jun 17, 2022
a79206f
Merge remote-tracking branch 'paddle/develop' into optim_batchnorm1d
EsdeathYZH Jun 24, 2022
44ad03e
refine code
EsdeathYZH Jun 28, 2022
0b5ca0e
fix
EsdeathYZH Jul 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 4 additions & 2 deletions paddle/phi/kernels/gpu/batch_norm_grad_kernel.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -591,10 +591,12 @@ void BatchNormGradRawKernel(const Context &ctx,
// ctx.GetPlace()),
// epsilon, saved_mean_data, saved_var_data));
#else
// CUDNN PER_ACTIVATION mode only support small batch size
// CUDNN only support small batch size
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

和版本有关吗 ？

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

目前用到的版本是不支持太大的batch_size。

const size_t CUDNN_PER_ACTIVATION_THRESHOLD = 131070;
const size_t CUDNN_SPATIAL_THRESHOLD = 880801;
const bool use_native_kernel =
(x_dims.size() == 2 && N >= CUDNN_PER_ACTIVATION_THRESHOLD);
((x_dims.size() == 2 && N >= CUDNN_PER_ACTIVATION_THRESHOLD) ||
(x_dims.size() == 3 && N >= CUDNN_SPATIAL_THRESHOLD));
if (use_native_kernel) {
if (compute_format == DataLayout::kNCHW) {
BNBackward<T, block, DataLayout::kNCHW>
Expand Down