add mixed data type support for GroupNorm backward on CPU #88663

CaoE · 2022-11-08T09:33:56Z

Stack from ghstack (oldest at bottom):

Motivation

Amp provides convenience methods for mixed precision. If users use amp to run bfloat16 models, torch.autocast will keep module parameters in acc dtype which will leave gamma and beta in float while input/output will be in bfloat16. The same goes for backward: parameters are in float, and X & dX & dY are in bfloat16.
Mixed data type support for GroupNorm backward is also needed for model training with GroupNorm.

Testing

Single socket (28cores):

Contiguous:

shape	forward / s	forward / s	backward / s	backward / s
	fp32	mixed fp32 bf16	fp32	mixed fp32 bf16
[10, 128, 20, 20]	3.08E-05	3.50E-05	8.06E-05	7.69E-05
[10, 128, 50, 50]	0.000121	0.000114	0.000358	0.000203

Channels Last (inputs and outputs will be converted to contiguous):

shape	forward / s	forward / s	backward / s	backward / s
	fp32	mixed fp32 bf16	fp32	mixed fp32 bf16
[10, 128, 20, 20]	4.04E-05	4.41E-05	0.000226	0.000305
[10, 128, 50, 50]	0.000169	0.000166	0.001628	0.001169

Single core:

Contiguous:

shape	forward / s	forward / s	backward / s	backward / s
	fp32	mixed fp32 bf16	fp32	mixed fp32 bf16
[10, 128, 20, 20]	2.38E-04	2.51E-04	5.94E-04	4.50E-04
[10, 128, 50, 50]	0.00171	0.001395	0.0044455	0.00243

Channels Last (inputs and outputs will be converted to contiguous):

shape	forward / s	forward / s	backward / s	backward / s
	fp32	mixed fp32 bf16	fp32	mixed fp32 bf16
[10, 128, 20, 20]	2.28E-04	3.26E-04	0.0016528	0.003165
[10, 128, 50, 50]	0.001788	0.001302	0.0276621	0.019447

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10

pytorch-bot · 2022-11-08T09:33:59Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88663

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 Failures

As of commit 6e34c00:

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 96cd113ab39009e646bfef5f440882e7ee7498f5 Pull Request resolved: #88663

[ghstack-poisoned]

cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…port for GroupNorm" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing [ghstack-poisoned]

### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing [ghstack-poisoned]

…port for GroupNorm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 [ghstack-poisoned]

…rm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 [ghstack-poisoned]

…port for GroupNorm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 [ghstack-poisoned]

…rm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 [ghstack-poisoned]

### Motivation Amp provides convenience methods for mixed precision. If users use amp to run bfloat16 models, torch.autocast will keep module parameters in acc dtype which will leave gamma and beta in float while input/output will be in bfloat16. The same goes for backward: parameters are in float, and X & dX & dY are in bfloat16. Mixed data type support for GroupNorm backward is also needed for model training with GroupNorm. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.08E-05 | 3.50E-05 | 8.06E-05 | 7.69E-05 [10, 128, 50, 50] | 0.000121 | 0.000114 | 0.000358 | 0.000203 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.04E-05 | 4.41E-05 | 0.000226 | 0.000305 [10, 128, 50, 50] | 0.000169 | 0.000166 | 0.001628 | 0.001169 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.38E-04 | 2.51E-04 | 5.94E-04 | 4.50E-04 [10, 128, 50, 50] | 0.00171 | 0.001395 | 0.0044455 | 0.00243 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.28E-04 | 3.26E-04 | 0.0016528 | 0.003165 [10, 128, 50, 50] | 0.001788 | 0.001302 | 0.0276621 | 0.019447 [ghstack-poisoned]

aten/src/ATen/native/cpu/group_norm_kernel.cpp

ghstack-source-id: d81889bfa3cfa987ed81a42a1a315f0b598c7644 Pull Request resolved: pytorch#88663

…rm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 [ghstack-poisoned]

### Motivation Amp provides convenience methods for mixed precision. If users use amp to run bfloat16 models, torch.autocast will keep module parameters in acc dtype which will leave gamma and beta in float while input/output will be in bfloat16. The same goes for backward: parameters are in float, and X & dX & dY are in bfloat16. Mixed data type support for GroupNorm backward is also needed for model training with GroupNorm. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.08E-05 | 3.50E-05 | 8.06E-05 | 7.69E-05 [10, 128, 50, 50] | 0.000121 | 0.000114 | 0.000358 | 0.000203 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.04E-05 | 4.41E-05 | 0.000226 | 0.000305 [10, 128, 50, 50] | 0.000169 | 0.000166 | 0.001628 | 0.001169 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.38E-04 | 2.51E-04 | 5.94E-04 | 4.50E-04 [10, 128, 50, 50] | 0.00171 | 0.001395 | 0.0044455 | 0.00243 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.28E-04 | 3.26E-04 | 0.0016528 | 0.003165 [10, 128, 50, 50] | 0.001788 | 0.001302 | 0.0276621 | 0.019447 [ghstack-poisoned]

### Motivation Amp provides convenience methods for mixed precision. If users use amp to run bfloat16 models, torch.autocast will keep module parameters in acc dtype which will leave gamma and beta in float while input/output will be in bfloat16. The same goes for backward: parameters are in float, and X & dX & dY are in bfloat16. Mixed data type support for GroupNorm backward is also needed for model training with GroupNorm. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.08E-05 | 3.50E-05 | 8.06E-05 | 7.69E-05 [10, 128, 50, 50] | 0.000121 | 0.000114 | 0.000358 | 0.000203 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.04E-05 | 4.41E-05 | 0.000226 | 0.000305 [10, 128, 50, 50] | 0.000169 | 0.000166 | 0.001628 | 0.001169 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.38E-04 | 2.51E-04 | 5.94E-04 | 4.50E-04 [10, 128, 50, 50] | 0.00171 | 0.001395 | 0.0044455 | 0.00243 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.28E-04 | 3.26E-04 | 0.0016528 | 0.003165 [10, 128, 50, 50] | 0.001788 | 0.001302 | 0.0276621 | 0.019447 cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

malfet · 2022-12-14T23:54:42Z

aten/src/ATen/native/cpu/group_norm_kernel.cpp

+      // It will keep module parameters in acc dtype i.e. float
+      // while input/output will be in BFloat16.
+      // Using parameters in BFloat16 will cause high precision loss.
+      if(mixed_type) {


Same feedback as for #81852 : please consider who this could be constrained to just BFloat16, perhaps it could be handled outside of the generic switch statement

Fixed in #89485

…port for GroupNorm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…rm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

### Motivation Amp provides convenience methods for mixed precision. If users use amp to run bfloat16 models, torch.autocast will keep module parameters in acc dtype which will leave gamma and beta in float while input/output will be in bfloat16. The same goes for backward: parameters are in float, and X & dX & dY are in bfloat16. Mixed data type support for GroupNorm backward is also needed for model training with GroupNorm. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.08E-05 | 3.50E-05 | 8.06E-05 | 7.69E-05 [10, 128, 50, 50] | 0.000121 | 0.000114 | 0.000358 | 0.000203 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.04E-05 | 4.41E-05 | 0.000226 | 0.000305 [10, 128, 50, 50] | 0.000169 | 0.000166 | 0.001628 | 0.001169 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.38E-04 | 2.51E-04 | 5.94E-04 | 4.50E-04 [10, 128, 50, 50] | 0.00171 | 0.001395 | 0.0044455 | 0.00243 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.28E-04 | 3.26E-04 | 0.0016528 | 0.003165 [10, 128, 50, 50] | 0.001788 | 0.001302 | 0.0276621 | 0.019447 cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…port for GroupNorm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…rm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…port for GroupNorm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…rm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

### Motivation Amp provides convenience methods for mixed precision. If users use amp to run bfloat16 models, torch.autocast will keep module parameters in acc dtype which will leave gamma and beta in float while input/output will be in bfloat16. The same goes for backward: parameters are in float, and X & dX & dY are in bfloat16. Mixed data type support for GroupNorm backward is also needed for model training with GroupNorm. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.08E-05 | 3.50E-05 | 8.06E-05 | 7.69E-05 [10, 128, 50, 50] | 0.000121 | 0.000114 | 0.000358 | 0.000203 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.04E-05 | 4.41E-05 | 0.000226 | 0.000305 [10, 128, 50, 50] | 0.000169 | 0.000166 | 0.001628 | 0.001169 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.38E-04 | 2.51E-04 | 5.94E-04 | 4.50E-04 [10, 128, 50, 50] | 0.00171 | 0.001395 | 0.0044455 | 0.00243 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.28E-04 | 3.26E-04 | 0.0016528 | 0.003165 [10, 128, 50, 50] | 0.001788 | 0.001302 | 0.0276621 | 0.019447 cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

CaoE · 2022-12-22T01:10:55Z

@pytorchbot merge

pytorchmergebot · 2022-12-22T01:12:36Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

) ### Motivation Amp provides convenience methods for mixed precision. If users use amp to run bfloat16 models, torch.autocast will keep module parameters in acc dtype which will leave gamma and beta in float while input/output will be in bfloat16. The same goes for backward: parameters are in float, and X & dX & dY are in bfloat16. Mixed data type support for GroupNorm backward is also needed for model training with GroupNorm. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.08E-05 | 3.50E-05 | 8.06E-05 | 7.69E-05 [10, 128, 50, 50] | 0.000121 | 0.000114 | 0.000358 | 0.000203 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.04E-05 | 4.41E-05 | 0.000226 | 0.000305 [10, 128, 50, 50] | 0.000169 | 0.000166 | 0.001628 | 0.001169 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.38E-04 | 2.51E-04 | 5.94E-04 | 4.50E-04 [10, 128, 50, 50] | 0.00171 | 0.001395 | 0.0044455 | 0.00243 * Channels Last (inputs and outputs will be converted to contiguous): shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.28E-04 | 3.26E-04 | 0.0016528 | 0.003165 [10, 128, 50, 50] | 0.001788 | 0.001302 | 0.0276621 | 0.019447 Pull Request resolved: pytorch#88663 Approved by: https://github.com/jgong5, https://github.com/mingfeima, https://github.com/malfet

…port for GroupNorm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…rm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…port for GroupNorm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…rm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…port for GroupNorm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…rm backward" ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

…#89485) ### Motivation 1. Add channels last support for GroupNorm backward to make sure GroupNorm fully support channels last. 2. Same as #88663, mixed data type support is also needed for channels last implementation of GroupNorm backward. ### Testing Single socket (28cores): * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 3.20E-05 | 3.60E-05 | 8.31E-05 | 8.13E-05 [10, 128, 50, 50] | 0.000126 | 0.000115 | 0.000356 | 0.000257 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 4.11E-05 | 4.12E-05 | 9.74E-05 | 9.66E-05 [10, 128, 50, 50] | 0.000179 | 0.000178 | 0.000393 | 0.000317 Single core: * Contiguous: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.47E-04 | 2.53E-04 | 5.92E-04 | 4.50E-04 [10, 128, 50, 50] | 0.001559 | 0.001384 | 0.004343 | 0.002436 * Channels Last: shape | forward / s | forward / s | backward / s | backward / s -- | -- | -- | -- | -- | fp32 | mixed fp32 bf16 | fp32 | mixed fp32 bf16 [10, 128, 20, 20] | 2.27E-04 | 3.24E-04 | 0.0006224 | 0.000459 [10, 128, 50, 50] | 0.00167 | 0.001278 | 0.0041858 | 0.003027 Pull Request resolved: #89485 Approved by: https://github.com/jgong5, https://github.com/malfet

swolchok · 2023-06-02T17:07:11Z

aten/src/ATen/native/cpu/group_norm_kernel.cpp

-      T db_val = std::accumulate(db_arr.cbegin(), db_arr.cend(), T(0));
+      const PT* ds_ptr = ds + i * D;
+      const PT* db_ptr = db + i * D;
+      const PT* gamma_ptr = gamma + g * D;


this is UB if gamma is null -- you can't add a nonzero offset to a null pointer

Thanks for the comments. The CalcDsDb method also checks whether gamma is null. Will modify this in #100234.

CaoE mentioned this pull request Nov 8, 2022

add mixed data type support for LayerNorm and GroupNorm #88662

Closed

pytorch-bot bot added the release notes: nn release notes category label Nov 8, 2022

github-actions bot added the module: cpu CPU specific problem (e.g., perf, algorithm) label Nov 8, 2022

CaoE added a commit that referenced this pull request Nov 8, 2022

add mixed data type support for GroupNorm backward on CPU

8cc81dd

ghstack-source-id: 96cd113ab39009e646bfef5f440882e7ee7498f5 Pull Request resolved: #88663

pytorchbot added the open source label Nov 8, 2022

CaoE mentioned this pull request Nov 8, 2022

add channels last with mixed data type support for GroupNorm backward #88664

Closed

CaoE added 2 commits November 8, 2022 16:32

add mixed data type support for GroupNorm backward on CPU

62d7eda

[ghstack-poisoned]

Update on "add mixed data type support for GroupNorm backward on CPU"

1607593

cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

CaoE marked this pull request as draft November 9, 2022 01:29

Update on "add mixed data type support for GroupNorm backward on CPU"

ee6bf0d

cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

CaoE requested a review from mingfeima November 9, 2022 01:54

CaoE added the intel This tag is for PR from Intel label Nov 9, 2022

Update on "add mixed data type support for GroupNorm backward on CPU"

aeae09f

cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

CaoE requested a review from jgong5 November 17, 2022 11:26

CaoE added 2 commits November 17, 2022 10:10

jgong5 requested changes Nov 18, 2022

View reviewed changes

CaoE added a commit to CaoE/pytorch that referenced this pull request Nov 22, 2022

add mixed data type support for GroupNorm backward on CPU

0b874ee

ghstack-source-id: d81889bfa3cfa987ed81a42a1a315f0b598c7644 Pull Request resolved: pytorch#88663

CaoE added a commit to CaoE/pytorch that referenced this pull request Nov 22, 2022

add mixed data type support for GroupNorm backward on CPU

5476877

ghstack-source-id: d81889bfa3cfa987ed81a42a1a315f0b598c7644 Pull Request resolved: pytorch#88663

CaoE mentioned this pull request Nov 22, 2022

add channels last with mixed data type support for GroupNorm backward #89485

Closed

CaoE added the intel priority matters to intel architecture from performance wise label Dec 13, 2022

CaoE requested a review from malfet December 13, 2022 05:42

CaoE marked this pull request as ready for review December 13, 2022 13:35

CaoE removed the large We think that this is a pretty chunky piece of work label Dec 13, 2022

jgong5 approved these changes Dec 14, 2022

View reviewed changes

malfet approved these changes Dec 14, 2022

View reviewed changes

pytorchmergebot added the Merged label Dec 22, 2022

pytorchmergebot closed this in 59a5be3 Dec 22, 2022

swolchok reviewed Jun 2, 2023

View reviewed changes

facebook-github-bot deleted the gh/CaoE/2/head branch June 8, 2023 14:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add mixed data type support for GroupNorm backward on CPU #88663

add mixed data type support for GroupNorm backward on CPU #88663

CaoE commented Nov 8, 2022 •

edited

pytorch-bot bot commented Nov 8, 2022 •

edited

malfet Dec 14, 2022

CaoE Dec 21, 2022

CaoE commented Dec 22, 2022

pytorchmergebot commented Dec 22, 2022

swolchok Jun 2, 2023

CaoE Jun 4, 2023

add mixed data type support for GroupNorm backward on CPU #88663

add mixed data type support for GroupNorm backward on CPU #88663

Conversation

CaoE commented Nov 8, 2022 • edited

Motivation

Testing

pytorch-bot bot commented Nov 8, 2022 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88663

❌ 4 Failures

malfet Dec 14, 2022

Choose a reason for hiding this comment

CaoE Dec 21, 2022

Choose a reason for hiding this comment

CaoE commented Dec 22, 2022

pytorchmergebot commented Dec 22, 2022

Merge started

swolchok Jun 2, 2023

Choose a reason for hiding this comment

CaoE Jun 4, 2023

Choose a reason for hiding this comment

CaoE commented Nov 8, 2022 •

edited

pytorch-bot bot commented Nov 8, 2022 •

edited