add Half support for maxpool on CPU #98819

CaoE · 2023-04-11T05:52:10Z

Stack from ghstack (oldest at bottom):

-> add Half support for maxpool on CPU #98819

Testing

Single socket (28 cores):

shape	fp32 forward / ms	fp16 forward / ms	bf16 forward / ms	fp32 backward / ms	fp16 backward / ms	bf16 backward / ms
size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig	4.12895	6.9669	5.30297	0.55775	1.98917	0.72233
size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL	0.85093	1.88813	1.38063	5.5742	36.5086	10.58552
size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig	22.37212	37.90383	30.94482	6.85868	10.6116	3.9993
size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL	5.41658	4.71098	4.66578	6.69875	14.7171	5.1167
size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig	10.69831	18.0468	13.71657	2.61192	4.96172	1.68635
size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL	2.52637	2.0096	2.0055	2.60314	7.2093	2.49843
size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig	0.47605	0.88398	0.65326	0.06525	0.115489	0.0674
size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d	0.10902	0.25293	0.157475	0.11386	0.53319	0.17836

Single core:

shape	fp32 forward / ms	fp16 forward / ms	bf16 forward / ms	fp32 backward / ms	fp16 backward / ms	bf16 backward / ms
size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig	90.9809	163.473	126.1276	6.57721	41.40833	11.82505
size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL	9.88405	38.39137	29.62069	7.10636	36.97535	11.0525
size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig	476.782	855.4769	648.2248	46.6488	219.2586	67.10599
size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL	80.29271	91.33854	87.80345	48.81692	203.9974	63.39004
size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig	235.2113	419.0799	315.4284	20.6049	107.1524	32.39169
size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL	29.47653	33.54905	32.82823	22.59674	98.5586	30.05763
size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig	7.90684	13.9208	10.03272	0.23725	1.35269	0.41728
size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d	2.33638	3.36894	2.64635	0.26535	1.244	0.38895

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10

[ghstack-poisoned]

pytorch-bot · 2023-04-11T05:52:13Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/98819

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit dd17e4e with merge base 50fa588 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: e76294010171996fdba0dd15dbc43998bb12c73a Pull Request resolved: #98819

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: 3e27e12882fb7dda2f84f109da593f1de55dcfd6 Pull Request resolved: #98819

mingfeima

LGTM, just remember to add benchmark numbers.

CaoE · 2023-04-13T01:44:39Z

LGTM, just remember to add benchmark numbers.

Thank you, and I will add benchmark numbers.

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: 4d704dfed15b6fb9fe94edc116b791e17db0a6ea Pull Request resolved: #98819

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: 713c8b0eb404995655c1360b916f5cae4761bad4 Pull Request resolved: #98819

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: ea2b157c47c43fc1281000befa7191dd2eba6816 Pull Request resolved: #98819

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: bdf888857b0d74c8fce57eb93a5d429040d68cf7 Pull Request resolved: #98819

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: f43c46f15f67a5d07159fab39bb7c4428fc3bdd4 Pull Request resolved: #98819

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

### Motivation Scalar conversion between Half and Float on CPU is more time consuming compared to BFloat16 <-> Float. There is no direct data type conversion instruction for single Half value on CPU, so we add scalar conversion with avx instructions for Half to speed up. ### Testing Test maxpool, and compared with the results of #98819. Single socket (28 cores): shape | fp16 forward / ms | bf16 forward / ms | fp16 backward / ms | bf16 backward / ms | speedup ratio (fp16 forward) | speedup ratio (fp16 backward) -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 5.07165 | 5.418 | 0.5798 | 0.5123 | 1.373694951 | 3.430786 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 1.37455 | 1.2505 | 8.8336 | 9.7684 | 1.373635008 | 4.132924 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 28.72 | 30.7069 | 3.813 | 3.75 | 1.31977124 | 2.783006 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 4.5783 | 4.703 | 4.703 | 5.1 | 1.028980189 | 3.1293 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 13.896 | 14.8138 | 1.6635 | 1.6274 | 1.298704663 | 2.982699 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 2.11291 | 2.1158 | 2.26778 | 2.272 | 0.951105348 | 3.179012 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 0.4204 | 0.3843 | 0.0649 | 0.0633 | 2.102711703 | 1.779492 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 0.1134 | 0.11 | 0.1476 | 0.143 | 2.23042328 | 3.612398 Single core: shape | fp16 forward / ms | bf16 forward / ms | fp16 backward / ms | bf16 backward / ms | speedup ratio (fp16 forward) | speedup ratio (fp16 backward) -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 124.413 | 114.44 | 10.553 | 11.2486 | 1.31395433 | 3.923844 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 28.99 | 28.0781 | 9.5092 | 10.9258 | 1.324296999 | 3.888377 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 640.8276 | 591.964 | 59.18776 | 60.854 | 1.334956391 | 3.704458 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 88.57 | 90.214 | 54.358 | 59.205 | 1.031258214 | 3.75285 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 318.6197 | 285.155 | 28.4999 | 29.4387 | 1.315298144 | 3.759747 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 31.3981 | 34.0544 | 25.6557 | 28.7811 | 1.068505738 | 3.841587 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 8.87882 | 8.207 | 0.386056 | 0.3939 | 1.567866 | 3.50387 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 2.4167 | 2.38295 | 0.3769 | 0.4066 | 1.39402491 | 3.30061 Pull Request resolved: #102140 Approved by: https://github.com/jgong5, https://github.com/mingfeima, https://github.com/cpuhrsch

### Testing Single socket (28 cores): shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 4.12895 | 6.9669 | 5.30297 | 0.55775 | 1.98917 | 0.72233 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 0.85093 | 1.88813 | 1.38063 | 5.5742 | 36.5086 | 10.58552 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 22.37212 | 37.90383 | 30.94482 | 6.85868 | 10.6116 | 3.9993 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 5.41658 | 4.71098 | 4.66578 | 6.69875 | 14.7171 | 5.1167 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 10.69831 | 18.0468 | 13.71657 | 2.61192 | 4.96172 | 1.68635 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 2.52637 | 2.0096 | 2.0055 | 2.60314 | 7.2093 | 2.49843 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 0.47605 | 0.88398 | 0.65326 | 0.06525 | 0.115489 | 0.0674 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 0.10902 | 0.25293 | 0.157475 | 0.11386 | 0.53319 | 0.17836 Single core: shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 90.9809 | 163.473 | 126.1276 | 6.57721 | 41.40833 | 11.82505 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 9.88405 | 38.39137 | 29.62069 | 7.10636 | 36.97535 | 11.0525 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 476.782 | 855.4769 | 648.2248 | 46.6488 | 219.2586 | 67.10599 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 80.29271 | 91.33854 | 87.80345 | 48.81692 | 203.9974 | 63.39004 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 235.2113 | 419.0799 | 315.4284 | 20.6049 | 107.1524 | 32.39169 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 29.47653 | 33.54905 | 32.82823 | 22.59674 | 98.5586 | 30.05763 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 7.90684 | 13.9208 | 10.03272 | 0.23725 | 1.35269 | 0.41728 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 2.33638 | 3.36894 | 2.64635 | 0.26535 | 1.244 | 0.38895 cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: acd2ae93d5bf92a65bd4ac9fe3ae7a891800daa0 Pull Request resolved: #98819

### Testing Single socket (28 cores): shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 4.12895 | 6.9669 | 5.30297 | 0.55775 | 1.98917 | 0.72233 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 0.85093 | 1.88813 | 1.38063 | 5.5742 | 36.5086 | 10.58552 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 22.37212 | 37.90383 | 30.94482 | 6.85868 | 10.6116 | 3.9993 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 5.41658 | 4.71098 | 4.66578 | 6.69875 | 14.7171 | 5.1167 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 10.69831 | 18.0468 | 13.71657 | 2.61192 | 4.96172 | 1.68635 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 2.52637 | 2.0096 | 2.0055 | 2.60314 | 7.2093 | 2.49843 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 0.47605 | 0.88398 | 0.65326 | 0.06525 | 0.115489 | 0.0674 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 0.10902 | 0.25293 | 0.157475 | 0.11386 | 0.53319 | 0.17836 Single core: shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 90.9809 | 163.473 | 126.1276 | 6.57721 | 41.40833 | 11.82505 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 9.88405 | 38.39137 | 29.62069 | 7.10636 | 36.97535 | 11.0525 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 476.782 | 855.4769 | 648.2248 | 46.6488 | 219.2586 | 67.10599 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 80.29271 | 91.33854 | 87.80345 | 48.81692 | 203.9974 | 63.39004 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 235.2113 | 419.0799 | 315.4284 | 20.6049 | 107.1524 | 32.39169 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 29.47653 | 33.54905 | 32.82823 | 22.59674 | 98.5586 | 30.05763 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 7.90684 | 13.9208 | 10.03272 | 0.23725 | 1.35269 | 0.41728 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 2.33638 | 3.36894 | 2.64635 | 0.26535 | 1.244 | 0.38895 cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: 9bbc5b073d26ccd5518ffefa88b0c909298359b1 Pull Request resolved: #98819

ghstack-source-id: 9bbc5b073d26ccd5518ffefa88b0c909298359b1 Pull Request resolved: pytorch#98819

### Testing Single socket (28 cores): shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 4.12895 | 6.9669 | 5.30297 | 0.55775 | 1.98917 | 0.72233 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 0.85093 | 1.88813 | 1.38063 | 5.5742 | 36.5086 | 10.58552 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 22.37212 | 37.90383 | 30.94482 | 6.85868 | 10.6116 | 3.9993 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 5.41658 | 4.71098 | 4.66578 | 6.69875 | 14.7171 | 5.1167 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 10.69831 | 18.0468 | 13.71657 | 2.61192 | 4.96172 | 1.68635 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 2.52637 | 2.0096 | 2.0055 | 2.60314 | 7.2093 | 2.49843 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 0.47605 | 0.88398 | 0.65326 | 0.06525 | 0.115489 | 0.0674 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 0.10902 | 0.25293 | 0.157475 | 0.11386 | 0.53319 | 0.17836 Single core: shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 90.9809 | 163.473 | 126.1276 | 6.57721 | 41.40833 | 11.82505 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 9.88405 | 38.39137 | 29.62069 | 7.10636 | 36.97535 | 11.0525 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 476.782 | 855.4769 | 648.2248 | 46.6488 | 219.2586 | 67.10599 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 80.29271 | 91.33854 | 87.80345 | 48.81692 | 203.9974 | 63.39004 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 235.2113 | 419.0799 | 315.4284 | 20.6049 | 107.1524 | 32.39169 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 29.47653 | 33.54905 | 32.82823 | 22.59674 | 98.5586 | 30.05763 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 7.90684 | 13.9208 | 10.03272 | 0.23725 | 1.35269 | 0.41728 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 2.33638 | 3.36894 | 2.64635 | 0.26535 | 1.244 | 0.38895 cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: 684b3bb039749812f9742b0d0f63e7690e995f3e Pull Request resolved: #98819

### Testing Single socket (28 cores): shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 4.12895 | 6.9669 | 5.30297 | 0.55775 | 1.98917 | 0.72233 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 0.85093 | 1.88813 | 1.38063 | 5.5742 | 36.5086 | 10.58552 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 22.37212 | 37.90383 | 30.94482 | 6.85868 | 10.6116 | 3.9993 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 5.41658 | 4.71098 | 4.66578 | 6.69875 | 14.7171 | 5.1167 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 10.69831 | 18.0468 | 13.71657 | 2.61192 | 4.96172 | 1.68635 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 2.52637 | 2.0096 | 2.0055 | 2.60314 | 7.2093 | 2.49843 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 0.47605 | 0.88398 | 0.65326 | 0.06525 | 0.115489 | 0.0674 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 0.10902 | 0.25293 | 0.157475 | 0.11386 | 0.53319 | 0.17836 Single core: shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 90.9809 | 163.473 | 126.1276 | 6.57721 | 41.40833 | 11.82505 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 9.88405 | 38.39137 | 29.62069 | 7.10636 | 36.97535 | 11.0525 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 476.782 | 855.4769 | 648.2248 | 46.6488 | 219.2586 | 67.10599 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 80.29271 | 91.33854 | 87.80345 | 48.81692 | 203.9974 | 63.39004 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 235.2113 | 419.0799 | 315.4284 | 20.6049 | 107.1524 | 32.39169 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 29.47653 | 33.54905 | 32.82823 | 22.59674 | 98.5586 | 30.05763 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 7.90684 | 13.9208 | 10.03272 | 0.23725 | 1.35269 | 0.41728 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 2.33638 | 3.36894 | 2.64635 | 0.26535 | 1.244 | 0.38895 cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: 2d02a08cd4a72457613812f94ab8386621fa4bf4 Pull Request resolved: #98819

### Motivation Scalar conversion between Half and Float on CPU is more time consuming compared to BFloat16 <-> Float. There is no direct data type conversion instruction for single Half value on CPU, so we add scalar conversion with avx instructions for Half to speed up. ### Testing Test maxpool, and compared with the results of #98819. Single socket (28 cores): shape | fp16 forward / ms | bf16 forward / ms | fp16 backward / ms | bf16 backward / ms | speedup ratio (fp16 forward) | speedup ratio (fp16 backward) -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 5.07165 | 5.418 | 0.5798 | 0.5123 | 1.373694951 | 3.430786 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 1.37455 | 1.2505 | 8.8336 | 9.7684 | 1.373635008 | 4.132924 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 28.72 | 30.7069 | 3.813 | 3.75 | 1.31977124 | 2.783006 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 4.5783 | 4.703 | 4.703 | 5.1 | 1.028980189 | 3.1293 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 13.896 | 14.8138 | 1.6635 | 1.6274 | 1.298704663 | 2.982699 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 2.11291 | 2.1158 | 2.26778 | 2.272 | 0.951105348 | 3.179012 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 0.4204 | 0.3843 | 0.0649 | 0.0633 | 2.102711703 | 1.779492 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 0.1134 | 0.11 | 0.1476 | 0.143 | 2.23042328 | 3.612398 Single core: shape | fp16 forward / ms | bf16 forward / ms | fp16 backward / ms | bf16 backward / ms | speedup ratio (fp16 forward) | speedup ratio (fp16 backward) -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 124.413 | 114.44 | 10.553 | 11.2486 | 1.31395433 | 3.923844 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 28.99 | 28.0781 | 9.5092 | 10.9258 | 1.324296999 | 3.888377 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 640.8276 | 591.964 | 59.18776 | 60.854 | 1.334956391 | 3.704458 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 88.57 | 90.214 | 54.358 | 59.205 | 1.031258214 | 3.75285 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 318.6197 | 285.155 | 28.4999 | 29.4387 | 1.315298144 | 3.759747 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 31.3981 | 34.0544 | 25.6557 | 28.7811 | 1.068505738 | 3.841587 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 8.87882 | 8.207 | 0.386056 | 0.3939 | 1.567866 | 3.50387 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 2.4167 | 2.38295 | 0.3769 | 0.4066 | 1.39402491 | 3.30061 Pull Request resolved: #102140 Approved by: https://github.com/jgong5, https://github.com/mingfeima, https://github.com/cpuhrsch

### Testing Single socket (28 cores): shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 4.12895 | 6.9669 | 5.30297 | 0.55775 | 1.98917 | 0.72233 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 0.85093 | 1.88813 | 1.38063 | 5.5742 | 36.5086 | 10.58552 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 22.37212 | 37.90383 | 30.94482 | 6.85868 | 10.6116 | 3.9993 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 5.41658 | 4.71098 | 4.66578 | 6.69875 | 14.7171 | 5.1167 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 10.69831 | 18.0468 | 13.71657 | 2.61192 | 4.96172 | 1.68635 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 2.52637 | 2.0096 | 2.0055 | 2.60314 | 7.2093 | 2.49843 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 0.47605 | 0.88398 | 0.65326 | 0.06525 | 0.115489 | 0.0674 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 0.10902 | 0.25293 | 0.157475 | 0.11386 | 0.53319 | 0.17836 Single core: shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 90.9809 | 163.473 | 126.1276 | 6.57721 | 41.40833 | 11.82505 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 9.88405 | 38.39137 | 29.62069 | 7.10636 | 36.97535 | 11.0525 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 476.782 | 855.4769 | 648.2248 | 46.6488 | 219.2586 | 67.10599 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 80.29271 | 91.33854 | 87.80345 | 48.81692 | 203.9974 | 63.39004 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 235.2113 | 419.0799 | 315.4284 | 20.6049 | 107.1524 | 32.39169 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 29.47653 | 33.54905 | 32.82823 | 22.59674 | 98.5586 | 30.05763 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 7.90684 | 13.9208 | 10.03272 | 0.23725 | 1.35269 | 0.41728 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 2.33638 | 3.36894 | 2.64635 | 0.26535 | 1.244 | 0.38895 cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: 15fdbd125bc46a681b342bafada3e0858c1e6cfe Pull Request resolved: #98819

### Testing Single socket (28 cores): shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 4.12895 | 6.9669 | 5.30297 | 0.55775 | 1.98917 | 0.72233 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 0.85093 | 1.88813 | 1.38063 | 5.5742 | 36.5086 | 10.58552 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 22.37212 | 37.90383 | 30.94482 | 6.85868 | 10.6116 | 3.9993 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 5.41658 | 4.71098 | 4.66578 | 6.69875 | 14.7171 | 5.1167 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 10.69831 | 18.0468 | 13.71657 | 2.61192 | 4.96172 | 1.68635 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 2.52637 | 2.0096 | 2.0055 | 2.60314 | 7.2093 | 2.49843 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 0.47605 | 0.88398 | 0.65326 | 0.06525 | 0.115489 | 0.0674 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 0.10902 | 0.25293 | 0.157475 | 0.11386 | 0.53319 | 0.17836 Single core: shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 90.9809 | 163.473 | 126.1276 | 6.57721 | 41.40833 | 11.82505 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 9.88405 | 38.39137 | 29.62069 | 7.10636 | 36.97535 | 11.0525 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 476.782 | 855.4769 | 648.2248 | 46.6488 | 219.2586 | 67.10599 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 80.29271 | 91.33854 | 87.80345 | 48.81692 | 203.9974 | 63.39004 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 235.2113 | 419.0799 | 315.4284 | 20.6049 | 107.1524 | 32.39169 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 29.47653 | 33.54905 | 32.82823 | 22.59674 | 98.5586 | 30.05763 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 7.90684 | 13.9208 | 10.03272 | 0.23725 | 1.35269 | 0.41728 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 2.33638 | 3.36894 | 2.64635 | 0.26535 | 1.244 | 0.38895 cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: 313d2fa857da72b42ce6e099b886d7b12187e290 Pull Request resolved: #98819

CaoE · 2023-08-31T02:37:56Z

@mikaylagawarecki For the memory format tests of maxpool defined in https://github.com/pytorch/pytorch/blob/main/test/nn/test_pooling.py#L878, it's hard to be replace by test_memory_format, as they include tests with stride and batch=0.

CaoE · 2023-08-31T02:56:26Z

@ngimel Could you please review this PR ? Thanks.

### Testing Single socket (28 cores): shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 4.12895 | 6.9669 | 5.30297 | 0.55775 | 1.98917 | 0.72233 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 0.85093 | 1.88813 | 1.38063 | 5.5742 | 36.5086 | 10.58552 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 22.37212 | 37.90383 | 30.94482 | 6.85868 | 10.6116 | 3.9993 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 5.41658 | 4.71098 | 4.66578 | 6.69875 | 14.7171 | 5.1167 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 10.69831 | 18.0468 | 13.71657 | 2.61192 | 4.96172 | 1.68635 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 2.52637 | 2.0096 | 2.0055 | 2.60314 | 7.2093 | 2.49843 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 0.47605 | 0.88398 | 0.65326 | 0.06525 | 0.115489 | 0.0674 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 0.10902 | 0.25293 | 0.157475 | 0.11386 | 0.53319 | 0.17836 Single core: shape | fp32 forward / ms | fp16 forward / ms | bf16 forward / ms | fp32 backward / ms | fp16 backward / ms | bf16 backward / ms -- | -- | -- | -- | -- | -- | -- size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: contig | 90.9809 | 163.473 | 126.1276 | 6.57721 | 41.40833 | 11.82505 size: (1, 56, 264, 264), kernel: 3, stride: 1, mem_format: CL | 9.88405 | 38.39137 | 29.62069 | 7.10636 | 36.97535 | 11.0525 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: contig | 476.782 | 855.4769 | 648.2248 | 46.6488 | 219.2586 | 67.10599 size: (32, 16, 200, 200), kernel: 3, stride: 1, mem_format: CL | 80.29271 | 91.33854 | 87.80345 | 48.81692 | 203.9974 | 63.39004 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: contig | 235.2113 | 419.0799 | 315.4284 | 20.6049 | 107.1524 | 32.39169 size: (32, 32, 100, 100), kernel: 3, stride: 1, mem_format: CL | 29.47653 | 33.54905 | 32.82823 | 22.59674 | 98.5586 | 30.05763 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: contig | 7.90684 | 13.9208 | 10.03272 | 0.23725 | 1.35269 | 0.41728 size: (4, 19, 10, 16, 16), kernel: 3, stride: 1, mem_format: CL3d | 2.33638 | 3.36894 | 2.64635 | 0.26535 | 1.244 | 0.38895 cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

ghstack-source-id: b2656301420ef5ea5416d471b2f9dc35936b773f Pull Request resolved: #98819

mikaylagawarecki · 2023-09-01T18:01:51Z

test/test_mps.py

@@ -10754,10 +10757,6 @@ def req_grad(t):
            cpu_grad_inputs = torch.autograd.grad(diff_cpu_out, diff_cpu_arg, grad_outputs=cpu_grad_outputs, allow_unused=True)
            mps_grad_inputs = torch.autograd.grad(diff_mps_out, diff_mps_arg, grad_outputs=mps_grad_outputs, allow_unused=True)

-            if op.name in ["nn.functional.gelu", "nn.functional.glu"] and dtype == torch.float16:


out of curiosity: why this change, do gelu/glu use max_pool?

Just move gelu and glu to FP16_LOW_PRECISION_LIST to handle such cases uniformly.

oh, I see that you had added this path in a previous PR, got it, thanks!

mikaylagawarecki · 2023-09-05T18:19:52Z

@pytorchbot merge

pytorchmergebot · 2023-09-05T18:23:34Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

add Half support for maxpool3d on CPU

e350784

[ghstack-poisoned]

CaoE mentioned this pull request Apr 11, 2023

add channel last 3d support for batch_norm on CPU #97774

Closed

CaoE mentioned this pull request Apr 11, 2023

add channel last 3d support for maxpool3d on CPU #97775

Closed

github-actions bot added the module: cpu CPU specific problem (e.g., perf, algorithm) label Apr 11, 2023

CaoE added a commit that referenced this pull request Apr 11, 2023

add Half support for maxpool3d on CPU

2a6468d

ghstack-source-id: e76294010171996fdba0dd15dbc43998bb12c73a Pull Request resolved: #98819

CaoE marked this pull request as draft April 11, 2023 05:55

pytorchbot added the open source label Apr 11, 2023

CaoE requested a review from mingfeima April 11, 2023 06:11

CaoE added the topic: not user facing topic category label Apr 11, 2023

Update on "add Half support for maxpool3d on CPU"

1e1b0e5

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

CaoE added a commit that referenced this pull request Apr 11, 2023

add Half support for maxpool3d on CPU

952ee0d

ghstack-source-id: 3e27e12882fb7dda2f84f109da593f1de55dcfd6 Pull Request resolved: #98819

mingfeima approved these changes Apr 11, 2023

View reviewed changes

mingfeima added ciflow/trunk Trigger trunk jobs on your pull request module: half Related to float16 half-precision floats labels Apr 11, 2023

Update on "add Half support for maxpool3d on CPU"

e94aba5

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

CaoE added a commit that referenced this pull request Apr 13, 2023

add Half support for maxpool3d on CPU

3e43178

ghstack-source-id: 4d704dfed15b6fb9fe94edc116b791e17db0a6ea Pull Request resolved: #98819

Update on "add Half support for maxpool3d on CPU"

1e80553

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

CaoE added a commit that referenced this pull request Apr 13, 2023

add Half support for maxpool3d on CPU

11899a5

ghstack-source-id: 713c8b0eb404995655c1360b916f5cae4761bad4 Pull Request resolved: #98819

Update on "add Half support for maxpool3d on CPU"

4a7c0e1

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

CaoE added a commit that referenced this pull request Apr 14, 2023

add Half support for maxpool3d on CPU

afe34f3

ghstack-source-id: ea2b157c47c43fc1281000befa7191dd2eba6816 Pull Request resolved: #98819

CaoE mentioned this pull request Apr 16, 2023

use single implementation for max_pool2d/max_pool3d on CPU #99261

Closed

pytorch-bot bot added the ciflow/mps Run MPS tests (subset of trunk) label Apr 16, 2023

Update on "add Half support for maxpool3d on CPU"

0aa0a25

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

CaoE added a commit that referenced this pull request May 4, 2023

add Half support for maxpool3d on CPU

3dd3bd6

ghstack-source-id: bdf888857b0d74c8fce57eb93a5d429040d68cf7 Pull Request resolved: #98819

Update on "add Half support for maxpool3d on CPU"

cacdc1a

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

CaoE added a commit that referenced this pull request May 9, 2023

add Half support for maxpool3d on CPU

25939d4

ghstack-source-id: f43c46f15f67a5d07159fab39bb7c4428fc3bdd4 Pull Request resolved: #98819

CaoE changed the title ~~add Half support for maxpool3d on CPU~~ add Half support for maxpool on CPU May 10, 2023

Update on "add Half support for maxpool on CPU"

251f608

cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]

kkontny mentioned this pull request Aug 16, 2023

2x performance drop using pytorch depending on how input data is fed into model AmpereComputingAI/ampere_model_library#198

Closed

CaoE added a commit that referenced this pull request Aug 18, 2023

add Half support for maxpool3d on CPU

5daee2c

ghstack-source-id: acd2ae93d5bf92a65bd4ac9fe3ae7a891800daa0 Pull Request resolved: #98819

CaoE added a commit that referenced this pull request Aug 25, 2023

add Half support for maxpool3d on CPU

d52f2a8

ghstack-source-id: 9bbc5b073d26ccd5518ffefa88b0c909298359b1 Pull Request resolved: #98819

CaoE added a commit to CaoE/pytorch that referenced this pull request Aug 25, 2023

add Half support for maxpool3d on CPU

30d4396

ghstack-source-id: 9bbc5b073d26ccd5518ffefa88b0c909298359b1 Pull Request resolved: pytorch#98819

CaoE added a commit that referenced this pull request Aug 25, 2023

add Half support for maxpool3d on CPU

ead2ec5

ghstack-source-id: 684b3bb039749812f9742b0d0f63e7690e995f3e Pull Request resolved: #98819

CaoE added a commit that referenced this pull request Aug 29, 2023

add Half support for maxpool3d on CPU

8339c4d

ghstack-source-id: 2d02a08cd4a72457613812f94ab8386621fa4bf4 Pull Request resolved: #98819

CaoE requested a review from mikaylagawarecki August 29, 2023 06:44

CaoE added a commit that referenced this pull request Aug 31, 2023

add Half support for maxpool3d on CPU

684ba8a

ghstack-source-id: 15fdbd125bc46a681b342bafada3e0858c1e6cfe Pull Request resolved: #98819

CaoE added a commit that referenced this pull request Aug 31, 2023

add Half support for maxpool3d on CPU

36fc8df

ghstack-source-id: 313d2fa857da72b42ce6e099b886d7b12187e290 Pull Request resolved: #98819

CaoE added a commit that referenced this pull request Sep 1, 2023

add Half support for maxpool on CPU

cbd0c80

ghstack-source-id: b2656301420ef5ea5416d471b2f9dc35936b773f Pull Request resolved: #98819

mikaylagawarecki approved these changes Sep 1, 2023

View reviewed changes

pytorchmergebot added the merging label Sep 5, 2023

pytorchmergebot added Merged and removed merging labels Sep 5, 2023

pytorchmergebot closed this in 42f94d7 Sep 5, 2023

facebook-github-bot deleted the gh/CaoE/12/head branch September 9, 2023 14:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add Half support for maxpool on CPU #98819

add Half support for maxpool on CPU #98819

CaoE commented Apr 11, 2023 •

edited

pytorch-bot bot commented Apr 11, 2023 •

edited

mingfeima left a comment

CaoE commented Apr 13, 2023

CaoE commented Aug 31, 2023

CaoE commented Aug 31, 2023

mikaylagawarecki Sep 1, 2023 •

edited

CaoE Sep 4, 2023

mikaylagawarecki Sep 5, 2023

mikaylagawarecki commented Sep 5, 2023

pytorchmergebot commented Sep 5, 2023

add Half support for maxpool on CPU #98819

add Half support for maxpool on CPU #98819

Conversation

CaoE commented Apr 11, 2023 • edited

Testing

pytorch-bot bot commented Apr 11, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/98819

✅ No Failures

mingfeima left a comment

Choose a reason for hiding this comment

CaoE commented Apr 13, 2023

CaoE commented Aug 31, 2023

CaoE commented Aug 31, 2023

mikaylagawarecki Sep 1, 2023 • edited

Choose a reason for hiding this comment

CaoE Sep 4, 2023

Choose a reason for hiding this comment

mikaylagawarecki Sep 5, 2023

Choose a reason for hiding this comment

mikaylagawarecki commented Sep 5, 2023

pytorchmergebot commented Sep 5, 2023

Merge started

CaoE commented Apr 11, 2023 •

edited

pytorch-bot bot commented Apr 11, 2023 •

edited

mikaylagawarecki Sep 1, 2023 •

edited