Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

src: cpu: aarch64: re-enable fp16 post-ops for aarch64 #1588

Merged
merged 1 commit into from
May 8, 2023

Conversation

fadara01
Copy link
Contributor

Perform the eltwise post-ops in fp32 instead of fp16 by casting up (to fp32) before executing the eltwise op and then casting back down (to fp16) after the operation completes. With this change, all fp16 benchdnn tests pass on aarch64.

Description

The oneDNN API specifies that post-ops need to (always) be executed in fp32, but the Arm Compute Library (ACL) executes them in f16 (when running in fp16 mode) which caused a lot of conv+relu benchdnn tests to fail.
To mitigate this, we submitted this RP (#1506) which disables ACL post-ops in fp16. As a result, all primitives which are followed by a post-op in fp16 started falling to the reference kernels.
Hence, a dramatic drop in performance for fp16.
This PR fixes the inaccuracies resulting from ACL executing the eltwise post-ops in fp16 by:

  • Not fusing any post-ops when in fp16 mode.
  • Casting the fp16 destination tensor up to fp32 before sending to ACL (for the eltwise post-op) - this forces ACL to run them in fp32 as required.
  • Casting the fp32 destination tensor back to fp16 after doing the eltwise.

With this change:

  • Eltwise post-op is executed in fp32 in accordance with the oneDNN guidelines.
  • Primitives in fp16 are executed by the ACL kernels, which are orders of magnitudes faster than ref, e.g. for this moderately sized convolution mb1ic128ih52oc256oh52kh3 folowed by a relu activation
    we run 76x faster on Neoverse-V1.

Checklist

General

  • [ yes ] Do all unit and benchdnn tests (make test and make test_benchdnn_*) pass locally for each commit?
  • [ yes ] Have you formatted the code using clang-format?

Performance improvements

  • [ X ] Have you submitted performance data that demonstrates performance improvements?

New features

  • [ N/A ] Have you published an RFC for the new feature?
  • [ N/A ] Was the RFC approved?
  • [ N/A ] Have you added relevant tests?

Bug fixes

  • [ N/A ] Have you included information on how to reproduce the issue (either in a github issue or in this PR)?
  • [ N/A ] Have you added relevant regression tests?

RFC PR

  • [ N/A ] Does RFC document follow the template?

Perform the eltwise post-ops in fp32 instead of fp16 by
casting up (to fp32) before executing the eltwise op and
then casting back down (to fp16) after the operation completes.
With this change, all fp16 benchdnn tests pass on aarch64.
@fadara01 fadara01 force-pushed the aarch64_enable_fp16_postops branch from 4446cd3 to 299c070 Compare April 19, 2023 14:25
@igorsafo igorsafo added this to the v3.2 milestone Apr 20, 2023
@dzarukin dzarukin merged commit f0229f0 into oneapi-src:master May 8, 2023
@dzarukin
Copy link
Contributor

dzarukin commented May 8, 2023

Thank you for the contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants