Improve batch_norm fp16 accuracy #1450
Conversation
github-actions
bot
added
component: conversion
mfeliz-cruise
changed the title
| auto calculation_beta = beta; | ||
| auto calculation_mean = mean; | ||
| auto calculation_var = var; | ||
| if (orig_dtype == torch::kHalf) { |
There was a problem hiding this comment.
Quick question is this different than the normal pytorch behavior? If so can we add a debug message here saying that we are doing this to improve accuracy?
There was a problem hiding this comment.
It looks like the cudnn implementation at least asserts that the weight is fp32 which would force similar calculations to fp32:
https://github.com/pytorch/pytorch/blob/4bfe2a24505049fa4fe43d24c2e3a5f5d99d9f00/aten/src/ATen/native/cudnn/BatchNorm.cpp#L110
|
Seems like the BN FP16 test is failing, maybe we need to change the metric for this test since I guess by definition the numerics will be different? |
I lowered the bar to 2e-4 which may be more reasonable. Both pass locally for me so it may depend on GPU and the numerics of the tactic chosen by TensorRT. |
mfeliz-cruise
force-pushed
the
michael.feliz/improve_batchnorm_fp16_accuracy
branch
from
3cd10ec
to
609ddc6
Compare
|
I rebased this today, but it looks like there's a CI failure on pytorch 1.14 currently: |
# Description Use float types for compile-time calculations around batch_norm. Improves fp16 accuracy relative to pytorch. Fixes # (issue) ## Type of change Please delete options that are not relevant and/or add your own. - Bug fix (non-breaking change which fixes an issue) - New feature (non-breaking change which adds functionality) - Breaking change (fix or feature that would cause existing functionality to not work as expected) - This change requires a documentation update # Checklist: - [ ] My code follows the style guidelines of this project (You can use the linters) - [ ] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas and hacks - [ ] I have made corresponding changes to the documentation - [ ] I have added tests to verify my fix or my feature - [ ] New and existing unit tests pass locally with my changes - [ ] I have added the relevant labels to my PR in so that relevant reviewers are notified
mfeliz-cruise
force-pushed
the
michael.feliz/improve_batchnorm_fp16_accuracy
branch
from
609ddc6
to
6468416
Compare
mfeliz-cruise
force-pushed
the
michael.feliz/improve_batchnorm_fp16_accuracy
branch
from
6468416
to
0b0666a
Compare
Description
Use float types for compile-time calculations around batch_norm. Improves fp16 accuracy relative to pytorch. Fixes # (issue)
Type of change
Please delete options that are not relevant and/or add your own.
Checklist: