Skip to content

Long Vector 16-bit trig function tolerances #7936

@alsepkow

Description

@alsepkow

Some of the trig functions have tolerance issues for fp16. It sounds like we just need to adjust the tolerance for fp16 as it is currently defaulted to .0008, but that is intended for floats.

DxilConf_SM69_Vectorized::Asin_HLSLHalf_t

    Error: Mismatch at Index: 5 Actual Value:-1.046875, Expected Value:-1.0458984375           diff = 0.0009765625

    Error: Mismatch at Index: 8 Actual Value:-0.927734375, Expected Value:-0.9267578125   diff = 0.0009765625

DxilConf_SM69_Vectorized::Tan_HLSLHalf_t

    Error: Mismatch at Index: 0 Actual Value:-1.5556640625, Expected Value:-1.5576171875   diff = 0.001953125

DxilConf_SM69_Vectorized::Acos_HLSLHalf_t

    Error: Mismatch at Index: 9 Actual Value:2.703125, Expected Value:2.701171875                   diff = 0.001953125

Maybe the tolerance for all of these should just be set to 0.003?

Update: After some further discussion we decided to use 0.003 as the epsilon for the half trig functions with a bounded range. For half trig functions that can go to infinity we will use 2 ULP

Metadata

Metadata

Assignees

Labels

bugBug, regression, crash

Type

No type

Projects

Status

Triaged

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions