Skip to content
This repository has been archived by the owner on Dec 31, 2020. It is now read-only.

nvDebugCheck fail with toFloat10 and toFloat11 with 0.f as input #267

Open
Zvork opened this issue Sep 13, 2017 · 0 comments
Open

nvDebugCheck fail with toFloat10 and toFloat11 with 0.f as input #267

Zvork opened this issue Sep 13, 2017 · 0 comments

Comments

@Zvork
Copy link

Zvork commented Sep 13, 2017

Float10 and Float11 encoding of zero (and very small float values) in toFloat10 and toFloat11 functions of CompressorRGB.cpp produce and assertion in the nvDebugCheck for the exponent field. The E variable is of type unsigned int and very small values produce a negative value of E, which when cast to unsigned gives a number greater than the 32 limit in nvDebugCheck. In this state, setting a compressor output format to R11_G11_B10F has a high chance of producing the assertion and failing.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant