Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

f32 -> u8 conversion should multiply by 256-ε instead of by 255 #13

Open
nigeltao opened this issue Aug 28, 2016 · 1 comment
Open

Comments

@nigeltao
Copy link

The Rust code has SIMD and non-SIMD versions of this conversion
(255.0 * y) as u8
from an f32 in the range [0.0, 1.0] to a u8 in the range [0, 255].

Due to rounding errors, this sometimes means that a pixel well inside a glyph ends up with an 0xfe value instead of 0xff, since its floating point value is slightly less than 1.

In my Go port, I embiggened the constant from 255.0 to 255.99998, or 0x437fffff as a float32 bit pattern, which eliminated these central "0xfe"s. See the comment at the bottom of google/font-go@7c31afd

I think that the Rust code should do the same.

@ishitatsuyuki
Copy link

I'd suggest using centered quantization which is widely used in games, basically it can be implemented by using round instead of floor in the final step of calculation.

Though I don't think accuracy matters in the case of fonts (apart from avoiding triggering a different codepath due to 0xff turning into 0xfe), so your approach also seems OK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants