Fix machine epsilon for double

[rstm-sf]

What does the pull request do?

Fixing inaccuracy in machine epsilon found by @MarchingCube #4092 (comment)

What is the current behavior?

The result of methods from the MathUtilities class using machine epsilon was not always correct.

What is the updated/expected behavior with this PR?

The result of methods from the MathUtilities class using machine epsilon.

Checklist

• Added unit tests (if possible)?
• Added XML documentation to any related classes?
• Consider submitting a PR to https://github.com/AvaloniaUI/Avaloniaui.net with user documentation

[danwalmsley]

@rstm-sf is this something we should also upstream to dotnet? so that double.Epsilon is correct in the framework?
disclaimer I have no idea what this is ;)

[jmacato]

@rstm-sf i think this PR could be expanded elsewhere in the whole repo since we use double.Epsilon alot.

[rstm-sf]

@rstm-sf is this something we should also upstream to dotnet? so that double.Epsilon is correct in the framework?
disclaimer I have no idea what this is ;)

This good answer what is double.Epsilon ;)

[rstm-sf]

There is another class similar to MathUtilities. He has the next epsilon:

Avalonia/src/Avalonia.Controls.DataGrid/Utils/DoubleUtil.cs

Line 12 in 954c702

internal const double DBL_EPSILON = 1e-6;

Which is better to choose?

[MarchingCube]

There is another class similar to MathUtilities. He has the next epsilon:

Avalonia/src/Avalonia.Controls.DataGrid/Utils/DoubleUtil.cs

Line 12 in 954c702

internal const double DBL_EPSILON = 1e-6;

Which is better to choose?

Even WPF has like 3 copies of this class, each one with a different constant. Would need to figure what is the reason behind it.

[rstm-sf]

Even WPF has like 3 copies of this class, each one with a different constant. Would need to figure what is the reason behind it.

It seems to me that so historically. I prefer the single-precision option, since the pixels are integer?

[MarchingCube]

Even WPF has like 3 copies of this class, each one with a different constant. Would need to figure what is the reason behind it.

It seems to me that so historically. I prefer the single-precision option, since the pixels are integer?

Depends during which phase - during layout (and many of our calculation happen during this phase) pixels are floating point due to DPI scaling. So even if user is using integers values everything will get multiplied by a scaling factor. Because of that one must be quite careful since precision errors accumulate easily and one of the reasons why layout has own set of rounding functions.

[rstm-sf]

Ok, now I began to think that the error for double precision is better

[rstm-sf]

Maybe it’s worth just creating a methods with a float signature and a single precision check?

[MarchingCube]

Problem is that almost all of our calculations are done using double precision (I am still not sure why WPF is using double and not single precision), so exposing single precision API will both force users to cast and potentially introduce truncation errors.

[rstm-sf]

I mean, it’s better to add similar checks for single precision ;)

[MarchingCube]

LGTM.