Scientific Notation - Precision

[0xferit]

I changed a test value from 14173.2283465 to 1.41732283465E4, in SpeedTest.cs, and it made tests fail.

When I debugged I saw that there is a small difference in actual and expected. So scientific notation seems less precise.

Do you know anything about this? If this is the case, we should not use scientific notation in test values I think (which was what I'm doing so far in my commits, as it looks nicer).

[angularsen]

I guess you are at the very edge of passing the tests with the default max error of 1e-5 for that particular unit vs base unit. I guess I'm not surprised there is a slight difference in precision with and without scientific notation, but I suspect it's more of a random deviation rather than "scientific notation is always less precise" - but that is just speculation on my part.

As for test values, they don't need to be in scientific notation really. The key thing is for them to verify the conversion to some known precision, so either keep the value as-is or override the max error for that unit to 1e-4.

[0xferit]

I wouldn't prefer increasing tolerance value. I think we should strive for lowest possible tolerance. So let's keep test values as is instead of writing in scientific notation.

Also, I wonder, is there a specific reason that we use absolute tolerance instead of relative tolerance in tests?

[angularsen]

It's a good question that I don't have a good answer to. We have discussed relative errors in the past, also when discussing the new Equals() method that takes a tolerance argument, but the problem is that we don't have any context on what unit the consumer is working with so we can't have provide any good default values that work for all units.

As for tests, if I recall correctly the tolerance specifies the max error we allow for a round-trip conversion from unit A to base unit B and back to unit A, and the error is measured for the value of unit A. This gives an idea of what precision to expect for all our units, since some units are very different (smaller/larger) than the base unit and thus a lot of precision is lost in the conversion.

Does this answer your question or could you perhaps sketch out some pseudo code for how we could better write these tests using relative values?

[gojanpaolo]

@ferittuncer I tried changing the SpeedTests to scientific notation but it didn't fail. I think the test you're talking about is InchesPerHourInOneMeterPerSecond which is 141732.283465 = 1.41732283465E5
Should I create PR for this?

[0xferit]

Well, if you create, we can discuss it. I realized that scientific notation is sometimes less precise, so it sometimes break tests. I don't know why it doesn't fail when you try.

[0xferit]

@angularsen

but the problem is that we don't have any context on what unit the consumer is working with so we can't have provide any good default values that work for all units.

I think that's why we could benefit from relative tolerance instead of fixed absolute tolerance. Because there is no good absolute tolerance value that fits all units. We could define an absolute tolerance for each unit, but this is not elegant.

For example,
Expected: 1E-5
Actual 1.5E-5
Relative difference is huge, actual is 1.5 times larger than expected, but it would satisfy 1E-5 absolute tolerance.

Another example,
Expected: 2E8
Actual: 2.0001E8
They are relatively very close and computation is obviously accurate but it will not satisfy absolute tolerance of 1E-5.

If we agree that it is better to use relative tolerance, we can implement it similar to MathNet.Numerics.Precision.Equality

[angularsen]

I get your point. For Equals(), we could definitely support both - AlmostEqualAbsolute<T>(T a, T b, T maxErrorAbsolute) and AlmostEqualRelative<T>(T a, T b, T maxErrorRelativeToA) where T is for example Length. Then the user can use whichever method makes sense in the situation.

As for relative vs absolute in our test suite, these are the two usages of tolerance per unit today for a small unit nanometers relative to the base unit meters.

// These two are essentially the same:
// meter.Nanometers is approx. 1e9 +/- 1e-5 nm
// We can convert 1 meter to nanometers with +/- 1e-5 nm precision
AssertEx.EqualTolerance(NanometersInOneMeter, meter.Nanometers, NanometersTolerance);
AssertEx.EqualTolerance(NanometersInOneMeter, meter.As(LengthUnit.Nanometer), NanometersTolerance);

// These two are essentially the same:
// Round-trip via internal representation 1 nm => 1e-9m => 1 nm +/- 1e-5 nm
// We can represent 1 nanometer with Length with a precision of +/- 1e-5 nm
AssertEx.EqualTolerance(1, Length.From(1, LengthUnit.Nanometer).Nanometers, NanometersTolerance);
AssertEx.EqualTolerance(1, Length.FromNanometers(meter.Nanometers).Meters, NanometersTolerance);

Now the same for a large unit relative to its base unit:

// These two are essentially the same:
// watt.Petawatts is approx. 1e-12 +/- 1e-5 pw
// Oops.. this is a very large allowed error!
AssertEx.EqualTolerance(PetawattsInOneWatt, watt.Petawatts, PetawattsTolerance);
AssertEx.EqualTolerance(PetawattsInOneWatt, watt.As(PowerUnit.Petawatt), PetawattsTolerance);

// These two are essentially the same:
// Round-trip via internal representation 1 pw => 1e12 w => 1 pw +/- 1e-5 pw
// We can represent 1 petawatt with Power with a precision of +/- 1e-5 pw
// I think this test is what ensure a reasonable precision, since we are effectively testing a scenario
// where the user constructs a Power with petawatts and tries to read back that value later and
// would like it to be reasonably close to his original value.
AssertEx.EqualTolerance(1, Power.From(1, PowerUnit.Petawatt).Petawatts, PetawattsTolerance);
AssertEx.EqualTolerance(1, Power.FromPetawatts(watt.Petawatts).Watts, PetawattsTolerance);

Got to run, will revisit this post later, but as you can see one of the test cases absolutely allows too large errors for large units but is "saved" by the round-trip tests.

Next up is to consider using relative values instead and how it improves these tests - and more importantly - what guarantee we can give to the user about precision.

[angularsen]

Continuing..

So the same example using relative errors would maybe be something like this:

// NanometersRelativeTolerance = 1e-5, which means ABS(A-B) is less than 0.001% of value A (or A*1e-5)
// meter.Nanometers is approx. 1e9 +/- 0.001% nm or 1e9 +/- 1e4 nm (9 orders of magnitude less precise than the absolute error example)
AssertEx.EqualWithRelativeTolerance(NanometersInOneMeter, meter.Nanometers, NanometersRelativeTolerance);

// Round-trip via internal representation 1 nm => 1e-9m => 1 nm with +/- 1e4 nm (0.001%, 9 orders of magnitude less precise than absolute error example)
AssertEx.EqualWithRelativeTolerance(1, Length.FromNanometers(meter.Nanometers).Meters, NanometersRelativeTolerance);

So our choice of A*1e-5 (0.001% of A) relative error was 9 order of magnitude less precise than the absolute error example using 1e-5.

And again for the bigger unit:

// PetawattsRelativeTolerance = 1e-5 (difference of ABS(A-B) must be less than 0.001% of value A)
// watt.Petawatts is approx. 1e-12 +/- 1e-17 PW (0.001% of 1e-12 PW)
AssertEx.EqualWithRelativeTolerance(PetawattsInOneWatt, watt.Petawatts, PetawattsRelativeTolerance);

// Round-trip via internal representation 1 PW => 1e12 w => 1 PW +/- 1e-5 PW (0.001% of 1 PW)
AssertEx.EqualWithRelativeTolerance(1, Power.FromPetawatts(watt.Petawatts).Watts, PetawattsRelativeTolerance);

The relative error behavior works better than the absolute error for large units, where the former correctly uses a smaller allowed error than the value itself and the latter actually allowed larger error than the value itself, which doesn't make much sense as a test case.

I don't know what is a sensible default relative error, but I guess some trial and error will quickly identify the value that passes most of the existing units and we can always override per unit as we do today for the few outliers that need to allow bigger errors.

[angularsen]

TLDR; Yes, I agree max relative errors probably makes more sense for our test suite than max absolute errors - particularly for units that are significantly larger than the base unit (ex: Petawatts vs Watts), where the base-unit-to-unit conversion tests allowed larger errors than the value itself.

Do you want to do a PR on this @ferittuncer ? Should be straight forward and you just need to figure out what default value works best to pass the majority of our test cases, and add a few overrides where necessary. Then update the README section on precision with the new definition of precision to expect.