Fix high precision decimal calculations in v3.13 (#3898)

[stevenaw]

Fixes #3898
Fixes high-precision decimal calculations by internally using decimal when appropriate.

I tried a few approaches here, and the one I found worked best while maintaining full backwards compatibility is to compare expected and actual as decimals when at least one of the values is decimal and BOTH are within the valid values for decimals (between decimal.MinValue and decimal.MaxValue).

I also tried to treat all floating point operations as decimal internally when expected and actual are within valid range, however this caused some a change in failure messages when comparing two double or float values (or a double to an int, etc) with a tolerance since the number of decimal places in the "+/-" portion of the error message changed. This approach broke some existing tests in nunit, and likely would've broken others as well.

NOTE: After merging, this should also be ported to the main development branch too

[realfaisalk]

EDIT - It could be that my tolerance level is too high so the code tries to "compensate" by adding more decimal places?

Hi,
Will this change fix the issue where doubles are being compared and the assertion seems to "change" a value by 19 or 20 decimal places?

using System;
using NUnit.Framework;

public class Program
{
	public static void Main()
	{ 
		var myCheck = EqualExample.EqualStringIgnoreCase();
	}
	
	public static class EqualExample
	{
		[Test]
		public static bool EqualStringIgnoreCase()
		{
			bool myCheck = false;
			double expected = -0.09404701272000002;
			double actual = -0.09;

			Console.WriteLine("BeforeExpected = " + expected + " BeforeActual =  " + actual);

			Assert.That(actual, Is.EqualTo(expected).Within(0.001));
			//Assert.That(actual, Is.EqualTo(expected));
			
			myCheck = true;
			Console.WriteLine("AfterExpected = " + expected + " AfterActual =  " + actual);
			
			return myCheck;
		}
	}
}

Result:

_BeforeExpected = -0.09404701272000002 BeforeActual = -0.09
Unhandled exception. NUnit.Framework.AssertionException: Expected: -0.094047012720000017d +/- 0.001d
But was: -0.089999999999999997d
Off by: -0.0040470127200000205d

at NUnit.Framework.Assert.ReportFailure(String message)
at NUnit.Framework.Assert.ReportFailure(ConstraintResult result, String message, Object[] args)
at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression, String message, Object[] args)
at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression)
at Program.EqualExample.EqualStringIgnoreCase()
at Program.Main()_

[stevenaw]

Hi @realfaisalk
Thanks for asking. I see in your edit that you suspect the tolerance in your example is the reason for your result, and I think you're right. Those numbers differ by 0.004 which is greater than your tolerance of 0.001. You may need a higher tolerance, such as 0.01, for your example test to pass.

[realfaisalk]

@stevenaw - Yup, exactly right. The tolerance was asked for by the customer but changed to 0.01 due the the 2dp numbers.
The "But was: -0.89999...." message threw me off. Don't know if it should be "But was: -0.09000000...." or "Your actual number does not have enough dps for the tolerance you set" or similar?

[stevenaw]

Oh that's a good point @realfaisalk thanks for pointing that out! That could probably be made clearer. Would you be willing to file a GitHub issue about it?