Unit Tests failing for .NET 4.0

[vivian-farrell]

For the .NET 4.0 build for the RC1 code, the following tests are failing in Spring.Core.Tests.2010:

Spring.Globalization.Formatters.CurrencyFormatterTests
-FormatUsingCustomSettings
-FormatUsingDefaults
-ParseUsingCustomSettings
-ParseUsingDefaults
Spring.Globalization.Formatters.DateTimeFormatterTests
-FormatUsingDefaults
-ParseUsingDefaults

I have verified on my machine commented these tests out on my machine and the build completes successfully. I haven't attempted to fix them because some of the assertions have funny characters in them and I'm not sure what's required there.

Also I'm not sure if this is machine specific. It's possible there are issues with my environment. However, I have compiled the milestone versions in the past with no issues.

Build log extract is included below.

        test:


        common.run-tests:


        common.run-tests.nunit:

             [echo] Unit Testing Spring.NET, Spring.Core.Tests in C:\devenv\spring-net-spring-net-2.0.0-RC1/build/net/4.0/debug
             [exec] Starting 'C:\devenv\spring-net-spring-net-2.0.0-RC1/packages/NUnit.Runners.2.6.1/tools/nunit-console-x86.exe (Spring.Core.Tests.dll /xml:Spring.Core.Tests.dll-TestResults.xml /framework:net-4.0 /nologo /noshadow)' in 'C:\devenv\spring-net-spring-net-2.0.0-RC1\build\net\4.0\debug'
             [exec] ProcessModel: Default    DomainUsage: Single
             [exec] Execution Runtime: net-4.0
             [exec] ...........................................................................................................................................................................................................................................................................Source      : Spring.Context.Events.ConsoleListenerTests
             [exec] Event fired : 7/03/2014 12:34:02 PM
             [exec] ........................................................................................................................N.........................................N.......post process Spring.Objects.TestObject#91FC31
             [exec] ...Spring.Context.Support.XmlApplicationContextTests+DestroyTester
             [exec] ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F.F..F.F...F..F..................................................................................................................................................................................................................................................................................................................................................................................................init factory
             [exec] .N............................................N........................N............................................................................DisposableClass.Dispose()
             [exec] .N......N..............................................................................................................................N.......................................................................................................N...........................................................................................................................................................................N..N............................N....................................................................................................N....N........................................................................................................................N...................N.N........N................N.N..............................................................................................................................................................................................................................................................................................................................................................................................................................................
             [exec] Tests run: 2666, Errors: 3, Failures: 3, Inconclusive: 0, Time: 20.8386908 seconds
             [exec]   Not run: 20, Invalid: 0, Ignored: 20, Skipped: 0
             [exec] 
             [exec] Errors and Failures:
             [exec] 1) Test Failure : Spring.Globalization.Formatters.CurrencyFormatterTests.FormatUsingCustomSettings
             [exec]      String lengths are both 14. Strings differ at index 10.
             [exec]   Expected: "1'23'4,00 ???."
             [exec]   But was:  "1'23'4,00 ???."
             [exec]   ---------------------^
             [exec] 
             [exec] at Spring.Globalization.Formatters.CurrencyFormatterTests.FormatUsingCustomSettings() in c:\devenv\spring-net-spring-net-2.0.0-RC1\test\Spring\Spring.Core.Tests\Globalization\Formatters\CurrencyFormatterTests.cs:line 153
             [exec] 
             [exec] 2) Test Failure : Spring.Globalization.Formatters.CurrencyFormatterTests.FormatUsingDefaults
             [exec]      String lengths are both 13. Strings differ at index 9.
             [exec]   Expected: "1.234,00 din."
             [exec]   But was:  "1.234,00 Din."
             [exec]   --------------------^
             [exec] 
             [exec] at Spring.Globalization.Formatters.CurrencyFormatterTests.FormatUsingDefaults() in c:\devenv\spring-net-spring-net-2.0.0-RC1\test\Spring\Spring.Core.Tests\Globalization\Formatters\CurrencyFormatterTests.cs:line 69
             [exec] 
             [exec] 3) Test Error : Spring.Globalization.Formatters.CurrencyFormatterTests.ParseUsingCustomSettings
             [exec]    System.FormatException : Input string was not in a correct format.
             [exec]    at System.Number.ParseDouble(String value, NumberStyles options, NumberFormatInfo numfmt)
             [exec]    at System.Double.Parse(String s, NumberStyles style, IFormatProvider provider)
             [exec]    at Spring.Globalization.Formatters.CurrencyFormatter.Parse(String value) in c:\devenv\spring-net-spring-net-2.0.0-RC1\src\Spring\Spring.Core\Globalization\Formatters\CurrencyFormatter.cs:line 205
             [exec]    at Spring.Globalization.Formatters.CurrencyFormatterTests.ParseUsingCustomSettings() in c:\devenv\spring-net-spring-net-2.0.0-RC1\test\Spring\Spring.Core.Tests\Globalization\Formatters\CurrencyFormatterTests.cs:line 188
             [exec] 
             [exec] 4) Test Error : Spring.Globalization.Formatters.CurrencyFormatterTests.ParseUsingDefaults
             [exec]    System.FormatException : Input string was not in a correct format.
             [exec]    at System.Number.ParseDouble(String value, NumberStyles options, NumberFormatInfo numfmt)
             [exec]    at System.Double.Parse(String s, NumberStyles style, IFormatProvider provider)
             [exec]    at Spring.Globalization.Formatters.CurrencyFormatter.Parse(String value) in c:\devenv\spring-net-spring-net-2.0.0-RC1\src\Spring\Spring.Core\Globalization\Formatters\CurrencyFormatter.cs:line 205
             [exec]    at Spring.Globalization.Formatters.CurrencyFormatterTests.ParseUsingDefaults() in c:\devenv\spring-net-spring-net-2.0.0-RC1\test\Spring\Spring.Core.Tests\Globalization\Formatters\CurrencyFormatterTests.cs:line 104
             [exec] 
             [exec] 5) Test Failure : Spring.Globalization.Formatters.DateTimeFormatterTests.FormatUsingDefaults
             [exec]      Expected string length 16 but was 15. Strings differ at index 15.
             [exec]   Expected: "14. avgust 2004."
             [exec]   But was:  "14. avgust 2004"
             [exec]   --------------------------^
             [exec] 
             [exec] at Spring.Globalization.Formatters.DateTimeFormatterTests.FormatUsingDefaults() in c:\devenv\spring-net-spring-net-2.0.0-RC1\test\Spring\Spring.Core.Tests\Globalization\Formatters\DateTimeFormatterTests.cs:line 71
             [exec] 
             [exec] 6) Test Error : Spring.Globalization.Formatters.DateTimeFormatterTests.ParseUsingDefaults
             [exec]    System.FormatException : String was not recognized as a valid DateTime.
             [exec]    at System.DateTimeParse.ParseExact(String s, String format, DateTimeFormatInfo dtfi, DateTimeStyles style)
             [exec]    at System.DateTime.ParseExact(String s, String format, IFormatProvider provider, DateTimeStyles style)
             [exec]    at Spring.Globalization.Formatters.DateTimeFormatter.Parse(String value) in c:\devenv\spring-net-spring-net-2.0.0-RC1\src\Spring\Spring.Core\Globalization\Formatters\DateTimeFormatter.cs:line 116
             [exec]    at Spring.Globalization.Formatters.DateTimeFormatterTests.ParseUsingDefaults() in c:\devenv\spring-net-spring-net-2.0.0-RC1\test\Spring\Spring.Core.Tests\Globalization\Formatters\DateTimeFormatterTests.cs:line 100
             [exec] 
             [exec] 
             [exec] Tests Not Run:
             [exec] 1) Ignored : Spring.Context.Support.ResourceSetMessageSourceTests.GetNullResourceIn2PointOh
             [exec]    See SPRNET-246
             [exec] 2) Ignored : Spring.Context.Support.XmlApplicationContextTests.ContextWithInvalidLazyType
             [exec]    Need to add Spring.TypeLoadException
             [exec] 3) Ignored : Spring.Objects.Factory.DefaultListableObjectFactoryPerfTests.Test
             [exec] 4) Ignored : Spring.Objects.Factory.DefaultListableObjectFactoryTests.CreateObjectWithCtorArgsOnPrototypeOutOfOrderArgs
             [exec]    Ordering must now be strict when providing array of arguments for ctors
             [exec] 5) Ignored : Spring.Objects.Factory.DefaultListableObjectFactoryTests.GetObjectWithCtorArgsOnPrototypeOutOfOrderArgs
             [exec]    Ordering must now be strict when providing array of arguments for ctors
             [exec] 6) Ignored : Spring.Objects.Factory.SPRNET_1338.CanAvoidLockContentionDuringObjectFactoryDisposal
             [exec]    Test fails -- waiting for verification re: if bug exists in Java impl
             [exec] 7) Ignored : Spring.Objects.Factory.Support.AutowireUtilsTests.GetTypeDifferenceWeightWithMismatchedLengths
             [exec]    Investigate details of new type weight algorithm
             [exec] 8) Ignored : Spring.Objects.Factory.Xml.EventWiringTests.EventWiringInstanceSinkToPrototypeSource
             [exec]    SPRNET-21
             [exec] 9) Ignored : Spring.Objects.Factory.Xml.XmlObjectDefinitionReaderTests.AutoRegistersWellknownNamespaceParser
             [exec] 10) Ignored : Spring.Objects.ObjectWrapperTests.GetSetIndexerProperties
             [exec] 11) Ignored : Spring.Objects.ObjectWrapperTests.GetValueOfCustomIndexerProperty
             [exec] 12) Ignored : Spring.Objects.ObjectWrapperTests.SetIndexOutOfRangeFromListProperty
             [exec]    Addition of elements to the list via index that is out of range is not supported anymore.
             [exec] 13) Ignored : Spring.Pool.SimplePoolTests.PassivateBusyObjectsBeforeClose
             [exec]    Cannot figure out why this is failing?
             [exec] 14) Ignored : Spring.Pool.StessSimplePool.ScalingAgainstIncreasingNumberOfThreads
             [exec]    a try for a stress
             [exec] 15) Ignored : Spring.Reflection.Dynamic.DynamicFieldTests.CannotReadPrivateReadOnlyFieldIfNoReflectionPermission
             [exec]    TODO: this works as expected when run using TD.NET & R# (in VS2k8), but fails with nant/NET 2.0 ?!?
             [exec] 16) Ignored : Spring.Reflection.Dynamic.DynamicPropertyTests.BasePropertyTests.TestForRestrictiveGetter
             [exec]    test N/A anymore due to System.Reflection.Emit.DynamicMethod
             [exec] 17) Ignored : Spring.Reflection.Dynamic.DynamicPropertyTests.BasePropertyTests.TestForRestrictiveSetter
             [exec]    test N/A anymore due to System.Reflection.Emit.DynamicMethod
             [exec] 18) Ignored : Spring.Reflection.Dynamic.SafeFieldTests.DynamicFieldTests.CannotReadPrivateReadOnlyFieldIfNoReflectionPermission
             [exec]    TODO: this works as expected when run using TD.NET & R# (in VS2k8), but fails with nant/NET 2.0 ?!?
             [exec] 19) Ignored : Spring.Reflection.Dynamic.SafePropertyTests.BasePropertyTests.TestForRestrictiveGetter
             [exec]    test N/A anymore due to System.Reflection.Emit.DynamicMethod
             [exec] 20) Ignored : Spring.Reflection.Dynamic.SafePropertyTests.BasePropertyTests.TestForRestrictiveSetter
             [exec]    test N/A anymore due to System.Reflection.Emit.DynamicMethod
             [exec] 

        BUILD FAILED

        External Program Failed: C:\devenv\spring-net-spring-net-2.0.0-RC1/packages/NUnit.Runners.2.6.1/tools/nunit-console-x86.exe (return code was 6)

        Total time: 36.3 seconds.

[sbohlen]

Thanks for reporting this. I think that you've (possibly) surfaced that the changes in my commit here d8b8acd may not in fact correctly resolve the (similar/same?) issue I'd been seeing on my own system.

Prior to that commit, when I ran the entire test-suite on ,my system those same tests also failed under .NET 4.0 (and passed under .NET 3.5). From this, I concluded that (perhaps) some changes to .NET 4.0 libs may have introduced the failure I'd been seeing. As you can see from the commit, my response had been to provide a different set of expectations for these same tests under .NET 4.0 than .NET 3.5. This change has fixed the issue on my system, but apparently broken these same tests on your system :( This suggests that it may NOT in fact be .NET 4.0 changes that are in play here but instead potentially something else entirely.

If possible, to help us isolate the actual cause of the pass/fail results being one way on my system and another way on yours, could you please provide us some info (as follows) as well as perform an experiment for us?

Requested Information:

1. Your operating system (e.g., Windows7, Windows8, Windows8.1)
2. The bit-ness of your OS (e.g., 32-bit vs. 64-bit)
3. The version(s) of installed instances of Visual Studio (e.g., 2010, 2012, 2013) on the machine that experiences these failures

The experiment:
Could you please take the latest (RC1) source and revert the commit mentioned above in your own local copy and then re-run the entire test-suite? I have a suspicion that the tests will pass on your system once you revert this commit, and your experiment will confirm that (hopefully). If the tests still FAIL after reverting this commit, then the problem lies elsewhere.

Thanks again for identifying (and reporting!) this issue; hopefully your additional data-points will permit us to isolate the underlying cause of the issue that results in the different test-pass/fails you and I are seeing on our respective systems.

[sbohlen]

Oh, and one more piece of information if possible: can you tell me the locale for which your OS is configured when you run the tests (e.g., EN-US or other)?

Thanks!

[vivian-farrell]

My machine is Win7 64bit with Visual Studio 2010. I was using EN-AU but changed it to EN-US with the same result. I reverted my code to the commit you mentioned and as you expected, the tests passed. Hope this helps.

[sbohlen]

Yes, it very much does help, thanks for taking the time to do this.

I'm on Windows 8.1, x64, and VS 2013. Your results help to confirm my suspicion that this is in fact NOT a change introduced in .NET 4.0 but is instead a change made in Windows itself. I suspect strongly that what's happened here is that the localization support on Windows 8.x is different than that on Windows 7 (for the locale used in those tests) and that this is the source of the behavior you and I are each seeing here.

Its not at all uncommon for such minor tweaks in localization to be rolled out in OS releases, either to correct improper localization elements (e.g., bug-fix) or to permit localizations to continue to evolve as various locales change their 'rules' for the "official" representations of various elements in their respective languages. There are 'standards' that govern most of these things, and the two things we know about standards are a) everybody has their own :) but more importantly, b) they do change and evolve over time.

I'll dig into whether I can confirm that this is indeed the case here, and if so I'll try to work some recognition of the ambient OS into the tests or the #ifdef that's in these tests before we go to GA.

The good news (if there is any to be found here) is that this points to the ASSERTS being (possibly) incorrect based on the platform on which the tests are run, but also points to the actual behavior of the system-under-test actually still behaving 100% correctly, so its likely NOT a bug in the code per se.

Thanks again for your identifying this issue and and for working with us to gather more info to assist in isolating it. I'd like to leave this issue open until I'm able to confirm my hypothesis and to work a fix into the test code.

Cheers,

-Steve B.

[sbohlen]

Tracked under https://jira.spring.io/browse/SPRNET-1570 and fixed in 0311d36. Please do a fresh pull from ORIGIN/MASTER and verify that these changes result in passing tests for you on Windows7. If so, comment here and I'll close this issue and the corresponding JIRA ticket.

[lahma]

@vivf86 have you had the time to test this?

[sbohlen]

Closing issue as machine-specific or other since .NET 4.0 tests pass on several other computers as well as the latest CI builds as well.