Running under mono #3257

oznetmaster · 2019-05-22T11:30:46Z

trying to run the .NET 4.5 build under mono, executing nunitlite.tests.exe results in:

mono nunitlite.tests.exe
NUnitLite 3.13.0 (.NET Framework 4.5 Debug)
Copyright (c) 2019 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Unix 4.4.0.148
  CLR Version: 4.0.30319.17020

Test Files
    /media/sf_D_DRIVE/Projects/github/nunit/bin/debug/net45/nunitlite.tests.exe

Test Discovery
  Start time: 2019-05-22 11:19:56Z
    End time: 2019-05-22 11:19:56Z
    Duration: 0.023 seconds

System.TypeLoadException: Could not load type 'NUnit.Framework.Internal.TestSuiteResult' from assembly 'nunit.framework, Version=3.13.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb'.
  at NUnit.Framework.Internal.Execution.WorkItemBuilder.CreateWorkItem (ITest test, ITestFilter filter, Boolean recursive) <0x41997a00 + 0x00143> in <filename unknown>:0
  at NUnit.Framework.Api.NUnitTestAssemblyRunner.RunAsync (ITestListener listener, ITestFilter filter) <0x41994900 + 0x0010f> in <filename unknown>:0
  at NUnit.Framework.Api.NUnitTestAssemblyRunner.Run (ITestListener listener, ITestFilter filter) <0x41994880 + 0x0001b> in <filename unknown>:0
  at NUnitLite.TextRunner.RunTests (NUnit.Framework.Internal.TestFilter filter, IDictionary`2 runSettings) <0x41994610 + 0x00059> in <filename unknown>:0
  at NUnitLite.TextRunner.Execute () <0x41984560 + 0x0071f> in <filename unknown>:0

I have no idea what to do to debug this. This test exe runs correctly under .NET 4.5 on windows.

The text was updated successfully, but these errors were encountered:

oznetmaster · 2019-05-22T11:47:32Z

Never mind.
Mono runtime was apparently still at 4.0 on machine. Updated to latest, and all is ok.

oznetmaster · 2019-05-22T11:50:54Z

Well, I spoke to soon. Tried running full framework tests, and got:

 mono nunitlite-runner.exe nunit.framework.tests.dll
NUnitLite 3.13.0 (.NET Framework 4.5 Debug)
Copyright (c) 2019 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Unix 4.4.0.148
  CLR Version: 4.0.30319.42000

Test Files
    nunit.framework.tests.dll

Test Discovery
  Start time: 2019-05-22 11:47:28Z
    End time: 2019-05-22 11:47:28Z
    Duration: 0.032 seconds

Errors, Failures and Warnings

1) Invalid : /media/sf_D_DRIVE/Projects/github/nunit/bin/debug/net45/nunit.framework.tests.dll
Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Could not load file or assembly 'System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
Could not load file or assembly 'System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
  ----> Could not load file or assembly 'System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
  ----> Could not load file or assembly 'System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Run Settings
    Number of Test Workers: 4
    Work Directory: /media/sf_D_DRIVE/Projects/github/nunit/bin/debug/net45
    Internal Trace: Off

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-05-22 11:47:28Z
    End time: 2019-05-22 11:47:28Z
    Duration: 0.009 seconds

Results (nunit3) saved as /media/sf_D_DRIVE/Projects/github/nunit/bin/debug/net45/TestResult.xml

Since I know that the full test suite is run under mono for integration testing, I must be doing something wrong.

BTW, this is running under Ubuntu 16.0.4.

And:

mono -V
Mono JIT compiler version 5.20.1.19 (tarball Thu Apr 11 09:09:21 UTC 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        Interpreter:   yes
        LLVM:          yes(600)
        Suspend:       hybrid
        GC:            sgen (concurrent by default)

jnm2 · 2019-05-22T12:03:42Z

CI is using Mono 5.10 on Travis and Azure Pipelines is up to 5.11 I think.

I know that if we upgrade to Mono 5.13, a whole bunch of tests start failing with double precision issues and stack trace mismatches. (I'll have to dig up the issue I posted them in.) I saw this in CI and also on Ubuntu 18.0.4 with Mono 5.20.

Mono quirks have eaten up so much of my time that I'm being careful to remain good-humored about it. :')

Missing System.Web though makes me think you might have installed mono-devel instead of mono-complete, on the Mono install webpage's advice that it solves most missing assembly issues.

jnm2 · 2019-05-22T12:08:30Z

Here's the 24 tests I'd expect to see fail unless you roll back to Mono 5.11:

#3230

I should reopen that issue and get to work, I guess.

oznetmaster · 2019-05-22T14:40:44Z

These are the ones that failed for me under 5.20 (26 of them):

1) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultiple_InvalidAssertThrowsException("AssertPassInBlock","Assert.Pass")
  StackTrace
  Expected: not null and String containing "AssertPassInBlock"
  But was:  "  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0 "

2) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultiple_InvalidAssertThrowsException("AssertIgnoreInBlock","Assert.Ignore")
  StackTrace
  Expected: not null and String containing "AssertIgnoreInBlock"
  But was:  "  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0 "

3) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultiple_InvalidAssertThrowsException("AssertInconclusiveInBlock","Assert.Inconclusive")
  StackTrace
  Expected: not null and String containing "AssertInconclusiveInBlock"
  But was:  "  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0 "

4) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultiple_InvalidAssertThrowsException("AssumptionInBlock","Assume.That")
  StackTrace
  Expected: not null and String containing "AssumptionInBlock"
  But was:  "  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0 "

5) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleErrorTests("ExceptionThrown",0)
  StackTrace
  Expected: not null and String containing "ExceptionThrown"
  But was:  "  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0 "

6) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleErrorTests("ExceptionThrownAfterTwoFailures",2,"Failure 1","Failure 2","Simulated Error")
Multiple failures or warnings in test:
  1)   Failure 1
  Expected: 5
  But was:  4

  2)   Failure 2
  Expected: True
  But was:  False

  3) System.Exception : Simulated Error
  4)   StackTrace
  Expected: not null and String containing "ExceptionThrownAfterTwoFailures"
  But was:  "  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0 "

7) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleErrorTests("ExceptionThrownAfterWarning",0,"WARNING","Simulated Error")
Multiple failures or warnings in test:
  1) WARNING
  2) System.Exception : Simulated Error
  3)   StackTrace
  Expected: not null and String containing "ExceptionThrownAfterWarning"
  But was:  "  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0 "

8) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("TwoAsserts_FirstAssertFails",2,"RealPart")
Multiple failures or warnings in test:
  1)   RealPart
  Expected: 5.0d
  But was:  5.2000000000000002d

  2)   StackTrace
  Expected: not null and String containing "TwoAsserts_FirstAssertFails"
  But was:  <string.Empty>

9) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("TwoAsserts_SecondAssertFails",2,"ImaginaryPart")
Multiple failures or warnings in test:
  1)   ImaginaryPart
  Expected: 4.2000000000000002d
  But was:  3.8999999999999999d

  2)   StackTrace
  Expected: not null and String containing "TwoAsserts_SecondAssertFails"
  But was:  <string.Empty>

10) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("TwoAsserts_BothAssertsFail",2,"RealPart","ImaginaryPart")
Multiple failures or warnings in test:
  1)   RealPart
  Expected: 5.0d
  But was:  5.2000000000000002d

  2)   ImaginaryPart
  Expected: 4.2000000000000002d
  But was:  3.8999999999999999d

  3)   StackTrace
  Expected: not null and String containing "TwoAsserts_BothAssertsFail"
  But was:  <string.Empty>

11) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("NestedBlock_FirstAssertFails",3,"Expected: 5")
Multiple failures or warnings in test:
  1)   Expected: 5
  But was:  4

  2)   StackTrace
  Expected: not null and String containing "NestedBlock_FirstAssertFails"
  But was:  <string.Empty>

12) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("NestedBlock_TwoAssertsFail",3,"Expected: 5","ImaginaryPart")
Multiple failures or warnings in test:
  1)   Expected: 5
  But was:  4

  2)   ImaginaryPart
  Expected: 4.2000000000000002d
  But was:  3.8999999999999999d

  3)   StackTrace
  Expected: not null and String containing "NestedBlock_TwoAssertsFail"
  But was:  <string.Empty>

13) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("TwoNestedBlocks_FirstAssertFails",3,"Expected: 5")
Multiple failures or warnings in test:
  1)   Expected: 5
  But was:  4

  2)   StackTrace
  Expected: not null and String containing "TwoNestedBlocks_FirstAssertFails"
  But was:  <string.Empty>

14) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("TwoNestedBlocks_TwoAssertsFail",3,"Expected: 5","ImaginaryPart")
Multiple failures or warnings in test:
  1)   Expected: 5
  But was:  4

  2)   ImaginaryPart
  Expected: 4.2000000000000002d
  But was:  3.8999999999999999d

  3)   StackTrace
  Expected: not null and String containing "TwoNestedBlocks_TwoAssertsFail"
  But was:  <string.Empty>

15) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("MethodCallsFail",0,"Message from Assert.Fail")
Multiple failures or warnings in test:
  1) Message from Assert.Fail
  2)   StackTrace
  Expected: not null and String containing "MethodCallsFail"
  But was:  <string.Empty>

16) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("MethodCallsFailAfterTwoAssertsFail",2,"Expected: 5","ImaginaryPart","Message from Assert.Fail")
Multiple failures or warnings in test:
  1)   RealPart
  Expected: 5.0d
  But was:  5.2000000000000002d

  2)   ImaginaryPart
  Expected: 4.2000000000000002d
  But was:  3.8999999999999999d

  3) Message from Assert.Fail
  4)   StackTrace
  Expected: not null and String containing "MethodCallsFailAfterTwoAssertsFail"
  But was:  <string.Empty>

17) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("TwoAssertsFailAfterWarning",2,"WARNING","Expected: 5","ImaginaryPart")
Multiple failures or warnings in test:
  1) WARNING
  2)   RealPart
  Expected: 5.0d
  But was:  5.2000000000000002d

  3)   ImaginaryPart
  Expected: 4.2000000000000002d
  But was:  3.8999999999999999d

  4)   StackTrace
  Expected: not null and String containing "TwoAssertsFailAfterWarning"
  But was:  <string.Empty>

18) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("WarningAfterTwoAssertsFail",2,"Expected: 5","ImaginaryPart","WARNING")
Multiple failures or warnings in test:
  1)   RealPart
  Expected: 5.0d
  But was:  5.2000000000000002d

  2)   ImaginaryPart
  Expected: 4.2000000000000002d
  But was:  3.8999999999999999d

  3) WARNING
  4)   StackTrace
  Expected: not null and String containing "WarningAfterTwoAssertsFail"
  But was:  <string.Empty>

19) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("TwoAsserts_BothAssertsFail_Async",2,"RealPart","ImaginaryPart")
Multiple failures or warnings in test:
  1)   RealPart
  Expected: 5.0d
  But was:  5.2000000000000002d

  2)   ImaginaryPart
  Expected: 4.2000000000000002d
  But was:  3.8999999999999999d

  3)   StackTrace
  Expected: not null and String containing "TwoAsserts_BothAssertsFail_Async"
  But was:  <string.Empty>

20) Failed : NUnit.Framework.Assertions.AssertMultipleTests.AssertMultipleFails("TwoNestedBlocks_TwoAssertsFail_Async",3,"Expected: 5","ImaginaryPart")
Multiple failures or warnings in test:
  1)   Expected: 5
  But was:  4

  2)   ImaginaryPart
  Expected: 4.2000000000000002d
  But was:  3.8999999999999999d

  3)   StackTrace
  Expected: not null and String containing "TwoNestedBlocks_TwoAssertsFail_Async"
  But was:  <string.Empty>

21) Failed : NUnit.Framework.TestContextTests.TestContextStoresFailureInfoForTearDown
  Expected: String containing "NUnit.TestData.TestContextData.TestTestContextInTearDown.FailingTest"
  But was:  <string.Empty>

22) Failed : NUnit.Framework.Internal.ReflectTests.DynamicInvokeWithTransparentExceptionsPreservesStackTrace
  Expected: property StackTrace String containing "MethodThrowingTargetInvocationException"
  But was:  "  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0
--- End of stack trace from previous location where exception was thrown ---

  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0
--- End of stack trace from previous location where exception was thrown ---

  at NUnit.Framework.Internal.ExceptionHelper.Rethrow (System.Exception exception) [0x00007] in <0a29a5784ee940e3ad5246f4d41be0cf>:0
  at NUnit.Framework.Internal.Reflect.DynamicInvokeWithTransparentExceptions (System.Delegate delegate) [0x00019] in <0a29a5784ee940e3ad5246f4d41be0cf>:0
  at NUnit.Framework.Internal.ExceptionHelper.RecordException (System.Delegate parameterlessDelegate, System.String parameterName) [0x00078] in <0a29a5784ee940e3ad5246f4d41be0cf>:0 "
  at NUnit.Framework.Internal.ReflectTests.DynamicInvokeWithTransparentExceptionsPreservesStackTrace () [0x00000] in <7b2e67d9e0c74508bd73a23d431def03>:0

23) Failed : NUnit.Framework.Internal.ReflectTests.InvokeWithTransparentExceptionsPreservesStackTrace
  Expected: property StackTrace String containing "MethodThrowingTargetInvocationException"
  But was:  "  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0
--- End of stack trace from previous location where exception was thrown ---

  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0
--- End of stack trace from previous location where exception was thrown ---

  at NUnit.Framework.Internal.ExceptionHelper.Rethrow (System.Exception exception) [0x00007] in <0a29a5784ee940e3ad5246f4d41be0cf>:0
  at NUnit.Framework.Internal.Reflect.DynamicInvokeWithTransparentExceptions (System.Delegate delegate) [0x00019] in <0a29a5784ee940e3ad5246f4d41be0cf>:0
  at NUnit.Framework.Internal.ExceptionHelper.RecordException (System.Delegate parameterlessDelegate, System.String parameterName) [0x00078] in <0a29a5784ee940e3ad5246f4d41be0cf>:0 "
  at NUnit.Framework.Internal.ReflectTests.InvokeWithTransparentExceptionsPreservesStackTrace () [0x00000] in <7b2e67d9e0c74508bd73a23d431def03>:0

24) Failed : NUnit.Framework.Internal.SetUpTearDownTests.HandleExceptionInBothSetUpAndTearDown
  Expected: String containing "NUnit.TestData.SetUpData.SetupAndTearDownExceptionFixture"
  But was:  "  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0
--TearDown
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0 "

25) Failed : NUnit.Framework.Internal.SetUpTearDownTests.HandleExceptionInSetUp
  Expected: String containing "NUnit.TestData.SetUpData.SetupAndTearDownExceptionFixture"
  But was:  "  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0 "

26) Failed : NUnit.Framework.Internal.SetUpTearDownTests.HandleExceptionInTearDown
  Expected: String containing "NUnit.TestData.SetUpData.SetupAndTearDownExceptionFixture"
  But was:  "--TearDown
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0 "

oznetmaster · 2019-05-22T14:42:14Z

I had installed mono-runtime. Installing mono-complete did solve the missing dll problem.

I do not really need mono-complete, however :(

oznetmaster · 2019-05-22T15:16:27Z

@jnm2 There is no mono release 5.11 :(

Mono 5.x

    Mono 6.0.0 (not released yet)

    Mono 5.20.0 (11 Apr 2019)

    Mono 5.18.0 (21 Dec 2018)

    Mono 5.16.0 (08 Oct 2018)

    Mono 5.14.0 (07 Aug 2018)

    Mono 5.12.0 (08 May 2018)

    Mono 5.8.1 (26 Mar 2018)

    Mono 5.10.0 (26 Feb 2018)

    Mono 5.8.0 (01 Feb 2018)

    Mono 5.4.1 (31 Oct 2017)

    Mono 5.4.0 (05 Oct 2017)

    Mono 5.2.0 (14 Aug 2017)

    Mono 5.0.1 (23 May 2017)

    Mono 5.0.0 (10 May 2017)

oznetmaster · 2019-05-22T21:53:34Z

It really surprises me that we are running the integration tests under a mono release which is over a year old!

There have been four releases of mono since then.

I would have thought that the problems with the newer versions would have been identified and fixed by now :(

jnm2 · 2019-05-23T00:22:40Z

@oznetmaster I have no idea what I was remembering. The last version of Mono that works for us is 5.18, and 5.20 introduces all those issues you mention.

It really surprises me that we are running the integration tests under a mono release which is over a year old!

This is what happens when we pin the Mono version in Travis due to needing to force a newer version.
Our Azure Pipelines build is constantly kept closer up to date. 18 days ago it was at 5.18.1, and now it's at 5.20.1: https://github.com/Microsoft/azure-pipelines-image-generation/blob/master/images/linux/Ubuntu1604-README.md

So the failures should start rolling in at any time from our AzDO build.

I would have thought that the problems with the newer versions would have been identified and fixed by now :(

Based on the dates you posted, the earliest we could have detected this incoming situation was ~5 weeks ago. Some of these failures are due to overspecified tests (stack traces), and others look like Mono must be plain broken because its behavior is not matching .NET Core or .NET Framework:

  2)   ImaginaryPart
  Expected: 4.2000000000000002d
  But was:  3.8999999999999999d

Mono had me on a fine chase recently. ThreadAbortExceptions disappeared, waited until the next unrelated exception is thrown, and came back to life with all the original context missing. The cause of this was a fundamental runtime spec violation. We saw the side effects of this ten years ago in a number of tests but couldn't figure out what was behind it, so we just disabled those tests on Mono.

I'm flattered by your faith in us that we're able to keep up with whatever Mono throws at us next, but it's also a time sink. Other than CI, it's hard for the team to even be aware of this because it's not practical for us to use Ubuntu when working on NUnit on a regular basis. (This thread abort bug for example afflicts only the non-Windows builds of Mono.)

jnm2 · 2019-05-23T00:27:51Z

I do not really need mono-complete, however :(

Yeah, sounds like mono-devel might contain System.Web.

oznetmaster · 2019-05-23T01:45:00Z

Since I only maintain the CF version of NUnit, which does not involve CI at all, I have not been exposed to all these issues.

Now, I only need a reliable NUnit that works on the current Mono (5.20.1) on Ubunto 16.0.2.

I would be happy to help find the problems, but my environment is Mono on a different computer then I am running VS 2019 on, so debugging is very difficult.

oznetmaster · 2019-05-23T01:46:57Z

I do not control the mono version on that machine either.

jnm2 · 2019-05-23T02:00:46Z

Thanks. It's probably down to you and me to sort these out. I will start working on it absolutely next (I just fixed #3258.) That means probably this weekend.

oznetmaster · 2019-05-23T02:17:44Z

Ok. Let me know what I can do.

oznetmaster · 2019-05-23T02:18:55Z

oznetmaster commented 17 seconds from now

Is this time travel on github now?

CharliePoole · 2019-05-23T04:50:12Z

Reading through this, I noticed that the first comment uses nunitlite, while the second uses nunitlite runner. I wonder if we are all thinking of the same thing as we read this, so I suggest clarifying.

NUnitlite is used from an executable test that uses the AutoRun class.

NUnitlite runner is something I banged together to allow running tests from a dll and to run under a third target, different from the test target and the sut target. We distribute it but do not really explain it to users. Maybe that is a mistake.

Anyway, it seems important to be clear in every example which of the two we are talking about.

oznetmaster · 2019-05-23T04:57:15Z

Yes, I tried both to see what happened. In this environment, I can only use NUnitlite, so I used NUnitlite runner to run the framework tests, since they are only a dll.

oznetmaster · 2019-05-23T05:07:23Z

For some reason, the NUnit build creates the nunitlite.tests.exe, but does not create a nunit.framework.tests.exe, hence the need to use the runner.

CharliePoole · 2019-05-23T10:11:31Z

That's because we wanted to exercise use of NUnitLite when testing NUnitLite.

In that case, if you are only using the runner for the NUnit tests themselves. Its possible to add another build of the runner that targets 4.7. Nothing else would need to be re-targeted.

jnm2 · 2019-05-25T00:46:40Z

For this one:

21) Failed : NUnit.Framework.TestContextTests.TestContextStoresFailureInfoForTearDown
  Expected: String containing "NUnit.TestData.TestContextData.TestTestContextInTearDown.FailingTest"
  But was:  <string.Empty>

I stepped through the test using MonoDevelop with both Mono 5.18 and 5.20. It appears that Mono 5.20 loses the stack trace during the MethodInfo.Invoke call. I can't think of any option but to ignore this test on Mono from now on. We should find or file a bug too and link it.

jnm2 · 2019-05-26T20:15:13Z

@oznetmaster This is out of the way then as soon as you have time to review #3261. :)

jnm2 · 2019-05-29T13:51:23Z

I noticed that our master branch was still failing. I must have missed these two tests when I did #3261 and I submitted the PR while AzDO was still using Mono 5.18, so that's why the build succeeded for #3261.

https://nunit.visualstudio.com/NUnit/_build/results?buildId=1760:

❌ ReflectTests.DynamicInvokeWithTransparentExceptionsPreservesStackTrace
❌ ReflectTests.InvokeWithTransparentExceptionsPreservesStackTrace

The fix is to do the same as #3261 for these two. I may have time late tonight.

jnm2 mentioned this issue May 26, 2019

Workaround for lost MethodInfo.Invoke stack trace beginning in Mono 5.20 #3261

Merged

rprouse mentioned this issue May 29, 2019

Enable and begin using C# 7.0 syntax (requires capable compiler) #3263

Merged

rprouse closed this as completed in #3261 May 29, 2019

jnm2 reopened this May 29, 2019

jnm2 mentioned this issue May 30, 2019

Ignores Mono stack trace inconsistency (missed these two in the first PR) #3271

Merged

jnm2 self-assigned this May 30, 2019

jnm2 added is:build pri:high labels May 30, 2019

jnm2 added this to the 3.13 milestone May 30, 2019

mikkelbu closed this as completed in #3271 May 30, 2019

mikkelbu added the closed:done label May 30, 2019

Running under mono #3257

Running under mono #3257

Comments

oznetmaster commented May 22, 2019

oznetmaster commented May 22, 2019

Uh oh!

oznetmaster commented May 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jnm2 commented May 22, 2019

Uh oh!

jnm2 commented May 22, 2019

Uh oh!

oznetmaster commented May 22, 2019

Uh oh!

oznetmaster commented May 22, 2019

Uh oh!

oznetmaster commented May 22, 2019

Uh oh!

oznetmaster commented May 22, 2019

Uh oh!

jnm2 commented May 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jnm2 commented May 23, 2019

Uh oh!

oznetmaster commented May 23, 2019

Uh oh!

oznetmaster commented May 23, 2019

Uh oh!

jnm2 commented May 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oznetmaster commented May 23, 2019

Uh oh!

oznetmaster commented May 23, 2019

Uh oh!

CharliePoole commented May 23, 2019

Uh oh!

oznetmaster commented May 23, 2019

Uh oh!

oznetmaster commented May 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CharliePoole commented May 23, 2019

Uh oh!

jnm2 commented May 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jnm2 commented May 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jnm2 commented May 29, 2019

Uh oh!

oznetmaster commented May 22, 2019 •

edited

Loading

jnm2 commented May 23, 2019 •

edited

Loading

jnm2 commented May 23, 2019 •

edited

Loading

oznetmaster commented May 23, 2019 •

edited

Loading

jnm2 commented May 25, 2019 •

edited

Loading

jnm2 commented May 26, 2019 •

edited

Loading