New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NUnit 3.2.1 with real-time progress reporting #820

Closed
FeodorFitsner opened this Issue May 26, 2016 · 23 comments

Comments

Projects
None yet
2 participants
@FeodorFitsner
Member

FeodorFitsner commented May 26, 2016

No description provided.

@FeodorFitsner FeodorFitsner added this to the may-29 milestone May 26, 2016

@FeodorFitsner FeodorFitsner modified the milestones: may-29, jun-5 Jun 2, 2016

@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@MikhailTymchukDX

MikhailTymchukDX Jun 7, 2016

Hi,

I see my tests still failing when using NUnit Console Runner 3.2.1 with the following message:

Errors and Failures

1) Error : 
Exception has been thrown by the target of an invocation.

Server stack trace: 
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(String assemblyString, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo)
   at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Run(ITestEventListener listener, String filter)
   at NUnit.Engine.Runners.DirectTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Inconclusive: 0, Skipped: 0
  Start time: 2016-06-07 12:00:36Z
    End time: 2016-06-07 12:09:16Z
    Duration: 520.283 seconds

Results (nunit3) saved as TestResult.xml
Command exited with code -100

Please see this build for example. And good build with NUnit 3.2.0.

MikhailTymchukDX commented Jun 7, 2016

Hi,

I see my tests still failing when using NUnit Console Runner 3.2.1 with the following message:

Errors and Failures

1) Error : 
Exception has been thrown by the target of an invocation.

Server stack trace: 
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(String assemblyString, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo)
   at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Run(ITestEventListener listener, String filter)
   at NUnit.Engine.Runners.DirectTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Inconclusive: 0, Skipped: 0
  Start time: 2016-06-07 12:00:36Z
    End time: 2016-06-07 12:09:16Z
    Duration: 520.283 seconds

Results (nunit3) saved as TestResult.xml
Command exited with code -100

Please see this build for example. And good build with NUnit 3.2.0.

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jun 7, 2016

Member

Looks like now all tests are passing:https://ci.appveyor.com/project/FeodorFitsner/ajaxcontroltoolkit - what can be wrong?

Member

FeodorFitsner commented Jun 7, 2016

Looks like now all tests are passing:https://ci.appveyor.com/project/FeodorFitsner/ajaxcontroltoolkit - what can be wrong?

@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@MikhailTymchukDX

MikhailTymchukDX Jun 8, 2016

It seems build in your example fails under 2 minutes, while in my case it lasts more than 4 minutes.
I think this is the error I faced with: nunit/nunit#1509

So, until new version of NUnit Console Runner is released, how can I fall back to NUnit Console Runner 3.2.0 in AppVeyor build agent?

MikhailTymchukDX commented Jun 8, 2016

It seems build in your example fails under 2 minutes, while in my case it lasts more than 4 minutes.
I think this is the error I faced with: nunit/nunit#1509

So, until new version of NUnit Console Runner is released, how can I fall back to NUnit Console Runner 3.2.0 in AppVeyor build agent?

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jun 8, 2016

Member

Add this to appveyor.yml:

install:
- ps: iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/nunit-3-2-0.ps1'))

Also, you can't use tests auto-discovery anymore, so replace it with a script:

test_script:
- nunit3-console.exe "%appveyor_build_folder%\bin\AjaxControlToolkit.Tests.dll" --result=results.xml;format=AppVeyor
Member

FeodorFitsner commented Jun 8, 2016

Add this to appveyor.yml:

install:
- ps: iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/nunit-3-2-0.ps1'))

Also, you can't use tests auto-discovery anymore, so replace it with a script:

test_script:
- nunit3-console.exe "%appveyor_build_folder%\bin\AjaxControlToolkit.Tests.dll" --result=results.xml;format=AppVeyor
@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@MikhailTymchukDX

MikhailTymchukDX Jun 9, 2016

Thank you for the code, it works just fine!

I do not use test auto-discovery, my script is as follows:

test:
  assemblies: bin\$(configuration)\AjaxControlToolkit.Tests.dll

I have no problems with it now. Should I prefer your way over existing one?

MikhailTymchukDX commented Jun 9, 2016

Thank you for the code, it works just fine!

I do not use test auto-discovery, my script is as follows:

test:
  assemblies: bin\$(configuration)\AjaxControlToolkit.Tests.dll

I have no problems with it now. Should I prefer your way over existing one?

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jun 9, 2016

Member

I still can't run the project without errors: https://ci.appveyor.com/project/FeodorFitsner/ajaxcontroltoolkit

I've forked the latest and I'm basically changing the image to run on OSS environment: https://github.com/MikhailTymchukDX/AjaxControlToolkit/compare/add-artifacts-4...FeodorFitsner:add-artifacts-4?expand=1

Do you have some hidden environment variables set to make it run?

Member

FeodorFitsner commented Jun 9, 2016

I still can't run the project without errors: https://ci.appveyor.com/project/FeodorFitsner/ajaxcontroltoolkit

I've forked the latest and I'm basically changing the image to run on OSS environment: https://github.com/MikhailTymchukDX/AjaxControlToolkit/compare/add-artifacts-4...FeodorFitsner:add-artifacts-4?expand=1

Do you have some hidden environment variables set to make it run?

@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@MikhailTymchukDX

MikhailTymchukDX Jun 10, 2016

Yes, I do :)
Environment variables are shared between appveyor.yml and UI, so I moved AjaxControlToolkitTestSiteUrl variable to the UI. This variable has http://localhost/TestRunner.aspx value.
Don't you think that appveyor.yml have to read environment variables defined within this file only?

MikhailTymchukDX commented Jun 10, 2016

Yes, I do :)
Environment variables are shared between appveyor.yml and UI, so I moved AjaxControlToolkitTestSiteUrl variable to the UI. This variable has http://localhost/TestRunner.aspx value.
Don't you think that appveyor.yml have to read environment variables defined within this file only?

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jun 10, 2016

Member

Looks like it's working with nunit 3.2.1 and AppVeyor logger (at least I can't get an error from your original post), however there is one test failing for Chrome: https://ci.appveyor.com/project/FeodorFitsner/ajaxcontroltoolkit/build/tests. What could be a reason for that? I'm running it on a different environment and 4 cores.

Member

FeodorFitsner commented Jun 10, 2016

Looks like it's working with nunit 3.2.1 and AppVeyor logger (at least I can't get an error from your original post), however there is one test failing for Chrome: https://ci.appveyor.com/project/FeodorFitsner/ajaxcontroltoolkit/build/tests. What could be a reason for that? I'm running it on a different environment and 4 cores.

@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@MikhailTymchukDX

MikhailTymchukDX Jun 14, 2016

Hmm, I see failing test, but it never failed earlier. Can you please run the build again?

MikhailTymchukDX commented Jun 14, 2016

Hmm, I see failing test, but it never failed earlier. Can you please run the build again?

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jun 14, 2016

Member

Still failing: https://ci.appveyor.com/project/FeodorFitsner/ajaxcontroltoolkit/build/7
If you'd like to help I can move your account to that environment?

Member

FeodorFitsner commented Jun 14, 2016

Still failing: https://ci.appveyor.com/project/FeodorFitsner/ajaxcontroltoolkit/build/7
If you'd like to help I can move your account to that environment?

@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@MikhailTymchukDX

MikhailTymchukDX Jun 15, 2016

Yes, please.
This will help me fixing this test, at least.

MikhailTymchukDX commented Jun 15, 2016

Yes, please.
This will help me fixing this test, at least.

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jun 15, 2016

Member

It's been moved.

Member

FeodorFitsner commented Jun 15, 2016

It's been moved.

@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@MikhailTymchukDX

MikhailTymchukDX Jun 17, 2016

Do you have any time/timezone/regional settings difference between these environments?

MikhailTymchukDX commented Jun 17, 2016

Do you have any time/timezone/regional settings difference between these environments?

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jun 17, 2016

Member

Yes, looks like they are different.

But you shouldn't fail because of that - properly "globalized" app should work in any time zone - that's the whole purpose of integration testing 😉

Use tzutil utility to set required time zone if you need to.

Member

FeodorFitsner commented Jun 17, 2016

Yes, looks like they are different.

But you shouldn't fail because of that - properly "globalized" app should work in any time zone - that's the whole purpose of integration testing 😉

Use tzutil utility to set required time zone if you need to.

@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@MikhailTymchukDX

MikhailTymchukDX Jun 20, 2016

Well, test condition was not perfect in this case, but I tweaked it and build seems to pass: https://ci.appveyor.com/project/MikhailTymchukDX/ajaxcontroltoolkit/build/job/6kgtd31tg4win5tc

So, can you continue to investigate what's wrong with NUnit 3.2.1?

MikhailTymchukDX commented Jun 20, 2016

Well, test condition was not perfect in this case, but I tweaked it and build seems to pass: https://ci.appveyor.com/project/MikhailTymchukDX/ajaxcontroltoolkit/build/job/6kgtd31tg4win5tc

So, can you continue to investigate what's wrong with NUnit 3.2.1?

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jun 20, 2016

Member

It's already working with nuget runner 3.2.1: https://ci.appveyor.com/project/FeodorFitsner/ajaxcontroltoolkit/build/9

But I'm wondering if this is because all tests pass? Maybe it was crashing because of that failing test rather than correctly logging failing test? Can you make that test failing again and run under 3.2.1 to check that?

Member

FeodorFitsner commented Jun 20, 2016

It's already working with nuget runner 3.2.1: https://ci.appveyor.com/project/FeodorFitsner/ajaxcontroltoolkit/build/9

But I'm wondering if this is because all tests pass? Maybe it was crashing because of that failing test rather than correctly logging failing test? Can you make that test failing again and run under 3.2.1 to check that?

@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@MikhailTymchukDX

MikhailTymchukDX Jun 21, 2016

Yes, I added a test that does nothing during 5 minutes, and it fails as expected: https://ci.appveyor.com/project/MikhailTymchukDX/ajaxcontroltoolkit/build/job/k1cjb408mdomhs5a

MikhailTymchukDX commented Jun 21, 2016

Yes, I added a test that does nothing during 5 minutes, and it fails as expected: https://ci.appveyor.com/project/MikhailTymchukDX/ajaxcontroltoolkit/build/job/k1cjb408mdomhs5a

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jun 22, 2016

Member

I did some changes to NUnit 3 logger for AppVeyor. Can you add these lines installing updated logger to your appveyor.yml and let me know how it worked: https://github.com/FeodorFitsner/AjaxControlToolkit/blob/add-artifacts/appveyor.yml#L11-L14 - this is for 3.2.1 (so, don't include the script replacing default nunit 3.2.1 runner).

Member

FeodorFitsner commented Jun 22, 2016

I did some changes to NUnit 3 logger for AppVeyor. Can you add these lines installing updated logger to your appveyor.yml and let me know how it worked: https://github.com/FeodorFitsner/AjaxControlToolkit/blob/add-artifacts/appveyor.yml#L11-L14 - this is for 3.2.1 (so, don't include the script replacing default nunit 3.2.1 runner).

@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jun 28, 2016

Member

OK, looks like there was a bug in NUnit 3.2.1 as I was able to reproduce the issue locally, without AppVeyor, with a test doing nothing but waiting for 5 minutes and a simple NUnit event listener.

However, this error is not reproducible in NUnit 3.4.0 anymore. I didn't make NUnit 3.4.0 into today's update, but you can add this line to your appveyor.yml to install NUnit 3.4.0 with the updated AppVeyor logger:

install:
- ps: iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/install-nuget-3-4-0.ps1'))
Member

FeodorFitsner commented Jun 28, 2016

OK, looks like there was a bug in NUnit 3.2.1 as I was able to reproduce the issue locally, without AppVeyor, with a test doing nothing but waiting for 5 minutes and a simple NUnit event listener.

However, this error is not reproducible in NUnit 3.4.0 anymore. I didn't make NUnit 3.4.0 into today's update, but you can add this line to your appveyor.yml to install NUnit 3.4.0 with the updated AppVeyor logger:

install:
- ps: iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/install-nuget-3-4-0.ps1'))
@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@MikhailTymchukDX

MikhailTymchukDX Jun 28, 2016

Thank you, @FeodorFitsner !
At the moment, I don't see any advantages of moving to NUnit 3.4.0 explicitly, because this will change the digit in the script only :)
I'll discuss it with my lead and we'll decide what to do in this situation.

Please don't forget to default my environment.

BTW, do you have any RC or beta VM images, so we can check its stability before it reaches production?

MikhailTymchukDX commented Jun 28, 2016

Thank you, @FeodorFitsner !
At the moment, I don't see any advantages of moving to NUnit 3.4.0 explicitly, because this will change the digit in the script only :)
I'll discuss it with my lead and we'll decide what to do in this situation.

Please don't forget to default my environment.

BTW, do you have any RC or beta VM images, so we can check its stability before it reaches production?

@FeodorFitsner

This comment has been minimized.

Show comment
Hide comment
@FeodorFitsner

FeodorFitsner Jun 28, 2016

Member

You don't have to move your tests to 3.4.0 - you are just updating build runner which should work with tests using 3.2.0

Member

FeodorFitsner commented Jun 28, 2016

You don't have to move your tests to 3.4.0 - you are just updating build runner which should work with tests using 3.2.0

@MikhailTymchukDX

This comment has been minimized.

Show comment
Hide comment
@MikhailTymchukDX

MikhailTymchukDX Jun 29, 2016

Actually, I meant NUnit Runner, not NUnit, but upgrading NUnit is not in our priority either.

MikhailTymchukDX commented Jun 29, 2016

Actually, I meant NUnit Runner, not NUnit, but upgrading NUnit is not in our priority either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment