Slow performance of nunit-console

[xplicit]

I've noticed, that tests are being run in nunit-console performing much slower than running them in Visual Studio or in simple console application. nunit-console performs 4-5 times slower on some tests.
I've tested this with NUnit 2.6.4, NUnit 3.0.0-alpha4 (from nuget packages) and with NUnit compiled from master branch (with changes from 2.0 target framework to 4.0 target framework of nunit-console and other projects). I tried various settings of --domain and --process parameters, they didn't help.

Here is the project which demonstrates the issue:

https://github.com/xplicit/Bond/tree/nunit

test UnitTest.GenericsTests.GenericInheritance takes 15-17 seconds on my machine if run it from Visual Studio tests or make a simple console application, which calls this method.
When I run it from nunit-console, it takes about 80 seconds.

Also I wrote a small app, which can be used to investigate the issue. When I run test from Visual Studio or console or with nunitlite it takes 22 seconds to run. If I run it using nunit-console it takes 36 seconds. However I can't be sure for 100% that the app shows the same issue as the first one.

https://github.com/xplicit/nunitperf

The test case, I test: https://github.com/xplicit/nunitperf/blob/master/nunitPerf/Class1.cs

This is critical issue, because it does not allow to run unit-tests on build machines with limited build time (like appveyor service).

Also I am looking for a workaround how I can run unit tests on the build machine and get the results on time with current versions of nunit libraries. I tried to use nunitlite but seems that all tests must be located in the *.exe assembly, and nunitlite can't load tests from other assemblies (no matter are they references or don't referenced by nunitlited exe). If it were able to load assemblies I would have create the console app with references to current tests and run them from command line.

[CharliePoole]

It's definitely a strange issue. I'll looked at your project and I see it's
somewhat complex. I'll wait for the small app that demonstrates the issue.
:-)

A few comments...

1. Make sure your timing is measuring the same thing in both cases. For
   example, both should either include or exclude the time it takes to load
   the assemblies.
2. Settle on a single version of NUnit for your tests. The approach to
   loading and running tests in nunit 3.0 and 2.6.4 are completely different.
   NUnit 3.0 is expected to be significantly faster, especially for large
   numbers of relatively small tests. We are likely to fix any issues you find
   in 3.0 but I can't say the same for 2.6.4. :-(

Charlie

On Sat, Jan 31, 2015 at 1:48 AM, Sergey Zhukov notifications@github.com
wrote:

I've noticed, that tests are being run in nunit-console performing much
slower than running them in Visual Studio or in simple console application.
nunit-console performs 4-5 times slower on some tests.

I've tested this with NUnit 2.6.4, NUnit 3.0.0-alpha4 (from nuget
packages) and with NUnit compiled from master branch (with changes from 2.0
target framework to 4.0 target framework of nunit-console and other
projects). I tried various settings of --domain and --process parameters,
they didn't help.

Here is the project which demonstrates the issue:

https://github.com/xplicit/Bond/tree/nunit

test UnitTest.GenericsTests.GenericInheritance takes 15-17 seconds on my
machine if run it from Visual Studio tests or make a simple console
application, which calls this method.
When I run it from nunit-console, it takes about 80 seconds.

Also I wrote a small app, which can be used to investigate the issue. When
I run test from Visual Studio or console or with nunitlite it takes 22
seconds to run. If I run it using nunit-console it takes 36 seconds.
However I can't be sure for 100% that the app shows the same issue as the
first one.

(I'll post the link here, when upload the project on github)

This is critical issue, because it does not allow to run unit-tests on
build machines with limited build time (like appveyor service).

Also I am looking for a workaround how I can run unit tests on the build
machine and get the results on time with current versions of nunit
libraries. I tried to use nunitlite but seems that all tests must be
located in the *.exe assembly, and nunitlite can't load tests from other
assemblies (no matter are they references or don't referenced by nunitlited
exe). If it were able to load assemblies I would have create the console
app with references to current tests and run them from command line.

—
Reply to this email directly or view it on GitHub
#480.

[CharliePoole]

Try modifying your nunit-console.exe.config file so that NUnit runs under .NET 4.0 / 4.5. Then execute using -process:Single.

The alpha-4 release chooses .NET 2.0 by preference, if present, which would cause your tests to be run in a separate process. The coming alpha-5 changes this to prefer 4.0.

In either case, a separate process is used by default, which may impact your timings. The option -process:Single eliminates the use of a separate process and runs your tests in the NUnit process. This is generally best for timing and debugging.

[xplicit]

I tried all settings combinations for 'process', 'domain' and 'x86' options. Also I tried to run with these combinations:
nunit.framework 2.6.4 + nunit-console 2.6.4 (from nuget)
nunit-framework-3.0.0-alpha4 + nunit-console 3.0.0-alpha4 (from nuget)
nunit-framework-3.0.0 + nunit-console 3.0.0 (built from sources and with changing of target platform for all relevant exe and dll from 2.0 to 4.0)

I suspected that this could be the issue of some domain security settings or remoting and tried to change instantiation of LoadAction, RunAction, etc in NUnit driver from reflection to direct creation of the objects, but this didn't help.

Also this issue exists only for .NET platform (I tested on Win 7 64bit), when I run the test on the mono (tested with mono 3.12, Ubuntu 32 bit), I've got the same times.

There is a link to small project in the first post.

[CharliePoole]

Hi Sergey,

Regarding settings, please see my earlier post for the only setting change that has a hope of affecting this! Do not use -x86 unless you are building tests for -x86. The -domain setting has no meaning unless you are running multiple assemblies. The key is to do nothing that causes NUnit to create a separate process.

Rebuilding for 4.0 is not necessary and isn't something we are likely to do so long as we support running on a machine with 2.0 only installed. Changing the config file is sufficient.

Directly linking to the framework is also something we wouldn't do, although it was a good idea as a diagnostic. The security possibility raises this question: are you running as admin? If not, you might give it a try.

Interesting that the difference only occurs on Windows!

I missed seeing your sample originally - thought you said you were going to write one. I'll check it out.

Charlie

[xplicit]

When I tested I made an decartian product of this values:
[--domain=None, Single,Multiply, (no option)]x[--process=Single,Separate,Multiple,(no option)]x[-x86 (has option),(no option)]. And tested all the combinations from this product. So command line
nunit-console.exe --process=Single assembly.dll was tested too.

Also, when I test NUnit compiled for .NET 4 I have removed the *.config file to exclude affecting .NET 2.0 stuff on the test.

The user is not Administrator, but it is included in the administrator group, I'll try to run the test under 'Administrator' user.

Did you able to reproduce the issue with my small sample?

[CharliePoole]

When I run your nunitPerf program directly, I get 16980ms. When I run it under the 3.0 a5 console I get 17036ms. Running it under nunitlite got me 16860ms.

[xplicit]

So you've got the same time for all runs. Is 3.0 a5 compiled under .NET 4.0 or .NET 2.0 target like 3.0 a4? Did you do any additional steps to run? Did you try to run nunit-console.exe --process=Single nunitPerf.exe?

I'll try to make tomorrow appveyor.yml for this project to demonstrate the difference

[CharliePoole]

All the NUnit 3.0 engine and runner assemblies are compiled for 2.0, since they are supposed to run on any machine with 2.0 or greater. The console runner runs under CLR4 if available, otherwise CLR2.

I didn't use any options on the command-line, so the runner ran in a separate process. Being in the same or a separate process should not make a difference, since you aren't timing NUnit's startup.

Maybe something in your own setup is causing this.

[xplicit]

I've got finally the same times for nunit-console and console application. It was successful only for 3a5, because for 3a4 the command without options does not work (it throws exception about .NET configuration System.ArgumentException: NUnit components for version 4.0.30319 of the CLR are not installed
Имя параметра: targetRuntime
в NUnit.Engine.Services.TestAgency.LaunchAgentProcess(RuntimeFramework targetRuntime, Boolean enableDebug, String agentArgs, Boolean useX86Agent))

So, this command shows the same time on my machine (18 sec):
nunit-console.exe nunitPerf.exe
These commands show other time (30 sec):
nunit-console.exe --framework=net-4.0 nunitPerf.exe
nunit-sonsole.exe --process=Single nunitPerf.exe

[CharliePoole]

I tried the same commands: the first gives me 17172ms, the second 17236ms.

[xplicit]

I tried to create appveyor build for the small project, but it does not show any differences in timing like on my local machine. I'll look more closely on original project issue (which was raised on appveyor) and try to create small reproducable project.

[xplicit]

Some additional info. I wrote a sample application, which loads all tests from the assembly and run them. Then I run this application and nunit-console with the same test assembly on appveyor. The times differ very significant.

Execution time.
Sample console app: 94 sec
nunit-console 3.0: 334 sec

Here is the appveyor log:
https://ci.appveyor.com/project/xplicit/bond/build/1.0.27

The commands I've run:
https://ci.appveyor.com/project/xplicit/bond/build/1.0.27#L1263
https://ci.appveyor.com/project/xplicit/bond/build/1.0.27#L1264

This unit testing log starts here:
https://ci.appveyor.com/project/xplicit/bond/build/1.0.27#L1282

Here is the source code of sample app:
https://github.com/xplicit/bond/blob/nunit3/ConsoleApplication1/Program.cs