NUnit 3.5.0 defaults to single agent process when using an nunit project file

[davidjward30]

NUnit 3.5.0 defaults to single agent process when using an nunit project file. This is an unexpected change in behaviour from 3.4.1.

Reproduction:

nunit3-console.exe input.nunit

input.nunit:

 <NUnitProject>
 <Settings activeconfig="active" appbase="D:\path"/>
 <Config name="active">
 <assembly path="A.Tests.dll" />
 <assembly path="B.Tests.dll"/>
 </Config>
 </NUnitProject>

Result;

When 3.4.1 reads an .nunit config file, it defaults to multiple processes.
When 3.5.0 reads an .nunit config file it defaults to single process.
When either 3.4.1 or 3.5.0 are passed a list of assemblies at the command line, both default to multiple processes.

Expected behaviour:

Default to multiple processes when using a project file.

Workaround:

Add --process=Multiple

[CharliePoole]

My suspicion is that we are implementing one process per command-line item where we previously did one process per assembly. There were a number of changes in how runners were set up in 3.5, so it's credible that this error could have come up, but I'm marking it confirm until I can verify it.

[CharliePoole]

Confirmed based on code review. We even have a test that makes sure things happen as described by @davidjward30. When neither the project file nor the command line specifies running in multiple processes, we should either default to multiple process or base the decision on whether the project file contains more than one assembly.

[jaytonic]

I've huge drawback performances due to this issues(we have an 8 core computer that runs a project with 40 tests assemblies only 1 core at a time. I'm very interessted to know the current status of this and if there is a workaround(except going back to the V3.4.1 or specifying the 40 projects manually?

[CharliePoole]

As you can see from the issue itself, the status is that nobody is working on it. 😞

I triaged the issue, confirming it and giving it a high priority. Now it's waiting for somebody to step up to fix it. That's how it works with a group of volunteers, I'm afraid.

If I remember correctly, the workaround is to not rely on defaults but specify the process model and number of agents on the command-line. Have you tried that?

[jnm2]

@CharliePoole what are we talking here, is this the bad test line?

[CharliePoole]

@jnm2 That test isn't bad because EngineTests.nunit only contains a single assembly. We. would need another test using a project with multiple assemblies.

Getting the right runner chosen would be a first step in fixing this but there may be more to it once that's done.

[jaytonic]

@CharliePoole I tried to specify the number of process with the --workers=2, should I see more than one NUnit Agent in the process explorer of windows?

[CharliePoole]

You want --process=Multiple --agents=n rather than --workers, which gives the number of test threads within each process.

[jaytonic]

@CharliePoole Sorry, I took the parameter from the documentation and I did a mistake, in my real case I've --agents=2(I don't have any --process specified, what is the difference between multiple and separate? Because I only want 1 process per assembly.

I just tried to add the --process=multiple too, but currently I have one two process, one using one full core, and the other staying always at 0, it feels like it's created but not used.

[CharliePoole]

--process=multiple means to run each assembly in a separate process. Separate means run them all in the same process. --agents is used to limit the number of processes created, which is supposed to default to the number of assemblies.

The XML result will tell you the process id under which each assembly ran.

[CharliePoole]

@jnm2 Are you actually still working on this?

[CharliePoole]

According to a separate report, the same thing happens with multiple .csproj files specified on the command-line rather than assemblies.

[jnm2]

@CharliePoole This is dead last on my list of tracked issues. Back when I was working on it, I was much lower on the learning curve. Would you be interested in taking over?

[CharliePoole]

I thought about doing it but my question was intended more as a reminder that nobody is working on it if you aren't, because your name is there. In the past, I've done the same myself when I thought I was probably the only one to work on a particular issue, so if that's the case it's OK to keep it.

Otherwise, remember that nobody will work on a job that has your name on it, except in certain corporate infighting situations... 😸 It only comes up in this case because somebody mentioned it in a different issue, which we marked as a duplicate.

[jnm2]

Okay. Since a fresh start will be needed no matter who picks it up, I'll stop tracking it as mine.