NUnitLite could process multiple assemblies

[oznetmaster]

Currently in the built-in runner in nunitlite, TextUI correctly captures multiple assemblies on the command line, but then only processes the first of them.

                // TODO: For now, ignore all but first assembly
                Assembly assembly = _assemblies[0];

Is there a reason that all assemblies provided could not be processed? It would be seem to be beneficial to be able to.

[CharliePoole]

That makes sense to me. If we don't do it, we should get the code out of there that accepts multiple assemblies as input in the first place.

I think that this will become more feasible after we separate the runner assemblies from the framework assemblies.

[oznetmaster]

Since the nunit-console runner does this, it should be possible to migrate it to the nunitlite,runner.

I will investigate this.

[CharliePoole]

@oznetmaster We listed this as Future, which means we were not going to do it for 3.0. If you don't like that decision, it should be discussed, not just changed.

To tell you the truth, I'm not sure we should ever do this. It might make more sense to remove the capability from the NUnitLite runner of running any assemblies other than the one that links to it. After all, that's what it's for.

We can go to one of our mail lists or discuss it here. I've cleared the milestone and 3.0 label, which is for the purpose of identifying all the stuff that will go into 3.0. If you're comfortable with making the discussion happen, I'll leave you assigned for that purpose.

[oznetmaster]

I regularly run hundreds of different test assemblies under nunitlite on CF, none of which are directly linked with nunitlite build that has the nunitlite runner in it. I always load them independently of nunitframework/nunitrunner, and my gui allows me to select which test assembly(s) to run at any given time. I currently support running multiple assemblies in my CF build. However, like the command line stuff before I unified it, it is done differently than nunit-console does it.

Taking the ability to specify test assemblies on the command line would actually make nunit useless to me in the Crestron environment.

[oznetmaster]

I actually think the nunitlite runner should be able to do anything appropriate (that the platform supports) that the nunit-console can do. That is another discussion, but that is actually what I have/am movinf towards implementing in my CF build. That was my motivation for integrating the command line option processing.

I am working with Crestron to build the nunit framework and nunitlite.runner into the S# environment. This means that it will not be part of any test assembly, nor will any test assembly be packaged with it. It will be in the GAC for the CF device. Here again, the built-in console command line will be the basic way that a test assembly is referenced/run.

When a remote runner is available, that will be integrated as well, but will never be required. The local command line runner will always be available.

[CharliePoole]

It sounds like you are moving in a different direction from where NUnit has
been going. In general, tests are supposed to be executed under control of
the TestEngine, which can run as many assemblies as you like, so long as we
make a driver available to run them. In the engine architecture, a driver
can only run a single assembly, but you can have lots of drivers running at
one time, even in the same AppDomain. That's how we get multiple assemblies
to run. I mention this, because the NUnitLite "runner" is very much like a
sort of standalone driver.

Of course, this is open source, and if you would prefer to take your build
in a different direction, that's what it's all about. I'd feel better about
it, however, if I felt you had first grokked our approach, before rejecting
it.

That's enough ranting on my part. I'm hoping lots more folks than you and I
will have an opinion on this.

On Sat, Dec 6, 2014 at 8:21 PM, Neil Colvin notifications@github.com
wrote:

I actually think the nunitlite runner should be able to do anything
appropriate (that the platform supports) that the nunit-console can do.
That is another discussion, but that is actually what I have/am movinf
towards implementing in my CF build. That was my motivation for integrating
the command line option processing.

I am working with Crestron to build the nunit framework and
nunitlite.runner into the S# environment. This means that it will not be
part of any test assembly, nor will any test assembly be packaged with it.
It will be in the GAC for the CF device. Here again, the built-in console
command line will be the basic way that a test assembly is referenced/run.

When a remote runner is available, that will be integrated as well, but
will never be required. The local command line runner will always be
available.

From: CharliePoole [mailto:notifications@github.com]
Sent: Sunday, December 7, 2014 11:46 AM
To: nunit/nunit
Cc: Neil Colvin
Subject: Re: [nunit] NUnitLite could process multiple assemblies (#344)

@oznetmaster https://github.com/oznetmaster We listed this as Future,
which means we were not going to do it for 3.0. If you don't like that
decision, it should be discussed, not just changed.

To tell you the truth, I'm not sure we should ever do this. It might make
more sense to remove the capability from the NUnitLite runner of running
any assemblies other than the one that links to it. After all, that's what
it's for.

We can go to one of our mail lists or discuss it here. I've cleared the
milestone and 3.0 label, which is for the purpose of identifying all the
stuff that will go into 3.0. If you're comfortable with making the
discussion happen, I'll leave you assigned for that purpose.

—
Reply to this email directly or view it on GitHub <
https://github.com/nunit/nunit/issues/344#issuecomment-65925299> . <
https://github.com/notifications/beacon/AFh64flGXn0QxyhcsUsqTtLdbDNCX_Lbks5nU8TwgaJpZM4C4mZD.gif>

—
Reply to this email directly or view it on GitHub
#344 (comment).

[oznetmaster]

I have no choice, since there is no current architecture that allows the engine/agent/runner to be used in the CF environment (due to the lack of remoting).

We have talked about designing such an architecture.

However, I have to assume that this will also be used in a standalone CF environment, which does not support remoting, for the forseable future. If you have another suggestion, I am totally open to hearing it.

[oznetmaster]

My ideal solution would be to be able to just include nunit-console/engine/agent in replacement of nunit.runner, but that does not seem to be currently possible? It would be nice if it could be built as a single assembly as well.

[CharliePoole]

After thinking about this, one conclusion is that our ability to run tests remotely is too abstract right now. We need to do something to make it feel more real... like write the code!

[oznetmaster]

I guess that is what I am doing :) But not generically.

From: CharliePoole [mailto:notifications@github.com]
Sent: Sunday, December 7, 2014 11:29 PM
To: nunit/nunit
Cc: Neil Colvin
Subject: Re: [nunit] NUnitLite could process multiple assemblies (#344)

After thinking about this, one conclusion is that our ability to run tests remotely is too abstract right now. We need to do something to make it feel more real... like write the code!

—
Reply to this email directly or view it on GitHub #344 (comment) . https://github.com/notifications/beacon/AFh64Vk0VaCXU_-IfQM2lvz8OxJpyxllks5nVGnGgaJpZM4C4mZD.gif

[oznetmaster]

However, that is only half the issue. I am working with a class of CF devices where full control from the console is required. So having the equivalent of a totally local nunit-console is a necessity.

Also, I can only run a single exe per “app domain”, so everything else has to be in loadable assemblies. The only cross “appdomain” communication is TCP/IP, even on the same device.

From: CharliePoole [mailto:notifications@github.com]
Sent: Sunday, December 7, 2014 11:29 PM
To: nunit/nunit
Cc: Neil Colvin
Subject: Re: [nunit] NUnitLite could process multiple assemblies (#344)

After thinking about this, one conclusion is that our ability to run tests remotely is too abstract right now. We need to do something to make it feel more real... like write the code!

—
Reply to this email directly or view it on GitHub #344 (comment) . https://github.com/notifications/beacon/AFh64Vk0VaCXU_-IfQM2lvz8OxJpyxllks5nVGnGgaJpZM4C4mZD.gif

[CharliePoole]

After review, we have decided to keep NUnitLite as simple as possible. Running multiple assemblies will requier use of a runner that makes use of the test engine. This is in constrast to NUnitLite, which is intended to run a single test assembly without use of the engine.

[oznetmaster]

I perhaps need to be more precise.

In the nunitlite environment, allowing multiple test assemblies on the command line would seemingly simply sequentially run the tests in those assemblies. The test assemblies would all have been compiled for the same framework, and same environment. The output from those tests would either be concatenated in a single output file, or perhaps in multiple files based on the test assemblies name.

This is a significant simplification from the current console/engine/agent situation, as I thought was the intent on nunitlite.

[CharliePoole]

@oznetmaster I'm glad we are in agreement about using TCP and the console runner. That's our direction and has been for a long time. Where we seem to part company is that I feel very strongly that we lose a lot of time to work on the long-term solution as we attempt to make NUnitLite into something that will suit all needs.

I feel that I have to point out that you already did assign this issue to yourself, over a year ago. IIRC that was as a result of some discussion you and I had. But nothing more ever happened with it and I closed it a few months ago for that reason. It sounds (in the current discussion) as if you were expecting somebody else to do something, but I'm not sure what that was.

If you would like to re-open this and work on it, then I think you should describe what you would do and for what platforms. I'll respond separately to the description you just posted.

[CharliePoole]

Comments on @oznetmaster 's proposal:

"I perhaps need to be more precise."

Thanks. :-)

"In the nunitlite environment, allowing multiple test assemblies on the command line would seemingly simply sequentially run the tests in those assemblies. The test assemblies would all have been compiled for the same framework, and same environment. The output from those tests would either be concatenated in a single output file, or perhaps in multiple files based on the test assemblies name."

• I assume you are refering to the nunitlite-runner command-line, not that for self-executing tests.
• As you say, all the assemblies would have to be compiled for the same runtime framework for which the particular copy of nunitlite-runner was designed, that is, all silverlight, all net.4.5, all portable, etc.
• Sequential execution without any combination would be the easiest approach.
• Text output would have to be concatenated.
• XML output can't be concatenated, since that would produce an illegal XML file. We would have to add at least minimal code to combine the results into a single valid file.
• Creating separate output files is a bit problematic if we want to keep using the --result option to specify where it goes.

"This is a significant simplification from the current console/engine/agent situation, as I thought was the intent on nunitlite."

Of course, only running one file per execution is even simpler.

[oznetmaster]

Yes, I did assign this to myself, and was developing a "mini-engine" to base this on. Then you assigned the core engine to yourself. We all agreed that using the core engine would be a consistent approach for all platforms.

Then the decision was made by you to dump the core engine.

I have been waiting for some consensus about a replacement strategy, so that I do not go "running off in my own direction again").

Perhaps I missed the arrival at that consensus?

[oznetmaster]

I fully agree with everything @CharliePoole wrote in his elaboration of my description of how this would work.

If there is an agreement that this would make sense for the nunitlite-runner command line, I will assign it to myself to implement. I would probably implement it in TextRunner, so it would be available for other nunitlite runners if desired.

[CharliePoole]

@oznetmaster What do you mean by "other" nunitlite runners? That is, what's the scope of what you propose to implement versus what might be implemented by others later on.

[CharliePoole]

@oznetmaster Some of the points you are agreeing with were actually intended as negatives WRT this feature. See the last two bullet points.

I could be talked into something minimalistic that allowed multiple files on the command-line provided it didn't interfere with AutoRun and didn't change the fundamental purpose of nunitlite-runner in doing CI builds. My sense, however is that you actually want a runner that runs on CF, not the CI platform.

[oznetmaster]

Since I would implement the feature as in TextRunner, as a new Execute method with a bool flag indicating whether to allow multiple files on the command line, it could be used by anyone who calls TextRunner.Execute. The bool would allow for an error to be given if there are multiple files on the command line and only one is allowed.

I would also add the error for a testAssembly being provided in the constructor, and then files being specified on the command line. I fully agree with @CharliePoole that these things should not fail silently.

I know where I would use the capability.

It could be used in the CI builds, if desired, or anywhere else. It would not have any affect on AutoRun, unless a new AutoRun.Execute method were added to take advantage of it, but that would seem to be against the current philosophy (not sure why). Note that AutoRun can currently be constructed with a null testAssembly, and then file names can be passed on the command line, and the first file name will be used. Is this to be changed to cause an error instead?

[CharliePoole]

@oznetmaster Seems as if the presence or absence of a test assembly could serve as the flag without messing up TextRunner with an extra argument. Adding a guard clause to AutoRun for a null assembly argument makes sense as well.

However, my main question remains unanswered, perhaps because I have not spelled it out clearly enough: Would you plan on creating a new build of nunitlite-runner to allow executing tests from the command-line on CF devices that have a console?

If that's the plan, it would be good to test whether it can be used in the CI builds. There's a good chance that it may be useable, due to .NET's assembly unification feature, but I'm not entirely sure if that is still supported for CF in newer .NET releases. It should be easy to find out, however.

If OTOH you find that that's not possible, then I think the CI runner and the device runner will need to be entirely separate programs. For me, this would reopen the question of what we call the program and I would advocate a name like nunitlite-ci-runner for the programs that are designed to allow running tests compiled for a device on the desktop, while keeping nunitlite-runner for the programs that run entirely in the device environment.

[oznetmaster]

If you remember, I already have a version of AutoRun which has been adapted to run on the CF console on the device itself. It is more robust then the current nunitlite-runner, but basically does the same thing. I see no reason it should not work for the CI builds as well, but would certainly test that. There is nothing CF specific in it.

[CharliePoole]

@nunit/framework-team I propose we wrap this up once and for all. From our discussion, it's clear that we want nunitlite to be for primarily for self-executing tests. A secondary use is for single assemblies that can't be self-executing because of their platform. In that case, nunitlite-runner is used to pass a single assembly to nunitlite.

To handle both cases, I propose we eliminate the detection of a file on the nunitlite command-line and let the runner do it. The runner can pass the identity of a pre-loaded assembly using a special interface, separate from AutoRun, which handles the main, self-executing case.

Comments?

[rprouse]

I agree, I think the standard AutoRun should just execute the current assembly, or you should pass in the assembly (in code) that you want to execute.