Eliminate in process execution

[CharliePoole]

In-process test execution is a big factor in the complexity of the engine. I'd like to eliminate it, either from the engine as a whole, or from the GUI itself.

People who run tests under the Test Window in VS are already running in a separate process. I see no drawback if the GUI works the same way. Folks could still choose to use either a separate process for each assembly or to run multiple assemblies in a single, separate process.

If made at the GUI level, this is a trivial change. However, any .nunit projects that specify in-process execution would still need to be dealt with by the engine.

One problem with this is that we have typically asked people to debug agent problems by running in process. However, this is not an available option for all the platforms we plan on adding. I think we need to bite the bullet and provide better info from agent failures as well as a clean way to debug in the agent. Eliminating in-process execution may just force us to do that.

If we implement it in the engine itself, we can simplify the engine enormously. That's my preference.

Most of you on the @TestCentric/gui-team are pretty busy at this time, but if you have any time to comment on this as a general direction it will be extremely useful. And even though we are maintaining a separate build of the engine I'd appreciate comments from @ChrisMaddock and @jnm2 as well.

[CharliePoole]

BTW, if we proceed in this direction, I'll deprecate inprocess execution for the next release. Actual elimination of the capability is obviously a breaking change and would come in version 2.

[CharliePoole]

I forgot to mention one important benefit of this change... maybe the most important!

If we eliminate in-process execution then we no longer need to build the engine for .NET 2.0. We could use, for example .NET 4.7.1 and .NET Core 3.0! We would still need the .NET 2.0 version of engine.core, so long as we support running .NET 2.0 tests.

[CharliePoole]

@mikkelbu @immeraufdemhund @ChrisMaddock @jnm2
Any thoughts pro or con?

[immeraufdemhund]

for me it causes more problems than it solves to use in-process. DEP-RE-CATE! DEP-RE_CATE!

[CharliePoole]

I'm going to deprecate inprocess now and remove for the 2.0 release.

[CharliePoole]

I'm not sure if this is a GUI or an Engine issue. That is, we could remove the use of in-process from the GUI entirely but still allow the engine to be used by some other runner to execute in-process. Some design and discussion needed for this.

[mikkelbu]

I don't mind removing the in-process execution, but as you states then we will have to do better wrt error-handling. Right now it is often difficult to debug hard problems (both in the console and in the adapter).

Edit: I'm also in favor of removing it from the engine, as I don't think a new runner will appear.

[ChrisMaddock]

Personally, I consider the pros of inprocess to be:

1. Easier debugging
2. Easier path to providing support for new platforms.

For the Nunit engine, id be interested in seeing how much simplification it would provide. If I was approaching this area, I’d instead first look at deprecating app domain support - which I think adds a similar level of complexity, and brings much lower value, and has much less usage. (Given some of the options have now been broken for over a year anyway!)

[CharliePoole]

How could you deprecate AppDomains? In .Net Framework we use a domain in order to load the tests at all.

I'm curious about what has been broken and why it hasn't affected the gui.

Main positive I see from dropping in-process is that the engine only needs to support one or two engines, like 4.8 and core 3.0.

[TFTomSun]

Please don't remove the inprocess option. It allows us to start TestCentric directly with the debugger (as a external tool). Without the option the debugger won't be attached to the actual test domain.

@CharliePoole
Building the tool for all available TargetFrameworks is easy if you switch to the SDK style projects.
You simply define all TargetFrameworks in a property group and msbuild (compile and packaging) will handle everything else for you.

[CharliePoole]

@TFTomSun Yes, that's what we do for the engine (which is now separate from NUnit's).

OTOH, the GUI itself pushes us toward using newer runtimes as we improve it. Currently the GUI is only built for .NET Framework 4.5 although I plan to add a .NET CoreApp version (3.1?) at some future point.

Note that the complexity that comes from running inprocess doesn't mainly come from how we build it, although that's part of it. It has to do with detecting whether the runtime you want to use is runnable in-process. So, for example, if you are running under .NET 4.5 in the GUI and ask to run a .NET Core 3.1 test in process, we have to tell you that's not possible. If you ask to run .NET 2.0, thats possible, but may call for a warning, etc. All of this means lot's more information that has to be presented to the user - who may be a beginner needing a thorough explanation.

None of that is an issue for a user at your level, who wants to attach an external debugger. I think we need to give you some way to do that. I think that means providing a special mode of execution for the agents, so you can kick them off directly under a debugger and so they won't depend on having the GUI there to display tests.

I agree with you about the need for inprocess currently. In fact, I use it all the time for debugging myself. Getting rid of inprocess is a long-range goal and is currently on the back burner. It won't happen until we have a different way to debug tests using a debugger of choice.

UPDATE: BTW I have tried to do an SDK project for the gui components themselves and was not able to. It's either not possible or requires some tricks I don't know in order to do it for a Windows Forms application.

[TFTomSun]

I replied in the other thread. Regarding the SDK Style projects: There's a migration documentation available here: https://docs.microsoft.com/de-de/dotnet/core/porting/winforms
I use WPF together with SDK style and did face any problems so far.

[CharliePoole]

@TFTomSun Thanks for the article, which I hadn't seen.

On the broader questions, you seem to have assumed that I'm prevented from supporting multiple platforms for the GUI by technical issues, which you are kindly providing - even though it's less kind to assume I don't know the basics of .NET development by this time. 😄

Technical difficulty is not the primary problem. It's level of effort both in development and in maintenance of the application. Code with five runtime targets probably doesn't cause five times the level of bugs and questions, but it might be two or three times. Each build that is released comes with a moral obligation to support it.

Currently my planning (which I assume you have read) calls for two builds of the GUI itself... .NET Framework 4.x and .NET Core 3.x - x to be determined at the point when we upgrade. Since the objective of TestCentric GUI version 1 is to match the capability of the old NUnit V2 GUI, the 1.x releases are only being built using .NET Framework 4.5.

If we limit ourselves for the moment to the .NET Framework, there are 14 runtimes we support, from .NET 2.0 through 4.8. Currently, they execute on each machine using two agents, built to target .NET 2.0 and .NET 4.0, but of course running on whatever version in each family is installed on that machine. You seem to be suggesting that I build the GUI for some (all?) of these runtimes. You seem to say that it would not be a lot of work. I can't agree with you on that and - of course - it is I who will have to do the work!

I should mention that a major goal of this project is to run sets of tests together, which cannot be run in the same process. This means that any implementation of in-process has to check for incompatible assemblies and issue error or warning messages. IOW it's not impossible but it gets messy and requires lots of maintenance as bugs arrive. This is BTW the same thing that the NUnit Project has encountered and it's why they are debating eliminating in-process execution.

I choose to contribute to the community by working on TestCentric. Nobody pays me to work or for the software. Rightly or wrongly, I believe that gives me the right to formulate and carry out a vision of what I want the application to be. Others who work with me have a similar right, in proportion to the level of effort they put in. You are a member of the community and have the right to ask for things and to make suggestions. But in this case, the suggestion seems to be that we should have a different concept of the project entirely - one that matches your particular needs.

I'm convinced that what you want is desirable and even necessary. I once even wrote an application (non-GUI however) to do it. It was called NUnitLite and version 1.0 came out in 2007 or 2008. I only mention it so you know I'm not denying that you need what you need. At some future time, there might even be a tc-lite that does what you want. But we are talking about one particular project here - the TestCentric GUI. Currently, purely by coincidence, version 1 of the TestCentric GUI is able to do what you want. If version 2.0, 3.0 or higher eliminates in-process execution, you will still be able to use the old version for that purpose. Of course, you will run one assembly at a time or will make sure that multiple assemblies are compatible, just as you now do. Or perhaps, by that future time, I'll have a different solution that meets your approval. 😄

[TFTomSun]

@CharliePoole
I wonder a bit about the long reponse here in this thread. I just posted a link with a SDK style migration instruction, because you said you had issues with that. In the other thread, you said ideas are welcome. I posted my idea there. I got no response to it yet.

Btw. it's not clear to me why you want to support .Net 4.x and .Net Core 3.x for the GUI, then. Both are just running on windows (because of Windows Forms dependency). If your plan is to generally decouple the test execution from the visualization process , one UI platform for windows is enough, isn't it? If you want to provide a platform independent UI a Blazor or Xamarin app might be a good choice.

[CharliePoole]

@TFTomSun I lost track of where I was posting, so that long post was intended to reply to both of your comments. I wrote it in the middle of the night, which is probably why I was confused!

I'm not sure how much you know about this project. There are plans to drop use of SWF and to move to another GUI platform... I don't know to what yet but there is an issue for selecting the new platform. Blazor or Xamarin are definite options in the future.

Yes, the entire NUnit architecture was posited on the idea of separation of the runner from the engine that does the running. TestCentric is trying to take that even further. For that reason, the front end needs to make use of a fairly high level of technology while the various back ends (agents and frameworks) should run on a variety of platforms. For example, it should be possible to run tests under .NET 2.0 or .NET Core 1.1 even though the front end doesn't need to run on those platforms.

Why both .net framework and core? Mainly I suppose because I'm not yet entirely comfortable with relying on .NET Core. Over the 40 years I've worked with this stuff I have learned the value of being slow to adopt what Microsoft says is to be the next technology. About half the time their initiatives end up abandoned. 😞 In the end, if a single choice becomes obvious I'll stick with it, provided it runs on both Windows and Linux.

I'll move over now to #557...

[TFTomSun]

I can understand that jumping on a new train is always a risk. But I'm sure that risk is zero with .net core. .Net 5 is based on .net core, all other platforms (full framework, mono) will be phased out. The goal from Microsoft is to have just one dotnet framework in the future again and this is the successor of . net core. There are already the first prereleases for dotnet 5 out there...

[CharliePoole]

Sure... do you know the phrase "cautiously optimistic?" Well, I'm something more like "suspiciously optimistic." 😄 Right now, the big focus is on actually running tests under .NET Core rather than changing the GUI anyway. I think it would be a mistake to assume that .NET Framework will disappear in the wild right away. So the problem of running tests under diverse platforms continues to be worth solving. Once it works with a .NET Framework runner launching tests under .NET Core, it will be able to also work in the reverse direction.

[CharliePoole]

🎉 This issue has been resolved in version 2.0.0-alpha1 🎉

The release is available on:

• GitHub Release
• NuGet Package
• Chocolatey Package