appdomain per test (isolation per test), for dealing with static classes

[dbones]

<< Feature Request >>
Run each test in its own app domain.

My subject under test (the actual service) has a dependency on a library which uses static classes.

between each I create a new instance of the Subject, but as it has this dependency it is unable to be an isolated test.

in NUnit 2.6.x I created a new AppDomain in the test setup, and run the subject inside that. This seemed to work (tiny bit flaky), I moved to the latest verion of nunit 3.7.x and my small solution would pass one test and then hung.

I am going to give this ago https://bitbucket.org/zastrowm/nunit.applicationdomain/overview

however I would be epic to have this support out of the box.

[CharliePoole]

I think this would be a cool feature... we could have an attribute that ran either an entire fixture or an individual test case in a separate appdomain, depending where it ws placed. Could be the start of a whole new set of isolation options.

It's a fairly big item but if the referenced app could be a source of ideas that would help. I don't believe we can use it as a source of code, since it's Apache 2 licensed.

[jnm2]

Since AppDomains are not a thing in .NET Core, and there are actually decent reasons why, this would have to be disabled in .NET Core/.NET Standard builds of the framework or else we'd have to provide out-of-proc execution per test. Out-of-proc is where you're supposed to go now if you need that kind of isolation.

[jnm2]

https://tldrlegal.com/license/apache-license-2.0-(apache-2.0)
https://tldrlegal.com/license/mit-license

I don't know anything about them, but only additional restriction of Apache 2 vs MIT seems to be that you have to state changes and if you have a NOTICES file, you have to include their notice.

[CharliePoole]

There are two ways to go.

1. Assume it's OK if you can't see any reason why not.
2. Assume it's not OK unless you get a qualified legal opinion.

As you can probably guess, I'm in camp 2. I've done extensive research trying to find an IP lawyer who published something about this and I failed. I think I even asked Rosen once, without getting an answer.

If the "just take ideas" approach doesn't make you happy, then you can get the author to contribuite it. However, in this case, it's not rocket science so it's probably easiest to just re-invent. We certainly know how to spawn domains and processes in NUnit.

[CharliePoole]

@jnm2 Based on this: http://www.apache.org/foundation/license-faq.html#Distribute-changes I'd say we can use a component licensed with the Apache 2 license provided we jump through a few hoops. Slightly different hoops depending on whether we change the code or use it as is. Not something to worry about till it comes up, I would say.

[jnm2]

@CharliePoole The "just take ideas" approach actually makes me happier either way. I was just interested because I thought of Apache and MIT as almost exactly the same, and we're MIT, and I thought part of the point was so that people could just take code if they wanted.

[CharliePoole]

Every license is different. MIT lets you do just about anything. Apache lets you do a lot of stuff but requires a few things of you, like including the license and any notices. Corporations have somebody review this stuff and make sure they are in compliance. We have always had to make do with research and free legal help where available.

Over the years, I've had to take action myself against a violator of our license a few times. That was our earlier license, which required notices to be kept if they used the code. With MIT, there are only a few things they could do that violate it and none of them are profitable. 😄

[jnm2]

So we need to decide if we want to implement AppDomain isolation and keep it .NET Framework-only (Does Mono have AppDomain support?), or implement process isolation and have it work the same everywhere. We could end up doing both too, though it seems simpler to do just one than both.

Either way, I'm expecting AppDomain-reuse or process-reuse to be important for performance if you have a lot of individual test cases which all use static state.

(Or if all that sounds like it's too complex, we could go with the conventional wisdom that static state is anathema for testing and that dependencies should be injected.)

[CharliePoole]

@jnm2 This issue asks for AppDomain isolation.

Following our conventions, you actually already said we would do it, by adding the labels you added, but then your comment says we have to decide if we will do it. If I were the OP I would be confused. 😄 But then I already said elsewhere that our labels are pretty confusing!

Inability to implement something on every platform has never stopped us from doing something before. We just don't allow it on the platforms where it doesn't work. In fact, there's lots of stuff with AppDomains that we don't do under .NET Standard already. This is just a bit more user-facing than some other things.

My feeling is that implementing AppDomain isolation will eventually lead to Process isolation, but we don't have to figure it all out now. That's YAGNI. Same with reuse. Like you, I'm sure we will get want it at some point but it feels like an optimization that we shouldn't worry about in the first implementation of a new feature.

[jnm2]

@CharliePoole Which label communicates that we have decided to do this, the pri: label?

[CharliePoole]

Yes, together with the is:enhancement label rather than is:idea. See the wiki page about labels.

As already stated, I have come to the conclusion that the system of labels I set up is OK for us to use once we get used to it but doesn't say much to users.

If I had triaged this, I would have marked it as you did minus design and added it to BackLog. Everything needs design, but the label means "don't code till you show us a design." If this is just about what it says - running tests in a separate AppDomain - it's pretty straightforward design-wise even though there are lots of implementation issues to work out. Doing that would mean that somebody is free to do a PR.

[jnm2]

I did read the is: docs and saw An addition or improvement to an existing feature. but nothing suggesting that the team has decided to do the thing being discussed.

design:

Some design decisions need to be made before this can really be worked on. Sometimes this label may be applied before anything happens and other times the work may have started but reached a point where design decisions need to be made involving others in the team.

Seems to fit a scenario where I'm asking the team to decide whether it's worth adding an AppDomain feature versus an out-of-proc feature before anyone writes code =) I am leaning toward the latter. Once you and perhaps another team member respond that it's worth doing the AppDomain feature, then I would expect the label to be removed and an assignment or documentation for up-for-grabs to happen.

I also wouldn't expect it to go in the backlog if we're still discussing whether we want the feature as described vs alternatives.