NUnit

[gertjvr]

Ported AutoFixture.xUnit.net on a one to one basis and just created an NUnitAddin called AutoFixtureNUnitExtensions as suggested on NUnit.org.

I have ran all test in the scenario and they pass.

As explained on NUnit.org, all you have to do is copy the addin dll in bin\addins folder where you are running nunit.exe or nunit-console.exe from.

Example:

[TestFixture]
public class Scenario
{
    [Test, AutoData]
    public void AutoDataProvidesCorrectInteger(int primitiveValue)
    {
        Assert.AreNotEqual(0, primitiveValue);
    }
}

[gertjvr]

@ecampidoglio just for my own sanity you got it working with R# 8.0.1?

[gertjvr]

@ecampidoglio wrote:

For convenience, I think we should include some batch files that copy the required DLLs from their respective NuGet packages into the specific folders for different test runners. Something like:

We could include that in the install.ps1 and uninstall.ps1 file verify known locations if they exists copy the required dll to those locations? would make it a lot more convenient, so many comment back @moodmosaic said that it would be ideal as user would be installing using nuget package exclusively.

But we could have a two phase approach have the individual bat files, and then figure out which ones to run in the insall.ps1 and uninstall.ps1 files.

[ecampidoglio]

@gertjvr Yes, I got it working but the RequiredAddIn attribute didn't work as expected. Even if I hadn't copied the DLLs into the addin folder, I was still able to run the test with R#'s test runner and got the aforementioned exception.

[gertjvr]

@ecampidoglio aforementioned exception being parameter count mismatch or required addin exception?

[ecampidoglio]

@gertjvr Correct.

[gertjvr]

@ploeh @moodmosaic @ecampidoglio before we go down the rabbit hole again, allow me to summarise what my thoughts are.

I was able to get the AutoFixture.NUnit2 working with nunit.exe and nunit-console.exe, R# and now TestDriven.Net three independent implementation of the nunit-runner, not without a few issues like the location and what dll's to copy.

@moodmosaic assume you have verified it working with TestDriven.Net and @ecampidoglio has got it working with R#

@ploeh has already explained that nunit extensibility model is to blame for the distribution issues (install / uninstall) and accepted the friction it might cause.

I don't want to waste your time or mine anymore reviewing this PR, the code has been unchanged for 18 days and the PR is over a month old.

I think it getting to the point either this is adding value or its not? I don't expect that there won't be any growing pains but think we have covered most common cases in the PR that might pop-up.

What do you say?

[ecampidoglio]

@moodmosaic

But then, are we going to include batch files to handle updates and removals too?

Well, at the very least we should find a solution for the update problem.

Each test runner add-in is going to have its own private copy of the Ploeh.AutoFixture.dll assembly, which may or may not be of the same version as the one used in the current project. This isn't necessarily a problem per se. One version of AutoFixture would be used to build the project while another one would be used to run the tests. As long as their compatible with each other, everything is fine.

However, if we were to add a new feature or a bug fix in AutoFixture core that the user wishes to take advantage of in her project, she'd have to manually update the DLL for all of the test runners, both on the local machine as well as on any CI servers.
By the same token, if we were to introduce a breaking change in Ploeh.AutoFixture.dll, some tests might suddenly begin to fail on configurations where the test runner add-in and the package used in the project weren't kept in sync.

One way to mitigate this could be to include some code in the add-in that, upon startup, compares the version of AutoFixture that is being linked to from the project and the one used by the add-in itself. This way it could warn the user if the versions mismatch and, depending on which part of the version number is different, recommend to update.

[moodmosaic]

@moodmosaic assume you have verified it working with TestDriven.Net

No, it's not working for me out of the box – but please don't interpret this as a discouragement.

All I am saying is that it would be nice to protect our users so they should not be left frustrated on the release date because the batteries were not included..

[gertjvr]

@ploeh @moodmosaic @ecampidoglio I looked at the dependency hierarchy moved the NUnitAddin to its own project and moved DataAttribute to the same project thereby removing any dependency on AutoFixture itself. Allowing us to only have to copy one dll Ploeh.AutoFixture.NUnit2.Addins.dll to the addins folders. This will solve the updates and versioning issue.

I have ran the msbuild script without any errors or warnings. and retested with nunit.exe R# and testdriven.net with success.

Unless we include the xcopy of this single dll to known locations withing the install.ps1 there will never be an out of the box solutions, this will ensure as @moodmosaic put it "that the batteries are included".

Please could you guys review this for a final time, but I am going to park it there.

[ploeh]

@gertjvr wrote:

@ploeh What is outstanding before you can merge this PR? I haven't changed any of the code for 18 days if you exclude the install.ps1 script, only thing in question is how 3rd party tools interact with the addin, and inperticuar how TestDriven.Net report the missing RequiredAddin.

Well, I was pretty much ready to pull, but then you started churning on the Install.ps1 file. AFAICT, that particular file doesn't work.

I don't think we necessarily need this automation in order to release the package. I'd rather release the package without a faulty automation script, than with a faulty script.

If we release something with a script that doesn't always work, we'll have to support it, and it may be that we'll never be able to make it work. Releasing a script can be interpreted as a promise that this is part of the package.

If we don't release the script, it's pretty clear to everyone that some assembly may be required. Then we can always add the script later, if we can make it work.

Thus, it seems to me that it would be wiser to yank the script for now...

[gertjvr]

@ploeh The script isn't in question anymore @moodmosaic started the discussion about nunit extensibility, and down the discussion went.

I think we park this PR as its not going anywhere, and I don't have the energy, time or patience anymore.

Thanks everyone cheers.

[gertjvr]

@ploeh few of my colleagues convinced me this will be a waste not to pursue. This feature will be main counter argument for other devs / clients to use AutoFixture (thank you @robdmoore and @bradleyboveinis)

I might have overreacted, and I apologise (its no excuse but not getting much sleep these last few days)

The code is functional has been tested with at least three independent implementations of nunit-runners (nunit, R#, TestDriven.Net) (Also removed the AutoFixture dependency from AutoFixture.NUnit2.Addins.dll so only need to copy single assembly to the required addins folder, and is future proof as isn't depended on a specific version of the AutoFixture.NUnit2 assembly)

The install.ps1 uses only powershell functions and will add the RequiredAddin reliably imo, do you still want to remove this? (only remaining issue why this could fail is if the developer doesn't have execution rights)

@moodmosaic I am not going to discuss the nunit extensibility issue, out of the box isn't possible and will require some friction. (ie copying dll or orphan dll removal), it is what it is!

I'll understand if you don't want to merge this anymore.

[ploeh]

@gertjvr Glad to see you haven't given up completely. I agree that we can't keep thrashing on the NUnit extensibility part, because we can't do anything about it.

There's no purpose in attempting to make this perfect, because it's not going to be perfect - just like the rest of the AutoFixture code base isn't perfect. The question is whether or not this PR adds more value than friction, and I believe there's quite a good chance that this is the case.

I have some time later today when I'll attempt to pull and look this over again.

@moodmosaic and @ecampidoglio Thank you very much for your diligent and thorough review of this PR. I agree that perhaps more batch files etc. might be an added benefit, but isn't this something that can be added later? Given that the NUnit extensibility story is as it is, are there any blockers in this PR?

[moodmosaic]

@ploeh Up to commit ba9bd81 there are no blockers on my machine. (I haven't tested the changes from yesterday's commits.)

[gertjvr]

@ploeh Just as a guide, you only have to copy the Ploeh.AutoFixture.NUnit2.Addins.dll to addins folder

• NUnit -> C:\Program Files (x86)\NUnit 2.6.2\bin\addins
• TD.Net -> C:\Program Files (x86)\TestDriven.NET 3\NUnit\2.6\addins
• R# -> C:\Program Files (x86)\JetBrains\ReSharper\v8.0\Bin\addins

If you install the AutoFixture.NUnit2 package using package manager
Install-Package AutoFixture.NUnit2 -source <package location> then PowerShell ExecutionPolicy must be set to RemoteSigned if you use a predefined packages source like "local nuget feed" then this is unessesary.

[ploeh]

This is now live as AutoFixture.NUnit2 3.9.0. Thank you for your contribution and your patience.

P.S. I've noticed that the number of tests passed reported by the build server is constant at 4089 tests passed (and 0 failed). I would have expected that number to have increased, so it looks like the tests aren't being executed on the build server.

[moodmosaic]

Here is another surprising behavior:

If you make a few tests to fail (in the AutoFixture.NUnit2 extension) you will notice that the number of tests failed reported by the BuildRelease.ps1 is constant at 1 failed:

...\BuildRelease.msbuild" (default target) (1) ->
(Test target) ->
...\BuildRelease.msbuild(95,9): error MSB6006: "nunit-console.exe" exited with code 2.

    0 Warning(s)
    1 Error(s)

[gertjvr]

@moodmosaic the reasons for this is the nunit runner has been setup to not continue on failure 'ContinueOnError="false"'

<Target Name="Test" DependsOnTargets="CopyToNUnitAddinsFolder">
    <xunitproject ProjectFile="Src\All.xunit" />
    <NUnit ToolPath="$(NUnitToolsFolder)" Assemblies="Src\AutoFixture.NUnit2.UnitTest\bin\Release\Ploeh.AutoFixture.NUnit2.UnitTest.dll" ContinueOnError="false" />
    <NUnit ToolPath="$(NUnitToolsFolder)" Assemblies="Src\AutoFixture.NUnit2.Addins.UnitTest\bin\Release\Ploeh.AutoFixture.NUnit2.Addins.UnitTest.dll" ContinueOnError="false" />
</Target>

If this isn't the desired result then you can remove the attribute or set it to true.

[ploeh]

FWIW I've now changed ContinueOnError to true.

[kmcginnes]

I wanted to chime in quickly about a comment made a while back about TestDriven.Net failing to execute as expected when running a parameterized test with the AutoData attribute. The reason behind this is simple, but not obvious.

As a reminder, here is the test output that @gertjvr reported:

------ Test started: Assembly: ClassLibrary1.dll ------

Test 'M:ClassLibrary1.Class1.AutoTestCaseProvidesCorrectInteger(System.Int32)' failed:   Expected: True
  But was:  False

    NUnit.Framework.AssertionException:   Expected: True
      But was:  False

    at NUnit.Framework.Assert.That(Object actual, IResolveConstraint expression, String message, Object[] args)
    at NUnit.Framework.Assert.IsTrue(Boolean condition)
    Class1.cs(12,0): at ClassLibrary1.Class1.AutoTestCaseProvidesCorrectInteger(Int32 primitiveValue)

0 passed, 1 failed, 1 skipped (see 'Task List'), took 0.47 seconds (Ad hoc).

I'm going to quote Patrick Lioi who is currently on an adventure to write a convention based testing framework he calls Fixie. In his very first post regarding Fixie he encounters this very "issue". He states, "...TD.NET has a convenient feature that it can run methods even if they are not officially recognized as test methods in a test fixture."

When the test was executed, NUnit didn't recognize it as a test because the method signature included parameters, and TestCase attribute was not found. Therefore, TestDriven.Net executed the method in ad hoc mode. This is evident by the last line of the test output.

---

As an aside, I'm glad everyone decided to move ahead with this pull request. Even with the added friction of NUnit's poorly thought out extensibility model, the value added from this functionality is tremendous. My deepest thanks to all those involved.