Weird ApprovalTests issue when running in TeamCity

[kharabasz]

Hello,

We have integrated ApprovalTests into our CI environment - unfortunately we are running into a weird issue around build agents.

When ApprovalTests run on the same build agent as they were built, there is no problem. Tests run, everyone is happy.

When ApprovalTests are run on a different build agent, we see the following error:

[_____.ApprovalTests] Test method _____.ApprovalTests.SomeApprovalTest threw exception: 
ApprovalTests.Core.Exceptions.ApprovalMissingException: Failed Approval: Approval File "c:\TeamCity\BuildAgent10\work\cf9524f6b7a5308c\_____.ApprovalTests\ApprovalTests\SomeApprovalTest.approved.txt" Not Found.
[10:50:24]
[_____.ApprovalTests.SomeApprovalTest]     at ApprovalTests.Approvers.FileApprover.Fail()
   at ApprovalTests.Core.Approvals.Verify(IApprovalApprover approver, IApprovalFailureReporter reporter)
   at ApprovalTests.Approvals.Verify(IApprovalWriter writer, IApprovalNamer namer, IApprovalFailureReporter reporter)
   at ApprovalTests.Approvals.Verify(IApprovalWriter writer)
   at ApprovalTests.Approvals.Verify(String text)
   at ApprovalTests.Approvals.VerifyAll[T](IEnumerable`1 enumerable, String label)
   at _____.ApprovalTests.SomeApprovalTest () in c:\TeamCity\BuildAgent10\work\cf9524f6b7a5308c\_____.ApprovalTests\ApprovalsFixture.cs:line 34

It looks like the test runner is looking for the *.approved.txt files based on their first location?

We have tried to resolve this using DeploymentItems, and by using the [assembly: FrontLoadedReporter(typeof(TeamCityReporter))] annotation - with no success.

I have been able to recreate this issue locally, by running ApprovalTests using MSTest from the cmd line from a different folder than they were built to.

[jamesrcounts]

I can confirm that moving the assembly breaks things. We have an attribute that allows you to put your approval files in a subdirectory, but I haven't tested it with a full path. I think your situation would call for a full path, but I don't use TeamCity.

So I don't know of an out of the box fix, but something might be possible.

[kharabasz]

I think for this case, We would want to resolve against a relative path.

[jamesrcounts]

My understanding is that the path is derived from the debugging info embedded in the assembly. This is how it is able to find the directory where the source code is located rather than the directory where the assembly is located, its also why it doesn't work if you move the sources. I've never moved "only" the assembly, I have moved assembly and sources and seen it break.

To compute a relative path we would need some place to start. Lets say we start at Assembly.location, which would be the bin directory. We would need to know how high up the path we need to climb before finding the approved files.

Not saying it can't be done, just thinking out loud...

[kharabasz]

I had a hunch it was derived from the debugging info - that would fit the scenario we are seeing.

Can we build out a custom namer that places approval files in some sort of standardized directory structure?
example {Assembly.Location}/Approvals/{Test Namespace}/{Test Name}.approved.txt
(That may actually be too long to be useful)

Mark those as copy always and we should be set for most test runners?

[jamesrcounts]

Seems like that would work in principle, but we would need to make it easier to take control of the list of stack trace parsers. Right now I find this a difficult part of the codebase to wrap my head around.

[kharabasz]

Could you point me @ some documentation in what order components get fired?
Also @ documentation around how to integrate a custom namer?

[jamesrcounts]

Not much in the way of documentation I'm afraid. But here is where this gets coordinated:
https://github.com/approvals/ApprovalTests.Net/blob/2075e4d72f2f0170cb754010ecd1d62e2c35b6d9/ApprovalTests/StackTraceParsers/StackTraceParser.cs

[kharabasz]

This has been fun due to all of the gotchas w/ MSTest under VS2010 (we really need to upgrade to 2012/2013)

I have a prototype working that works as such:

[TestClass]
[UseNamer(typeof(RelativeToAssemblyNamer))]
[DeploymentItem(@"TestProject\Approvals", @"Approvals")]
public class SomeTests
{
    [TestMethod]
    public void SomeTest()
    {
        var resultingEnumerable = ...
        Approvals.VerifyAll(resultingEnumerable, "Something Descriptive");
    }
}

The UseNamerAttribute just lets you override the defaultNamerCreator on the Approvals class. The RelativeToAssemblyNamer just uses Assembly.GetExecutingAssembly().Location.

However, I think it would be a good idea to make it possible for the namer to be conditional based on the environment it is running in.

What do you think?

[kharabasz]

MSTest gotchas under VS2010:

If your folder structure looks like this:
SolutionFolder
----TestProjectFolder
-------- ApprovalsFolder\

Trying to do [DeploymentItem(@"ApprovalsFolder", @"ApprovalsFolder")] will fail silently, due to some silly edge case when sourcePath equals targetPath. You instead need to do [DeploymentItem(@"TestProjectFolder\ApprovalsFolder", @"ApprovalsFolder")]

Deployment Items need to be enabled in the .testsettingsfile and the approval files need to have their "Copy To Output Folder" property set to "Copy Always" or "Copy if Newer".

Oh, and this absolutely fails if the path + name of your approval files is too long as DeploymentItem silently fails if it encounters the old MAX_PATH limit. I don't know if newer versions of the MSTest runner fix this.

[jamesrcounts]

For starters, I think you are awesome!

[kharabasz]

The concept of an EnvironmentAwareReporter is mixed with the concept of a FileTypeAwareReporter :(

[kharabasz]

I still need to add some tests for RelativeAssemblyNamer. Take a peek.

Doing something like this for test set-up:

[UseNamer(typeof(RelativeToAssemblyNamer), ForReporterOfType = typeof(TeamCityReporter))]
[UseApprovalSubdirectory("Approvals")]
[FrontLoadedReporter(typeof(TeamCityReporter))]
[UseReporter(typeof(DiffReporter))]
[DeploymentItem("MyProject.Tests\\Approvals", "Approvals")]

[jnsn]

I am running into the same issue, using Bamboo. I have a plan consisting of multiple stages and each stage has it's own working directory. To share build items between two stages, the first stage has to publish them. So basically, what happens is the following:

• Stage 1
  • Check out sources
  • Build project
  • Zip entire MyProjects.Test folder (including .csproj, sources and bin/Release)
  • Upload zip
• Stage 2
  • Download zip
  • Run tests

I have not tried the attributes trick, but I'd like to give some input.

Our approved files are checked in, but they are not part of the .cproj file and therefore not copied to the output folder. But by default they show up in the same folder as where the test class was. I sometimes use the NamerFactory.AdditionalInfo to generate a more useful name.

Since I'm also using SpecFlow, I need to zip the entire MyProject.Test folder because SpecFlow analyses the .csproj file for .feature files. So the .csproj file is a mandatory argument.

Wouldn't it be possible for ApprovalTests to do the same? I don't know about the other test frameworks, but NUnit is capable of running the tests when I pass the .csproj file without the need to recompile everything. I've noticed that in #124 several assumptions are made based on the namespace and path, but if you'd were to analyze the .csproj file based upon the current class you are processing, wouldn't it be possible to match the relative path?

[julian-maughan]

@jnsn Good to hear from you. Your understanding of the way SpecFlow works doesn't make sense to me. In particular, to run SpecFlow, it shouldn't be necessary to copy the whole MyProjects.Test folder to Stage 2 - only the build outputs.

SpecFlow does not require a .feature file at run time; only design time. During development, .feature files produce a code-behind class containing NUnit tests (or whatever testing framework you are using), which can be run just as if they were normal NUnit tests that you had written yourself...they are compiled into the assembly.

I don't know if this helps, but you can run nunit-console.exe with an assembly parameter...not just .csproj file

IMO ApprovalTests should also be able to be run without the source-code folder, provided the approval files are available in a sensible location that can be discovered. This is what #71 and #124 attempt to achieve.

I'm using NUnit, SpecFlow and Bamboo (building on one machine and running tests on another) and it is all working as intended. Its a slightly different implementation from the one I provided in #124, but its the same technique of locating the approval files relative to the test assembly.

If you didn't want to add the approval files to your project file (so they can be copied to the build output folder at compile time), you could have a separate Bamboo task to copy them?

[jnsn]

Yeah, I should have clarified that. SpecFlow requires the .csproj file to generate an HTML report, not to run the test itself.

But I merely wanted to point out that, for me at least, it wouldn't be a problem if ApprovalTests would lay a dependency on the presence of the .csproj file.

[isidore]

Hey everyone. Any chance one or both of you could Skype tomorrow and we could fix this completely and push it?

Llewellyn Falco

On Jun 25, 2015, at 6:51 PM, Jensen notifications@github.com wrote:

Yeah, I should have clarified that.

SpecFlow requires the .csproj file to generate an HTML report, not to run the test itself.

—
Reply to this email directly or view it on GitHub.

[CarlVerret]

Hi,
I've been facing the same issue with a TFS build agent when trying to run (via Functional tests in TFS) the test runner can't find the *.approved files. Any ideas how to set the path to find them ?

Thanks

[danpowell88]

Did anything ever happen with this?

[davidalpert]

@danpowell88 @CarlVerret check out #219 - that fix worked for me to fix missing MSTest approval files in a Jenkins CIBuild

[SimonCropp]

given @davidalpert comment we can close this one.