Post failure diagnostic hook

[danmoseley]

I have a set of tests for a system that very sporadically fails on the CI machines, so it's difficult to debug directly. When there's a failure in one of the relevant tests, I'd like to gather some diagnostic information.

Is there a way to define an "on failure" hook for a test fixture, that would automatically be run if any test fails? As far as I can tell there isn't, so and I have to put try/finally around the body of each test. Is that right?

[bradwilson]

There is no such feature today.

[danmoseley]

@bradwilson I wonder whether this would be worth considering for 3.0. Examples of info to gather could be the tail of the system log, any diagnostic file that the app under test might write, etc. Eg., some of our crypto tests on Mac are interested in the content of the Mac cert store if they fail.

[bradwilson]

@danmoseley I think this will be somewhat enabled by the TestContext feature #621 as you'll be able to get access to information during Dispose like whether the test failed and how. If you were to design the ideal feature for your usage, what would it look like?

[danmoseley]

Interesting. That would certainly help. In our test libraries, tend to have fairly similar responsibilities (eg, a test library for Crypto) and so it's likely that many times, if any test in the library fails, you want to do essentially the same diagnostics, and only once for all the tests in the library. This is why I was thinking of an assembly level attribute designating an "On fail" method.

It sounds like TestContext would make that possible, if I add a Dispose to all my fixtures, and each calls into a utility method that skips after the first run.

@bartonjs you mentioned recently it would be nice to have a way to dump the state of the Mac certificate store when a relevant test failed. Would it suffice if you could define a Dispose of each relevant test fixture (class) that could access (via an ITestContextAccessor parameter passed to the fixture constructor -> TestContext object) the state of the test that just ran, ie whether it failed and any exception it threw?

That seems like what we would need, although I had in mind an assembly level hook, this is more code to write, but more powerful.

[bartonjs]

Well, it would require that we change our static methods into instance methods to participate. I don't remember from what context I was wistful about dumping the cert store, but I can come up with some hypotheticals.

I agree that my design would likely have been a static method with an attribute. Using Dispose here feels like everything's a nail, but from a functional flow (vs purpose-flow) it feels OK (heck, I'm the one who pushed for the updated guidance that we allow disposable objects to represent pairwise work (e.g. push/pop) instead of just resource cleanup... so this is just extending my own wedge).

Short summary of long winded response: It does feel like it solves the scenario, albeit a bit indirectly.

[danmoseley]

@bradwilson would it be reasonable to have an assembly level hook, ie., an assembly level attribute (singleton) that defines a static method somewhere in the assembly that is called after each test failure (say, after any Dispose completes, but still with TestEngineStatus.CleaningUp) and passes in a ITestContextAccessor?

That way folks could decide whether "near the test" is most applicable (eg., it's specific to a fixture or a few fixtures), or "global" (eg., it specific to many or all fixtures or you don't want to change to instance method tests). And in the "global" case they would have freedom to perform activities once or many times and for all tests or some. In both cases, output could go to the ITestOutputHelper on the TestContext. In our case, we might write it to a file in a location we know that CI will archive.

[sharwell]

One of the diagnostics we need is a screenshot at the point of failure¹. Since the Dispose method of the test itself changes the visual state significantly (restores application to a known good state), we need to be able to capture this diagnostic as close to the point it occurs as possible:

• Capture at the point of First Chance Exception derived from XunitException
• Capture when a test method throws an unhandled exception not derived from XunitException, prior to other cleanup

Both of the above need to attach the screenshot(s) as part of the test result.

We have other logs to capture as well, but they are less prone to timing issues.

¹ Historically, this is the single most actionable piece of diagnostic information in these tests.