Apply test output from timedout tests on non-threadabort platforms #4692
Conversation
|
|
||
| separateContext.CurrentResult.ApplyOutput(context.CurrentResult); | ||
|
|
||
| if (timedOut && !_debugger.IsAttached) |
There was a problem hiding this comment.
Note to reviewer: I had to move the Task.WaitAny to it's own line in order to apply the test output afterwards. I ended up inverting this check here as part of that as I found it read clearer this way but I'm happy to instead change it back to something like:
if (!timedOut || _debugger.IsAttached)
|
|
||
| separateContext.CurrentResult.ApplyOutput(context.CurrentResult); |
There was a problem hiding this comment.
Should this not be the other way around?
You want the separateContext to go to the context not the other way around?
There was a problem hiding this comment.
From memory (which may be flawed) but I think I saw in the debugger that these were getting written to the original context but that it was getting overridden when we create the testResult from the separateContext (which is what would then get output).
I'll load this up and rerun it soon to confirm
There was a problem hiding this comment.
I see the method wording here could be confusing too.
Leaving desired functionally aside for a moment, if the intention of this code were to take the output from separateContext and apply it to context perhaps the method name would be clearer if it were ApplyOutputTo?
There was a problem hiding this comment.
CopyOutputTo would be clearer. At the moment unless I got it wrong it does a CopyOutputFrom
There was a problem hiding this comment.
Of course, thanks! That's also more consistent with some BCL methods like Stream.CopyTo()
|
|
||
| Assert.That(testMethod.Output, Does.Contain("setup")); | ||
| Assert.That(testMethod.Output, Does.Contain("method output before pause")); | ||
| Assert.That(testMethod.Output, Does.Not.Contain("method output after pause")); |
There was a problem hiding this comment.
Under .Net core that data will be written after the test has timed out and is likely not available.
There was a problem hiding this comment.
I think that's the intended behaviour, right? The Does.Not here ensures that any output written after the test times out isn't captured
There was a problem hiding this comment.
It is not the intended behaviour, but the current behaviour on net 6.0`
There was a problem hiding this comment.
I believe it's also the current behavior on other runtimes like netfx. Changing that would be a bigger body of work than I was looking to do in this PR, and one which we may wish to discuss further with the team in a separate issue.
If we were to keep this PR focused on just resuming outputting of what is executed, would you prefer we leave this assertion in place knowing it may change if the behavior does, or should we remove the assertion?
|
|
||
| [TearDown] | ||
| public void TearDown() | ||
| { |
There was a problem hiding this comment.
The #4598 issue talks about TearDown output being lost. We need a test with that in.
There was a problem hiding this comment.
Good point! I had changed the test from having a TearDown to a SetUp during development and missed re-adding a separate TearDown case. I'll add one.
I think the originating issue sought more to keep output from the test method itself in the presence of a TearDown rather than output from the TearDown itself but that will be a good test for me to add here to. One thing to note here though is that I think this was a NET6-specific bug and that presently TearDown is presently only run for timedout test cases on netfx (#2397). I verified the behaviour described there and it's consistent as far back as 3.12
There was a problem hiding this comment.
Wait, sorry. @manfred-brands I think this does cover the example in the issue as it talks about test output from the test rather than the TearDown, but I'll still take another look and see if there's any edge cases I could add
|
@manfred-brands I've been able to get back to coding this. I've pushed the naming change to adjust To verify the 3.x functionality I'm trying to restore here, I've put up a branch with the same tests on the 3.x branch. CI is still verifying but they pass as-is locally for me. I'm a little unclear on if there's further changes you'd like to see here, can you please take a look and let me know? |
Fixes #4598
@manfred-brands You mentioned in the source issue that you may also be looking into this on your own branch.
I've put up this PR as I was able to get this working in a relatively self-contained way for the existing code which may be able to merge either independent of your larger rework or could perhaps be PR'd into it as well. This PR could also be discarded if you've already gotten it working on your end too