[9.x] Improve test failure output#43943
Conversation
aa14cc5 to
288583b
Compare
jbrooksuk
left a comment
There was a problem hiding this comment.
@jessarcher you may have put EOF where it is for indentation, so feel free to ignore this.
|
@nunomaduro I'd appreciate your thoughts on this one. Do you see any issues hooking into Would this feature also be beneficial for Pest or can you think of any issues it might cause? |
|
@jessarcher marking as draft until we get feedback from @nunomaduro |
93cedbe to
1370d3e
Compare
| : implode(PHP_EOL, Arr::flatten($errors)); | ||
|
|
||
| // JSON error messages may already contain the errors, so we shouldn't duplicate them. | ||
| if (str_contains($exception->getMessage(), $errors)) { |
There was a problem hiding this comment.
Does this works? Asking because it seems we change the format of the JSON string by using the JSON_PRETTY_PRINT flag.
There was a problem hiding this comment.
Yeah - the existing output in the exception is also pretty printed
|
@jessarcher I've rebased, added minor changes, and made a few comments. Generally, this pull request seems OK to me. In addition, I've tested this pull request against Vapor test suite - more than 600 test cases - and seems to be all working. |
In #38025 and #38046, we improved the test output for the
assertStatusmethods to display any exceptions or errors that may have caused the status assertion to fail.These improvements almost removed the need to call
withoutExceptionHandling. Almost...This PR expands on the previous work to display exceptions or errors on any failed assertion, not just status assertions, and not even just
TestResponseassertions.A common example is when asserting Inertia redirects, and an unexpected validation error occurs. The current output doesn't give any clues that the redirect assertion failure was due to a validation error:
Under the hood, this calls
assertStatusandassertLocation, but it's only theassertLocationthat fails, which doesn't currently display exceptions or errors.With the changes in this PR, the error now looks like this:
The existing
assertStatusbehaviour is now handled by this new logic, so there's no duplication of code. There is a minor change in output because the extra content is now appended after PHPUnit's raw assertion error instead of before it:Before:

After:

We'll now also see this extra context even if we don't assert against the TestResponse. For example, imagine this test where we expect a user record to be created:
If a validation error or exception occurred during the last request, we'll get the extra context on PHPUnit's built-in assertions like
assertSame:The only scenario I can think of where this feature is undesirable is the
assertJsonassertion. If a validation error occurs, theassertJsonassertion will already show the full response body. To prevent duplicate output, I've ensured that we don't append the JSON error response if the existing message already includes it.If the user makes multiple requests in one test, only errors/exceptions from the latest request will be displayed, hopefully preventing red herrings.
The extra context will only be displayed when an assertion fails, so you won't see any output if you expect an error or exception as long as your assertions pass.
Developers may opt-out of this feature by overriding the
onNotSuccessfulTestmethod in their base test case to throw the given exception like PHPUnit does by default.