tests(smoke): print multiple differences #14141
Conversation
| found: ${JSON.stringify(diff.actual)} | ||
| found: ${JSON.stringify(diff.actual)}\n`; | ||
| localConsole.log(msg); | ||
| } |
There was a problem hiding this comment.
praying for String.dedent
There was a problem hiding this comment.
first I'd like to see exactly what the smoke results might look like when too noisy
two examples I can think of:
- eliminate an entry in an array, e.g. the first
LinkElementsentry in dbw - rename an object, e.g. rename the dbw
LinkElementsentry toConsoleMessages
both are quite noisy.
That said, I tried this PR out a bit and it seems like it will still be worth it. The fact that expectations are usually limited in size and they're what drive the comparisons means that often even excessive errors aren't that bad (LinkElements and ConsoleMessages also happen to both be arrays, rename LinkElements to URL (which is an object) instead and the diff list is considerably smaller).
| @@ -427,19 +425,20 @@ function reportAssertion(localConsole, assertion) { | |||
| log.greenify(assertion.actual)); | |||
| } | |||
| } else { | |||
| if (assertion.diff) { | |||
| const diff = assertion.diff; | |||
| const fullActual = String(JSON.stringify(assertion.actual, null, 2)) | |||
There was a problem hiding this comment.
little did you know this was a load-bearing String() (try renaming LinkElements to LinkElementz or whatever :)
Though I certainly didn't remember why it was there, so we can change it to something more obvious that does the same thing
There was a problem hiding this comment.
Made it more explicit and added tests.
connorjclark
changed the title
Fixes #14128
Brendan mentioned here that presenting all the differences in a smoke test may be noisy. This PR doesn't take any of that into account (I did first try to limit the amount of differences as determined here, but that seems to reduce to the same behavior we have today), first I'd like to see exactly what the smoke results might look like when too noisy. @brendankenny can you think of a good way to test this?