Skip to content

add diffs back when errors happen during testing#42

Closed
jakecoffman wants to merge 3 commits intomainfrom
jakecoffman/add-diffs-back
Closed

add diffs back when errors happen during testing#42
jakecoffman wants to merge 3 commits intomainfrom
jakecoffman/add-diffs-back

Conversation

@jakecoffman
Copy link
Member

Following up #40 this adds output of a diff between expected and actual when there is a missed assertion of the output.

I went with a unified-diff style diff this time.

To keep an excessive amount of output from hitting the terminal I added explicit checks to the content of the DependencyFiles, so it will simply say there's no match there. In our smoke tests we still have a "Diff" step which will reveal the problem, and locally we can run with -o to output the actual results for diffing if needed.

I'll probably run this locally for a little while to make sure it's good and maybe add a test so I'll just put this up as draft for now.

@jakecoffman
Copy link
Member Author

We've been getting along nicely without this and relying on the diff of the output, so I think I'll just close this for now.

@jakecoffman jakecoffman closed this Feb 2, 2023
@jakecoffman jakecoffman deleted the jakecoffman/add-diffs-back branch February 2, 2023 16:06
@jeffwidman
Copy link
Member

Thanks for #58, it def felt like we needed some sort of middle ground between super-noisy-diff and none at all...

One other thought is that if there's a problem, adding a hint to the log saying "to see the diff, do X" because if we start recommending this in place of dry-run to drive-by users, we're going to get lots of questions...

@jakecoffman
Copy link
Member Author

Maybe when there's an error while running in test mode, and -o is not specified, it automatically creates err.yaml with the output for diffing with the user's tool of choice.

@jeffwidman
Copy link
Member

Personally, that feels too clever for me... just a simple "if you want to see the diff, run with --output <file name> and compare the actual output to the expected output" hint is all I really need...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants