Use GH CI to upload test output for any job that fails tests #2606
Conversation
This will make it much easier to investigate failures or update ref output when failures only happen on CI -- any test that fail tests will produce a downloadable "artifact" consisting of all tests' captured console output. Signed-off-by: Larry Gritz <lg@larrygritz.com>
|
Do we have any tests that output images? Or might we have any in the future? Maybe capture |
|
Most tests (other than a few utility unit tests) output images. But then they usually compare to a reference image and print the results. Lots of tests are printing metadata or hashes to verify images. Because of how large the artifact would be if I included all image output, and that my first important use of this was for text output (separate PR coming soon), I omitted it for now, thinking that I could add if/when we need it. But if you think we should just be proactive and bundle them all now, it's an easy change. Yeah, sure, that makes sense. I'll amend. I'm gonna add the same bits to OSL's CI as well, this is super useful, especially for those test differences that are acceptable differences, but just ever so slightly varying for each platform, so I can't quite get the same results on the machines I have direct access to. |
Signed-off-by: Larry Gritz <lg@larrygritz.com>
|
Amended: We now upload all outputs (including images), but only from the actual tests that fail. |
|
Sweet - will be very handy on the OSL side too. I was just thinking about the sampler change there and how to update the windows/mac ref images ... |
|
I'll submit the OSL version of this right away. |
…SoftwareFoundation#2606) This will make it much easier to investigate failures or update ref output when failures only happen on CI -- any test that fail tests will produce a downloadable "artifact" consisting of the outputs of all the failed tests. Signed-off-by: Larry Gritz <lg@larrygritz.com>
This will make it much easier to investigate failures or update ref
output when failures only happen on CI -- any test that fail tests
will produce a downloadable "artifact" consisting of all tests'
captured console output.
Signed-off-by: Larry Gritz lg@larrygritz.com