e2e-tests: Record an actual video of the test VM#1306
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1306 +/- ##
==========================================
+ Coverage 80.08% 86.44% +6.36%
==========================================
Files 20 99 +79
Lines 984 6685 +5701
Branches 0 111 +111
==========================================
+ Hits 788 5779 +4991
- Misses 196 850 +654
- Partials 0 56 +56 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
adombeck
force-pushed
the
e2e-test-vnc-recording
branch
10 times, most recently
from
41163c7 to
15987cf
Compare
YARF combines the screenshots it takes for image matching into a video. That misses everything that happens in between those screenshots. Let's record an actual video of the VNC output instead.
By default, VNC viewers can ask the VNC server for an exclusive connection. That would break YARF and the recording of the VNC output. Setting the share policy of the VNC server to "force-shared" disallows exclusive connections.
When using logger.error, the videos of all failing test cases are printed in the separate "Execution Errors" section of the HTML log file, without any context of which test case the video is from and the relevant error messages which caused the test case to fail. By using set_test_message, the video is printed as part of the error message of the test case.
We use the filename as the titles of the videos. They look better in uppercase.
Now that we use set_test_message instead of logger.error to log the video, we can't use the 'console=False' argument anymore which we added to Robot Framework. Work around that by filtering out the video from the output of run-tests.sh.
We log a video ourselves in a better way.
Sometimes the commit status is not created. Let's see if retrying helps.
The name was a bit confusing because steps starting with "Post" are usually from GitHub Actions using the 'runs.post' keyword to execute steps at the end of the job.
Should help us debug the cases where the commit status is not created.
Avoid creating an error annotation: Process completed with exit code <exit code> and instead create an error annotation with a link to the HTML logs.
Use a name that's consistent with the URL we use to publish the HTML log.
adombeck
force-pushed
the
e2e-test-vnc-recording
branch
from
15987cf to
7c4dbc2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
YARF combines the screenshots it takes for image matching into a video. That misses everything that happens in between those screenshots. Let's record an actual video of the VNC output instead.
Also log videos in the error message of the failing test case instead of the separate "Execution Errors" section.
UDENG-9379