-
-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Regression testing script #34
Comments
The following script, which can be run after the above, generates a report on all rendering discrepancies found and saves it as a directory of lossless animated webp files: #!/usr/bin/bash -ex
rm -rf /tmp/c/
mkdir -p /tmp/c/
cd /tmp/a/
for png in *.png
do
if ! compare $png /tmp/b/$png -metric ae /tmp/diff.ppm
then
convert -delay 50 \
'(' $png /tmp/diff.ppm -append ')' \
'(' /tmp/b/$png /tmp/diff.ppm -append ')' \
-define 'webp:lossless=true' /tmp/c/$png.webp
fi
done See the example report below (converted to a gif for github) that was generated from the following sample regression: SafeNumeric<uint64_t> num_keep;
- for_range(p_from, p_to, true, SNAME("RenderCullInstances"), [&](const int i) {
+ for_range(p_from, p_to/2, true, SNAME("RenderCullInstances"), [&](const int i) {
KeepInstance keep_instance; |
This looks very interesting 🙂 Thanks for your work on this! I'll take a look in the future as I'm currently focused on PR reviews for the 4.1 release. |
On my system, rendering seems to be fully reproducible, down to producing the same exact byte sequence given the same input, as long as it's rendering in single threaded mode (multithreading mode can sometimes change imperceptibly between runs and may require a fuzzy tester).
The following script leverages this property to test if the working tree contains rendering regressions via checksums, which is immediately helpful for anyone working on Godot's rendering engine, and can be useful as a starting point for a proper visual regression testing system.
This may also be relevant to https://github.com/Calinou/godot-rendering-tests
Requires #33 in order to allow every benchmark to run for an exact number of frames
Requires godotengine/godot#72689 for environment variable support
The text was updated successfully, but these errors were encountered: