Skip to content

Improve reference tests#2742

Merged
197g merged 7 commits intoimage-rs:mainfrom
fintelia:improve-reference-tests
Jan 24, 2026
Merged

Improve reference tests#2742
197g merged 7 commits intoimage-rs:mainfrom
fintelia:improve-reference-tests

Conversation

@fintelia
Copy link
Contributor

This PR uses the libtest-mimic and walkdir crates to simplify/improve our reference tests. It recursively walks the entire directory tree under the reference images directory to find tests, and tests that cannot be run because of a disabled feature are now marked as "ignored" rather than silently not run.

I removed the "render_images" test because I didn't see what purpose it was serving that we didn't get by just having the normal tests write to tests/output/ with the observed contents of any image that didn't match the reference.

Also corrects several reference images because due to a bug in the old implementation we were apparently not testing the images under tests/reference/jpg...

Old output

running 2 tests
test check_references ... ok
test render_images ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.15s

New output

     Running tests/reference_images.rs (target/debug/deps/reference_images-6fce3ac7ddc64c75)

running 135 tests
test reference tests/images/bmp/images/Info_X1_R5_G5_B5.bmp                              ... ok
test reference tests/images/bmp/images/V3_R5_G6_B5.bmp                                   ... ok
test reference tests/images/bmp/images/V3_X8_R8_G8_B8.bmp                                ... ok
test reference tests/images/bmp/images/Info_8_Bit_Top_Down.bmp                           ... ok
test reference tests/images/bmp/images/V3_X4_R4_G4_B4_Top_Down.bmp                       ... ok
test reference tests/images/bmp/images/Info_A8_R8_G8_B8_Top_Down.bmp                     ... ok
test reference tests/images/bmp/images/Info_1_Bit.bmp                                    ... ok
test reference tests/images/bmp/images/V3_R5_G6_B5_Top_Down.bmp                          ... ok
test reference tests/images/hdr/images/rgbr4x4.hdr                                       ... ok
test reference tests/images/jpg/progressive/test.jpg                                     ... ok
test reference tests/images/bmp/images/pal8v4.bmp                                        ... ok
test reference tests/images/bmp/images/Info_1_Bit_Top_Down.bmp                           ... ok
[...]
test reference tests/images/png/apng/ball.png[15]                                        ... ok
test reference tests/images/png/apng/ball.png[1]                                         ... ok
test reference tests/images/png/apng/ball.png[13]                                        ... ok
test reference tests/images/png/apng/ball.png[7]                                         ... ok
test reference tests/images/png/apng/ball.png[17]                                        ... ok
test reference tests/images/png/apng/ball.png[8]                                         ... ok
test reference tests/images/webp/lossless_images/multi-color.webp                        ... ok
test reference tests/images/png/bugfixes/debug_triangle_corners_widescreen.png           ... ok
test reference tests/images/jpg/progressive/3.jpg                                        ... ok

test result: ok. 135 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.11s

@197g
Copy link
Member

197g commented Jan 23, 2026

I like it except we still don't know how many tests to expect to be ran here. If the files are missing as from a crate then, if I read this correctly, silently no tests are generated or ran. Correct?

@fintelia
Copy link
Contributor Author

Yes, though in the specific case of running from a .crate it won't matter because our Cargo.toml excludes everything under tests/ which means that tests/reference_images.rs isn't captured either...

The other case that isn't handled is files that are present under tests/images but not under tests/reference. There's currently a couple dozen such images that are used by other tests though, so we can't just treat them as failing tests. I suppose that listing them as ignored would be an option?

Copy link
Member

@197g 197g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright let's worry about packaging at a later time.

@197g 197g merged commit d347e4d into image-rs:main Jan 24, 2026
32 checks passed
@fintelia fintelia deleted the improve-reference-tests branch January 25, 2026 01:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants