-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add differential testing harness #216
base: main
Are you sure you want to change the base?
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
Perhaps better DX is to have directories and just |
@Firestar99 FYI, I am going to revamp this a bit as it is a bit of a mess. |
FWIW, I am almost ready to update this. It is much cleaner now :-) |
2b484a8
to
4408a6e
Compare
I think this is ready for review. Sorry it is so big! I can break it down if need be. The main changes:
|
This runs wgsl shaders and rust shaders and compares the output. If the output differs, the test fails. Differential testing is better than snapshot testing or golden file testing as there are no reference files to get outdated.
Another option we could do rather than binaries, is sort of what I did before (have built in harnesses which are configured) but mirror If I get my naga rust-gpu backend working, we could generate the rust-gpu code on the fly from the source |
The test harness compiles and runs Rust
shaders/kernels and compares the output with those produced by equivalent WGSL
shaders/kernels. Unlike traditional reference, golden, or snapshot tests—which check
against static outputs—differential testing compares multiple independent implementations of the same logic.
This approach is more robust because discrepancies between them are more likely to
indicate real bugs rather than issues with outdated reference files.
You can run difftests via
cargo difftest
. This is an alias set up in.cargo/config
for
cargo run --release -p difftest --
. You can filter to run specific tests bypassing the (partial) filenames to
cargo difftest some_file_name
.