A testing framework for automatically finding and simplifying bugs in graphics shader compilers.
Clone or download
Permalink
Failed to load latest commit information.
.idea/codeStyles Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
android-client-dep Initial commit. Sep 24, 2018
assembly-binaries Initial commit. Sep 24, 2018
assembly-public Initial commit. Sep 24, 2018
assembly Add releases to ci (#35) Oct 3, 2018
ast Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
build Generate and run (#44) Oct 5, 2018
checkstyle-config Added PSNR comparison metric to reducer tool and WebUI (#8) Sep 26, 2018
client-tests Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
common Set name of image job. (#64) Oct 12, 2018
compare-asts Various improvements to the reducer (#37) Oct 3, 2018
docs Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
fuzzerserver Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
generate-and-run-shaders Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
generator Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
parent-all Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
parent-checkstyle Fix unused or undeclared dependencies (#20) Sep 27, 2018
platforms/libgdx/OGLTesting Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
python Ignore potential encoding errors in ADB logcat (#63) Oct 12, 2018
reducer Fix skipRender during no image reductions. (#65) Oct 15, 2018
run-configurations Cleanup (#27) Oct 1, 2018
server-public Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
server-static-public Initial commit. Sep 24, 2018
server-thrift-gen Add maven-dependency-plugin:analyze-all (#12) Sep 26, 2018
server Improve webui (#56) Oct 12, 2018
shaders Initial commit. Sep 24, 2018
shadersets-util Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
temp Initial commit. Sep 24, 2018
test-util Initial commit. Sep 24, 2018
tester Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
third_party Add releases to ci (#35) Oct 3, 2018
thrift Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
tool Generate and run (#44) Oct 5, 2018
util Squashed ShaderJobFile refactor. (#59) Oct 12, 2018
.editorconfig Initial commit. Sep 24, 2018
.gitattributes Initial commit. Sep 24, 2018
.gitignore Add releases to ci (#35) Oct 3, 2018
.travis.yml Add releases to ci (#35) Oct 3, 2018
AUTHORS Initial commit. Sep 24, 2018
CODEOWNERS Add CODEOWNERS file. (#4) Sep 25, 2018
CONTRIBUTING.md Initial commit. Sep 24, 2018
CONTRIBUTORS Added Imperial contributors, which includes Donaldson, Evrard and Tho… Oct 5, 2018
LICENSE Initial commit. Sep 24, 2018
README.md Initial commit. Sep 24, 2018
envbuild.py.template Initial commit. Sep 24, 2018
pom.xml Squashed ShaderJobFile refactor. (#59) Oct 12, 2018

README.md

Warning: this repository is a work-in-progress. Things may break while we transition this project to open source. This is not an officially supported Google product.

GraphicsFuzz

License

Introduction

GraphicsFuzz is a testing framework for automatically finding and simplifying bugs in graphics shader compilers. Our tools currently manipulate GLSL shaders, but we can indirectly test other targets such as SPIR-V, HLSL and Metal. Our current priority is testing Vulkan drivers.

The problem

A graphics driver takes a shader program as input and executes it on a GPU (graphics processing unit) to render an image.

shader program, to GPU, to image

Compiling and executing shaders is complex, and many graphics drivers are unreliable: a valid shader can lead to wrong images, driver errors or even security issues.

shader program, to GPU, to crash

Automatically finding bugs

We start with a reference shader that renders an image. The reference shader can be any shader you like, such as a high-value shader from a game or existing test suite.

reference, to GPU, to image

Shaders are programs, so by applying semantics-preserving source code transformations, we can obtain a shader with significantly different source code that still has the same effect.

transformation example: wrapping a statement in a do-while-false loop

For example, wrapping code in a single-iteration loop does not change the meaning (semantics) of a program. By applying various semantics-preserving transformations to the reference shader, we generate a family of variant shaders, where each variant must render the same image as the reference.

reference and variants, to GPU, to many equivalent images

If a variant shader leads to a seriously different image (or a driver error), then we have found a graphics driver bug!

reference and one variant, to GPU, to two different images

This approach is known as metamorphic testing.

Reduction

Finding bugs is not the end of the story: a variant shader that exposes a bug is typically very large (thousands of lines), full of code coming from the semantics-preserving transformations. Typically only a fraction of this code is needed to expose the bug.

Source code, the majority of which is highlighted in yellow, but parts of one statement are not highlighted.

Fortunately, our reducer is able to selectively reverse those transformations that are not relevant to the bug. After reduction, we obtain a small difference sufficient to expose the driver issue.

The same source code, the majority of which is highlighted in yellow and striked out, but parts of one statement remain.

The reduced variant still exposes the bug, and differs from the reference only slightly: this is a great starting point to isolate the root cause of the bug in the graphics driver.

Summary

GraphicsFuzz finds bugs in graphics drivers by rendering families of semantically equivalent shaders, and looking for output discrepancies. This approach is known as metamorphic testing. For each bug, the reducer saves a lot of debugging time by producing a simpler minimal-difference test case that still exposes the bug.

Further reading

GraphicsFuzz blog posts:

Academic research project blog posts:

Academic publications: