Audio Snapshot Testing
What is snapshot testing?
Snapshot tests compare the output of a previous known to be good version of a system with the output of the current version to protect against unintended changes.
Also known as golden master testing or characterization testing.
What can be snapshot tested?
- DOM tree rendered by a React component
- Image generated on an HTML
- Generated binary data in an
- Screenshot of a browser window or mobile app
- Sound generated by Web Audio API
The snapshot testing workflow
Set up the first snapshot
- Write a working implementation
- Save the output into a snapshot file
Make some changes to the implementation
- Compare the new output with the old one
- If the outputs are different, manually verify the new output (listen to it)
- If the new output sounds right (a change was intended), overwrite the saved snapshots
- Otherwise revert or fix the breaking changes
Practical Audio Snapshot testing
The problem: snapshot testing frameworks exist but no existing test framework supports audio snapshot testing.
- Take an audio snapshot
- Allow saving the snapshot to a file
- Load the snapshot
- Compare snapshots for difference
- Manually verify snapshots
Proof of concept audio snapshot testing
This repo contains the basics needed for audio snapshot testing. It is not a fully elaborate framework, rather a draft implementation of an idea.
Verifying audio data is not as easy as precisely comparing sample values. For example there might be small differences in how browsers render web audio, which can be tolerated for the given use case.
It is also important to note that AudioContext and OfflineAudioContext are not exactly the same, using the latter for testing might be problematic especially when testing real-time operations on audio nodes.