Make it easier to integration test config options

[benjaminjkraft]

Our integration tests all run with a single config, which makes it hard to integration test a change like #252. Ideally we should do some reworking so we can test them against several configs, like we do in the snapshot tests. We'll have to think about what the best way is.

Once we do, we should add integration tests for #252, which should be fairly easy once the tooling is there.