Snapshot recap

[novemberborn]

A summary of snapshot related issues. See #1218, #1220, #1254 and #1223.

---

Fundamentally, jest-snapshot is string based. The snapshots contain human-readable representations of expected values, which are then compared against another representation of the actual value when tests are run.

Snapshot testing is useful when code generates complex tree structures that are tedious to verify by writing assertions, and that may change significantly as part of day-to-day development. Humans manually verify that the snapshot is as expected the first time it's generated. To do this, the snapshot needs to be readable. Then, when the actual value changes and the snapshot is regenerated, source control diffing makes it easy to see what's actually changed and if that is as expected.

The original snapshot PR (#1113) used jest-snapshot as-is (thanks @lithin!). Here, jest-snapshot was also responsible for the error message:

expect(value).toMatchSnapshot()

Received value does not match stored snapshot 1.

- Snapshot
+ Received

Object {
-   "foo": "bar",
+   "foo": "bar!",
  }

Soon after this landed @vadimdemedes started working on improving our error output, specifically showing diffs for t.deepEqual() and t.is() assertions (#1154). This work also included t.snapshot(). However, to generate these diffs we need actual objects, not the error message that jest-snapshot returns. (Plus, that expect(value).toMatchSnapshot() line is a bit jarring.)

The solution was to send a JSON string into jest-snapshot, with a workaround for React trees. This means though that the snapshot contains a fairly unreadable JSON string (#1254). Also, whereas jest-snapshot normalizes key order in Object objects, AVA can no longer rely on that because we're now just comparing JSON strings. Then when we generate a diff it comes up empty since our diffing logic does not care about key order (#1220).

@vadimdemedes started a PR (#1223) to replace jest-snapshot with our own implementation, using t.deepEqual() for the comparisons. The advantage of this approach is that diffs look the same for all our (diffing) assertions. Unfortunately, because it uses JSON serialization it limits the kinds of object structures we can safely snapshot:

• undefined, Symbol() and function property values are dropped
• undefined, Symbol() and function array items are replaced by null
• regular expression property values and array items are replaced by {}
• no support for built-ins such as Map and Set
• no support for custom classes: instances are reduced to plain objects
• we're effectively using toJSON() for comparisons, which we don't do for t.deepEqual()

We need to decide whether we want error output for t.snapshot() to be the same as that for an equivalent error from t.deepEqual(). I don't think we can do this while still using jest-snapshot without hurting snapshot readability.

The current JSON proposal severely limits the kinds of snapshots that can be created. This should only be a short-term solution, but as this issue establishes we have some critical issues with our current snapshot implementation right now, so I'd rather we ship something that works real soon. We can then expand the functionality going forward.

If we go this route we should enforce that snapshots cannot contain any values incompatible with JSON serialization. We can do this when adding a snapshot by comparing the snapshot against the value being added.

On the other hand using jest-snapshot provides more immediate user value. We could go that route and try and improve the diff output later. At that point though a reduction in functionality would not be acceptable.

If you've made is this far I thank you for your dedication. Please respond with your feedback 😄

[vadimdemedes]

@novemberborn Thanks for a detailed writeup, very helpful!

I think I'd want to go with a new solution to snapshot testing and try to find a better way to serialize snapshot values. We need to look for a library that handles serializing/deserializing of data and which also includes types that JSON ignores. Any tips are welcome!

I think I've seen some time ago something like an extended JSON serializer, which handled most of its limitations. Need to look for it.

The reasons why I'd go without jest-snapshot are:

1. It brings unnecessary and unused dependencies, such as jest-diff and jest-matcher-utils. We aren't using jest-snapshot's error message, so the diffs provided are useless for us.
2. We can't pretty-print or diff the values and display the same rich output other assertions provide.
3. It's not as open as I'd wish it could be for other JSX-based libraries, like Preact, Inferno or others. See Snapshot is doubly-escaped / hard to see diffs #1254.

Unfortunately, because it uses JSON serialization it limits the kinds of object structures we can safely snapshot:
no support for custom classes: instances are reduced to plain objects

I don't think there's any possibility for that to work with any serialization/deserialization method, is there? We just can't "recreate" non-native (Map, Set, etc - language features) instances.

As for key order, I'm not even sure we'd want to check that. In what cases key order of objects/props (in React components) affects the output/functionality? I can't think of any at this moment, because I've never encountered such a problem.

[novemberborn]

The reasons why I'd go without jest-snapshot are:

It brings unnecessary and unused dependencies, such as jest-diff and jest-matcher-utils. We aren't using jest-snapshot's error message, so the diffs provided are useless for us.

According to cost-of-modules:

name	children	size
chokidar	168	8.71M
babel-core	27	3.11M
empower-core	2	1.92M
bluebird	0	0.58M
source-map-support	0	0.55M
meow	26	0.40M
jest-snapshot	7	0.36M
update-notifier	37	0.34M
@ava/babel-preset-stage-4	29	0.33M
diff	0	0.32M

So yes there are dependencies but it's not our worst offender.

We can't pretty-print or diff the values and display the same rich output other assertions provide.

Yes, but what should be the short-term priority?

It's not as open as I'd wish it could be for other JSX-based libraries, like Preact, Inferno or others.

pretty-format supports other serializers, so I think it's actually pretty open. We'd have to solve configuration but we need to do that even for a custom solution.

no support for custom classes: instances are reduced to plain objects

I don't think there's any possibility for that to work with any serialization/deserialization method, is there? We just can't "recreate" non-native (Map, Set, etc - language features) instances.

Nah, you'd just go "the snapshot says it was a Foo, and this object also says it's a Foo, so they match".

As for key order, I'm not even sure we'd want to check that. In what cases key order of objects/props (in React components) affects the output/functionality? I can't think of any at this moment, because I've never encountered such a problem.

jest-snapshot isn't sensitive to key order. We currently are because we pass it a JSON string.

[novemberborn]

However, to generate these diffs we need actual objects, not the error message that jest-snapshot returns.

This isn't entirely accurate. The diffs are done using formatted objects. jest-snapshot stores and compares formatted objects. It's just that we cannot configure how it formats objects. This means we can't do code highlighting, and there'll be other differences in the snapshot diff compared to the diff we generate for t.deepEqual().

I have a longer term plan that harmonizes t.deepEqual() with t.snapshot(), but for now I think it's fine if we use jest-snapshot as it was introduced in #1113. At least it'll work again!

[novemberborn]

Well, we've built our own, which has landed in #1341. A new release will be out soon.