Java Snapshot Testing

• Inspired by facebook's Jest framework

🎉 3.X is live - parallel test support, easily override configuration via snapshot.properties yet *.snap format remains unchanged! This release will require some mechanical refactoring for those upgrading as expect is no longer a static method.

Upgrade guide from 2.X to 3.X here

The testing framework loved by lazy productive devs

• Tired of needing to assertThat(foo).isEqualTo("bar") again & again?
• Are you just wanting to ensure you don't break - for example - REST interfaces
• Are you manually saving text files for verification in your tests?

Want a better way? Then java-snapshot-testing might just be what you are looking for!

Quick Start (Junit5 + Gradle example)

1. Add test dependencies

// In this case we are using the JUnit5 testing framework
testImplementation 'io.github.origin-energy:java-snapshot-testing-junit5:3.2.+'

// Many will want to serialize into JSON.  In this case you should also add the Jackson plugin
testImplementation 'io.github.origin-energy:java-snapshot-testing-plugin-jackson:3.2.+'
testImplementation 'com.fasterxml.jackson.core:jackson-core:2.11.3'
testImplementation 'com.fasterxml.jackson.core:jackson-databind:2.11.3'
testImplementation 'com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.11.3'
testImplementation 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.11.3'

// slf4j logging implementation if you don't already have one
testImplementation("org.slf4j:slf4j-simple:2.0.0-alpha0")

2. Create snapshot.properties and configure your global settings. Be sure to set output-dir appropriately for your JVM language.

• /src/test/resources/snapshot.properties

serializer=au.com.origin.snapshots.serializers.ToStringSnapshotSerializer
serializer.base64=au.com.origin.snapshots.serializers.Base64SnapshotSerializer
serializer.json=au.com.origin.snapshots.serializers.JacksonSnapshotSerializer
serializer.orderedJson=au.com.origin.snapshots.serializers.DeterministicJacksonSnapshotSerializer
comparator=au.com.origin.snapshots.comparators.PlainTextEqualsComparator
reporters=au.com.origin.snapshots.reporters.PlainTextSnapshotReporter
snapshot-dir=__snapshots__
output-dir=src/test/java
ci-env-var=CI

3. Enable snapshot testing and write your first test

package au.com.origin.snapshots.docs;

import au.com.origin.snapshots.annotations.SnapshotName;
import au.com.origin.snapshots.junit5.SnapshotExtension;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;

import java.util.HashMap;
import java.util.Map;
import au.com.origin.snapshots.Expect;

@ExtendWith({SnapshotExtension.class})
public class MyFirstSnapshotTest {

  private Expect expect;

  @SnapshotName("i_can_give_custom_names_to_my_snapshots")
  @Test
  public void toStringSerializationTest() {
    expect.toMatchSnapshot("Hello World");
  }

  @Test
  public void jsonSerializationTest() {
    Map<String, Object> map = new HashMap<>();
    map.put("name", "John Doe");
    map.put("age", 40);

    expect
        .serializer("json")
        .toMatchSnapshot(map);
  }

}

4. Run your test

Bingo - you should now see your snapshot in the __snapshots__ folder created next to your test. Try changing "Hello World" to "Hello Universe" and watch it fail with a .debug file.

au.com.origin.snapshots.docs.MyFirstSnapshotTest.jsonSerializationTest=[
  {
    "age": 40,
    "name": "John Doe"
  }
]


i_can_give_custom_names_to_my_snapshots=[
Hello World
]

Advantages of Snapshot Testing

• Great for testing JSON interfaces ensuring you don't break clients
• Fast and easy to test
• Will implicitly test areas of your code you did not think about
• Great of testing dynamic objects

You're responsible for making sure your generated snapshots do not include platform specific or other non-deterministic data.

Disadvantages of Snapshot Testing

• You need to ensure your test is deterministic for all fields (there are ways to ignore things like dates)
• Does not give great insight to why the snapshot failed
• Can be difficult to troll though large snapshot changes where you might only be interested in a small set of fields

Installation Maven

These docs are for the latest -SNAPSHOT version published to maven central. Select the tag X.X.X matching your maven dependency to get correct documentation for your version.

Only if you want to integrate with an unsupported framework. Show me how!

• Core

We currently support:

• JUnit4
• JUnit5
• Spock

Plugins

• Jackson for JSON serialization
  • You need jackson on your classpath (Gradle example)

        // Required java-snapshot-testing peer dependencies
       testImplementation 'com.fasterxml.jackson.core:jackson-core:2.11.3'
       testImplementation 'com.fasterxml.jackson.core:jackson-databind:2.11.3'
       testImplementation 'com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.11.3'
       testImplementation 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.11.3'

How does it work?

1. When a test runs for the first time, a .snap file is created in a __snapshots__ sub-directory
2. On subsequent test runs, the .snap file is compared with the one produced by the test
3. If they don't match, the test fails and a .snap.debug with the conflict is created
4. It is then your job to decide if you have introduced a regression or intentionally changed the output (Use your IDE file comparison tools to compare the two files or refer to the terminal output)
5. If you have introduced a regression you will need to fix your code
6. If you have intentionally changed the output you can manually modify the .snap file to make it pass or delete it and it will be generated again from scratch
7. Once you fix the test, the *.snap.debug file will get deleted

What is a Snapshot?

A text representation of your java object (toString() or JSON).

String snapshot example

expect("hello world","Hello world again!").toMatchSnapshot();

au.com.example.company.HelloWorldTest.helloWorld=[
Hello world
Hello world again!
]

JSON Snapshot Example

expect(userDto).serializer("json").toMatchSnapshot();

au.com.example.company.UserEndpointTest.shouldReturnCustomerData=[
  {
    "id": "1",
    "firstName": "John",
    "lastName": "Smith",
    "age": 34
  }
]

Usage Examples

All frameworks allow injection of the Expect expect via instance variable or method argument. In cases where parameterised tests are used, it's often better to use an instance variable in order to avoid conflicts with the underlying data table.

JUnit 5

package au.com.origin.snapshots.docs;

import au.com.origin.snapshots.junit5.SnapshotExtension;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import au.com.origin.snapshots.Expect;

// Ensure you extend your test class with the SnapshotExtension
@ExtendWith({SnapshotExtension.class})
public class JUnit5Example {

  // Option 1: inject Expect as an instance variable
  private Expect expect;

  @Test
  public void myTest1() {
    // Verify your snapshot
    expect.toMatchSnapshot("Hello World");
  }

  // Option 2: inject Expect into the method signature
  @Test
  public void myTest2(Expect expect) {
    expect.toMatchSnapshot("Hello World Again");
  }
}

JUnit 4

package au.com.origin.snapshots.docs;

import au.com.origin.snapshots.annotations.SnapshotName;
import au.com.origin.snapshots.junit4.SnapshotRunner;
import au.com.origin.snapshots.Expect;
import org.junit.Test;
import org.junit.runner.RunWith;

// Ensure you RunWith the SnapshotRunner
@RunWith(SnapshotRunner.class)
public class JUnit4Example {

  // Option 1: inject Expect as an instance variable
  private Expect expect;

  @SnapshotName("my first test")
  @Test
  public void myTest1() {
    // Verify your snapshot
    expect.toMatchSnapshot("Hello World");
  }

  @SnapshotName("my second test")
  @Test
  // Option 2: inject Expect into the method signature
  public void myTest2(Expect expect) {
    expect.toMatchSnapshot("Hello World Again");
  }
}

Spock

package au.com.origin.snapshots.docs

import au.com.origin.snapshots.annotations.SnapshotName
import au.com.origin.snapshots.spock.EnableSnapshots
import spock.lang.Specification

import au.com.origin.snapshots.Expect

// Ensure you enable snapshot testing support
@EnableSnapshots
class SpockExample extends Specification {

    // Option 1: inject Expect as an instance variable
    private Expect expect

    // With spock tests you should always use @SnapshotName - otherwise they become coupled to test order
    @SnapshotName("should_use_extension")
    def "Should use extension"() {
        when:
        expect.toMatchSnapshot("Hello World")

        then:
        true
    }

    @SnapshotName("should_use_extension_as_mehod_argument")
    // Option 2: inject Expect into the method signature
    def "Should use extension as method argument"(Expect expect) {
        when:
        expect.toMatchSnapshot("Hello World")

        then:
        true
    }
}

Using an unsupported framework

This library is in no way restricted to JUnit4, Junit5 or Spock.

Any framework can support the library as long as it follows the following rules:

1. Before all the tests in a single file execute (once only)

       SnapshotVerifier snapshotVerifier = new SnapshotVerifier(new YourFrameworkSnapshotConfig(), testClass, failOnOrphans);

2. After all the tests in a single file execute (once only)

     snapshotVerifier.validateSnapshots();

3. For each test class, setup your expectations

      Expect expect = Expect.of(snapshotVerifier, testMethod);
      expect.toMatchSnapshot("Something");

Here is a JUnit5 example that does not use the JUnit5 extension

package au.com.origin.snapshots.docs;

import au.com.origin.snapshots.Expect;
import au.com.origin.snapshots.PropertyResolvingSnapshotConfig;
import au.com.origin.snapshots.SnapshotVerifier;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestInfo;

// Notice we aren't using any framework extensions
public class CustomFrameworkExample {

  private static SnapshotVerifier snapshotVerifier;

  @BeforeAll
  static void beforeAll() {
    snapshotVerifier = new SnapshotVerifier(new PropertyResolvingSnapshotConfig(), CustomFrameworkExample.class);
  }

  @AfterAll
  static void afterAll() {
    snapshotVerifier.validateSnapshots();
  }

  @Test
  void shouldMatchSnapshotOne(TestInfo testInfo) {
    Expect expect = Expect.of(snapshotVerifier, testInfo.getTestMethod().get());
    expect.toMatchSnapshot("Hello World");
  }

}

Supplying your own snapshot name via @SnapshotName

By default, snapshots use the full method name as the identifier For example

au.com.origin.snapshots.docs.MyFirstSnapshotTest.helloWorldTest=[
Hello World
]

This strategy has a number of problems

• it's long and unwieldy
• if the method name or class name changes, your tests become orphans
• The Spock framework tests use a generated method name that is based on index (Spock tests should always use @SnapshotName)

You can supply a more meaningful name to your snapshot using @SnapshotName("your_custom_name") This will generate as follows

your_custom_name=[
Hello World
]

Much more concise and not affected by class name or method name refactoring.

Resolving conflicting snapshot comparison via *.snap.debug

Often your IDE has an excellent file comparison tool.

• A *.snap.debug file will be created alongside your *.snap file when a conflict occurs.
• You can then use your IDE tooling to compare the two files.
• *.snap.debug is deleted automatically once the test passes.

Note: *.snap.debug files should never be checked into version control so consider adding it to your .gitignore

snapshot.properties (required as of v2.4.0)

This file allows you to conveniently setup global defaults

key	Description
serializer	Class name of the serializer, default serializer
serializer.{name}	Class name of the serializer, accessible via .serializer("{name}")
comparator	Class name of the comparator
reporters	Comma separated list of class names to use as reporters
snapshot-dir	Name of sub-folder holding your snapshots
output-dir	Base directory of your test files (although it can be a different directory if you want)
ci-env-var	Name of environment variable used to detect if we are running on a Build Server

For example:

serializer=au.com.origin.snapshots.serializers.ToStringSnapshotSerializer
serializer.base64=au.com.origin.snapshots.serializers.Base64SnapshotSerializer
serializer.json=au.com.origin.snapshots.serializers.JacksonSnapshotSerializer
serializer.orderedJson=au.com.origin.snapshots.serializers.DeterministicJacksonSnapshotSerializer
comparator=au.com.origin.snapshots.comparators.PlainTextEqualsComparator
reporters=au.com.origin.snapshots.reporters.PlainTextSnapshotReporter
snapshot-dir=__snapshots__
output-dir=src/test/java
ci-env-var=CI

Parameterized tests

In cases where the same test runs multiple times with different parameters you need to set the scenario and it must be unique for each run

expect.scenario(params).toMatchSnapshot("Something");

Scenario Example

package au.com.origin.snapshots.docs

import au.com.origin.snapshots.Expect
import au.com.origin.snapshots.annotations.SnapshotName
import au.com.origin.snapshots.spock.EnableSnapshots
import spock.lang.Specification

@EnableSnapshots
class SpockWithParametersExample extends Specification {

    private Expect expect

    @SnapshotName("convert_to_uppercase")
    def 'Convert #scenario to uppercase'() {
        when: 'I convert to uppercase'
        String result = value.toUpperCase();
        then: 'Should convert letters to uppercase'
        // Check you snapshot against your output using a unique scenario
        expect.scenario(scenario).toMatchSnapshot(result)
        where:
        scenario | value
        'letter' | 'a'
        'number' | '1'
    }
}

Supplying a custom SnapshotSerializer

The serializer determines how a class gets converted into a string.

Serializers are pluggable, so you can write you own by implementing the SnapshotSerializer interface.

Currently, we support the following serializers.

Shipped with core

Serializer	Description
ToStringSnapshotSerializer	uses the toString() method
Base64SnapshotSerializer	use for images or other binary sources that output a byte[]. The output is encoded to Base64

Shipped with Jackson plugin

Serializer	Description
JacksonSnapshotSerializer	uses jackson to convert a class to a snapshot
DeterministicJacksonSnapshotSerializer	extension of JacksonSnapshotSerializer that also orders Collections for situations where the order changes on multiple runs

Serializers are resolved in the following order.

• (method level) explicitly expect(...).serializer(ToStringSerializer.class).toMatchSnapshot(); or via property file expect(...).serializer("json").toMatchSnapshot();
• (class level) explicitly @UseSnapshotConfig which gets read from the getSerializer() method
• (properties) implicitly via snapshot.properties

package au.com.origin.snapshots.docs;

import au.com.origin.snapshots.Expect;
import au.com.origin.snapshots.annotations.UseSnapshotConfig;
import au.com.origin.snapshots.junit5.SnapshotExtension;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;

@ExtendWith(SnapshotExtension.class)
@UseSnapshotConfig(LowercaseToStringSnapshotConfig.class)
public class JUnit5ResolutionHierarchyExample {

  @Test
  public void aliasMethodTest(Expect expect) {
    expect
        .serializer("json") // <------ Using snapshot.properties
        .toMatchSnapshot(new TestObject());
  }

  @Test
  public void customSerializerTest(Expect expect) {
    expect
        .serializer(UppercaseToStringSerializer.class)  // <------ Using custom serializer
        .toMatchSnapshot(new TestObject());
  }

  // Read from LowercaseToStringSnapshotConfig defined on the class
  @Test
  public void lowercaseTest(Expect expect) {
    expect.toMatchSnapshot(new TestObject());
  }
}

Example: HibernateSerializer

Sometimes the default serialization doesn't work for you. An example is Hibernate serialization where you get infinite recursion on Lists/Sets.

You can supply any serializer you like Gson, Jackson or something else.

For example, the following will exclude the rendering of Lists without changing the source code to include @JsonIgnore . This is good because you shouldn't need to add annotations to your source code for testing purposes only.

package au.com.origin.snapshots.docs;

import au.com.origin.snapshots.serializers.DeterministicJacksonSnapshotSerializer;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonIgnoreType;

import java.time.Instant;
import java.util.List;
import java.util.Set;

public class HibernateSnapshotSerializer extends DeterministicJacksonSnapshotSerializer {

  @Override
  public void configure(ObjectMapper objectMapper) {
    super.configure(objectMapper);

    // Ignore Hibernate Lists to prevent infinite recursion
    objectMapper.addMixIn(List.class, IgnoreTypeMixin.class);
    objectMapper.addMixIn(Set.class, IgnoreTypeMixin.class);

    // Ignore Fields that Hibernate generates for us automatically
    objectMapper.addMixIn(BaseEntity.class, IgnoreHibernateEntityFields.class);
  }

  @JsonIgnoreType
  class IgnoreTypeMixin {
  }

  abstract class IgnoreHibernateEntityFields {
    @JsonIgnore
    abstract Long getId();

    @JsonIgnore
    abstract Instant getCreatedDate();

    @JsonIgnore
    abstract Instant getLastModifiedDate();
  }
}

Supplying a custom SnapshotComparator

The comparator determines if two snapshots match.

Currently, we support one default comparator (PlainTextEqualsComparator) which uses string equals for comparison.

This should work for most cases. Custom implementations of SnapshotComparator can provide more advanced comparisons.

Comparators follow the same resolution order as Serializers

1. method
2. class
3. snapshot.properties

Example: JsonObjectComparator

The default comparator may be too strict for certain types of data. For example, when comparing json objects, formatting of the json string or the order of fields may not be of much importance during comparison. A custom comparator can help in such cases.

For example, the following will convert a json string to a Map and then perform an equals comparison so that formatting and field order are ignored.

package au.com.origin.snapshots.docs;

import au.com.origin.snapshots.comparators.SnapshotComparator;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.SneakyThrows;

public class JsonObjectComparator implements SnapshotComparator {
  @Override
  public boolean matches(String snapshotName, String rawSnapshot, String currentObject) {
    return asObject(snapshotName, rawSnapshot).equals(asObject(snapshotName, currentObject));
  }

  @SneakyThrows
  private static Object asObject(String snapshotName, String json) {
    return new ObjectMapper().readValue(json.replaceFirst(snapshotName, ""), Object.class);
  }
}

Supplying a custom SnapshotReporter

The reporter reports the details of comparison failures.

Currently, we support one default reporter (PlainTextSnapshotReporter) which uses assertj's DiffUtils to generate a patch of the differences between two snapshots.

Custom reporters can be plugged in by implementing SnapshotReporter.

Reporters follow the same resolution order as Serializers and Comparators

Example: JsonDiffReporter

For generating and reporting json diffs using other libraries like https://github.com/skyscreamer/JSONassert a custom reporter can be created like the one below.

package au.com.origin.snapshots.docs;

import au.com.origin.snapshots.reporters.SnapshotReporter;
import au.com.origin.snapshots.serializers.SerializerType;
import lombok.SneakyThrows;
import org.skyscreamer.jsonassert.JSONAssert;
import org.skyscreamer.jsonassert.JSONCompareMode;

public class JsonAssertReporter implements SnapshotReporter {
  @Override
  public boolean supportsFormat(String outputFormat) {
    return SerializerType.JSON.name().equalsIgnoreCase(outputFormat);
  }

  @Override
  @SneakyThrows
  public void report(String snapshotName, String rawSnapshot, String currentObject) {
    JSONAssert.assertEquals(rawSnapshot, currentObject, JSONCompareMode.STRICT);
  }
}

Supplying a custom SnapshotConfig

You can override the snapshot configuration easily using the @UseSnapshotConfig annotation

JUnit5 Example

package au.com.origin.snapshots.docs;

import au.com.origin.snapshots.Expect;
import au.com.origin.snapshots.annotations.UseSnapshotConfig;
import au.com.origin.snapshots.junit5.SnapshotExtension;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;

@ExtendWith(SnapshotExtension.class)
// apply your custom snapshot configuration to this test class
@UseSnapshotConfig(LowercaseToStringSnapshotConfig.class)
public class CustomSnapshotConfigExample {

  @Test
  public void myTest(Expect expect) {
    expect.toMatchSnapshot("hello world");
  }
}

Automatically updating a snapshot via -PupdateSnapshot=filter

Often - after analysing each snapshot and verifying it is correct, you will need to generate a new baseline for the snapshots.

Note that you may need to do some Gradle trickery to make this visible to your actual tests

test {
    systemProperty "updateSnapshot", project.getProperty("updateSnapshot")
}

Instead of deleting or manually modifying each snapshot you can pass -PupdateSnapshot which is equivalent to the --updateSnapshot flag in Jest

Update all snapshots automatically

-PupdateSnapshot

Update selected snapshots only using filter

pass the class names you want to update to filter

-PupdateSnapshot=UserService,PermissionRepository

Troubleshooting

I'm seeing this error in my logs

org/slf4j/LoggerFactory
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory

Solution: Add an SLF4J Provider such astestImplementation("org.slf4j:slf4j-simple:2.0.0-alpha0")

My test source files are not in src/test/java

Solution: Override output-dir in snapshot.properties

I see the following error in JSON snapshots java.lang.NoSuchFieldError: BINARY

Solution: This happened to me in a spring-boot app, I removed my jackson dependencies and relied on the ones from spring-boot instead.

Contributing

see CONTRIBUTING.md