Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add @Snapshot as built-in extension #1873

Merged
90 changes: 90 additions & 0 deletions docs/extensions.adoc
Expand Up @@ -723,6 +723,96 @@ runner {
}
----

[[snapshot-testing]]
=== Snapshot testing
leonard84 marked this conversation as resolved.
Show resolved Hide resolved

Since version 2.4, the `@Snapshot` extension has been added for snapshot testing.
This type of testing compares the current output of your code with a saved version (a snapshot) to check for any changes.
It's especially useful for checking if your code still produces the right text outputs after changes.
The `@Snapshot` extension also makes it easy to create and update these snapshots.

You use the `@Snapshot` extension along with the `Snapshotter` class.
The `Snapshotter` class provides the entrypoint for the tests.
You can use `@Snapshot` in two ways: you can add it to a field in your code, or you can use it as a parameter in a method.
The examples below will show you how to do both.

.Snapshot Extension as field
[source,groovy]
----
include::{sourcedir}/extension/SnapshotDocSpec.groovy[tag=field]
----

.Snapshot Extension as parameter
[source,groovy]
----
include::{sourcedir}/extension/SnapshotDocSpec.groovy[tag=parameter]
----

You can store and assert over multiple snapshots by providing an optional snapshotId as identifier.

.Multiple snapshots
[source,groovy]
----
include::{sourcedir}/extension/SnapshotDocSpec.groovy[tag=multi-snapshot]
----

By default, snapshots are stored and compared as plain text.
However, you can plug your own matching logic.
For example, you could compare two json structures with a third-party library.

.Custom matching
[source,groovy]
----
include::{sourcedir}/extension/SnapshotDocSpec.groovy[tag=custom-matching]
----

You can also extend from the `Snapshotter` class to provide your own convenience methods.
Check `org.spockframework.specs.extension.SpockSnapshotter` in the spock codebase for an example.

==== Configuration

The snapshots are stored in the `rootPath` directory, which can be configured either in the <<spock-configuration-file,Spock Configuration File>>, or via the `spock.snapshots.rootPath` system property.
The `rootPath` directory is required and the `@Snapshot` extension will throw an exception if it is not configured when the extension is used.

Snapshots can be updated when setting the `spock.snapshots.updateSnapshots` system property to `true`, or via the config file.

You can configure the extension to store the actual value in case of a mismatch by setting the `spock.snapshots.writeActual` system property to `true`, or via the config file.
If enabled the extension will store the result in a file next to the original one with an additional `.actual` extension.
The `.actual` file will automatically be deleted on the next successful run, as long as the feature is still enabled.
This option is useful, when the snapshot is too large or complex to analyze with the built-in reporting.

.Example snapshot config
[source,groovy]
----
snapshots {
rootPath = Paths.get("src/test/resources")
updateSnapshots = System.getenv("UPDATE_SNAPSHOTS") == "true"
writeActualSnapshotOnMismatch = !System.getenv("CI")
defaultExtension = 'snap.groovy'
}
----

The `@Snapshot` extension will also set the `snapshot` <<test-tag-extension,tag>> on the affected features.
This can be utilized to only run the features that produce snapshots when updating them.

.Example Gradle config for snapshot testing
[source,groovy]
----
tasks.named("test", Test) {
useJUnitPlatform()
// set the snapshot directory, as resources are already an input we don't need to track them separately
systemProperty("spock.snapshots.rootPath", "src/test/resources")
// allow updating the snapshots with running `gradlew test -PupdateSnapshots`
if (project.hasProperty("updateSnapshots")) {
systemProperty("spock.snapshots.updateSnapshots", "true")
// not strictly necessary but speeds up the process by only executing snapshot tests
useJUnitPlatform {
includeTags("snapshot")
}
}
}
----

== Third-Party Extensions

You can find a list of third-party extensions in the https://github.com/spockframework/spock/wiki/Third-Party-Extensions[Spock Wiki].
Expand Down
1 change: 1 addition & 0 deletions docs/release_notes.adoc
Expand Up @@ -29,6 +29,7 @@ include::include.adoc[]
=== Misc
* Add support for <<extensions.adoc#extension-store,keeping state in extensions>>
* Add support for parameter injection of `@TempDir`
* Add `@Snapshot` extension for <<extensions.adoc#snapshot-testing,snapshot testing>>
* Improve `@TempDir` field injection, now it happens before field initialization, so it can be used by other field initializers.
* Fix exception when configured `baseDir` was not existing, now `@TempDir` will create the baseDir directory if it is missing.
* Fix bad error message for collection conditions, when one of the operands is `null`
Expand Down
@@ -0,0 +1,53 @@
/*
* Copyright 2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* https://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.spockframework.runtime.extension.builtin;

import org.spockframework.util.Beta;
import org.spockframework.util.Nullable;
import spock.config.ConfigurationObject;

import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Optional;

/**
* Configuration for the {@link spock.lang.Snapshot} extension.
*
* @since 2.4
*/
@Beta
@ConfigurationObject("snapshots")
public class SnapshotConfig {
leonard84 marked this conversation as resolved.
Show resolved Hide resolved
/**
* Controls the where the snapshots are stored.
*/
@Nullable
public Path rootPath = Optional.ofNullable(System.getProperty("spock.snapshots.rootPath")).map(Paths::get).orElse(null);
/**
* Instructs the {@link spock.lang.Snapshotter} to update the snapshot instead of failing on a mismatch or missing snapshot.
*/
public boolean updateSnapshots = Boolean.getBoolean("spock.snapshots.updateSnapshots");
/**
* Controls whether the {@link spock.lang.Snapshotter} should write actual value next to the snapshot file with the '.actual' extension.
* <p>
* The file will be deleted upon a successful match.
*/
public boolean writeActualSnapshotOnMismatch = Boolean.getBoolean("spock.snapshots.writeActual");
/**
* The default extension to use.
*/
public String defaultExtension = "txt";
}
@@ -0,0 +1,92 @@
/*
* Copyright 2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* https://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.spockframework.runtime.extension.builtin;

import org.spockframework.runtime.extension.IAnnotationDrivenExtension;
import org.spockframework.runtime.extension.IMethodInterceptor;
import org.spockframework.runtime.extension.IMethodInvocation;
import org.spockframework.runtime.extension.ParameterResolver;
import org.spockframework.runtime.model.FieldInfo;
import org.spockframework.runtime.model.MethodInfo;
import org.spockframework.runtime.model.ParameterInfo;
import org.spockframework.runtime.model.SpecInfo;
import org.spockframework.util.Assert;
import org.spockframework.util.Checks;
import org.spockframework.util.ReflectionUtil;
import spock.lang.Snapshot;
import spock.lang.Snapshotter;

import java.nio.charset.Charset;

public class SnapshotExtension implements IAnnotationDrivenExtension<Snapshot> {
leonard84 marked this conversation as resolved.
Show resolved Hide resolved
private final SnapshotConfig config;

public SnapshotExtension(SnapshotConfig config) {
this.config = Assert.notNull(config);
Checks.notNull(config.rootPath, () -> "Root path must be set, when using @Snapshot");
}

@Override
public void visitFieldAnnotation(Snapshot annotation, FieldInfo field) {
Checks.checkArgument(Snapshotter.class.isAssignableFrom(field.getType()), () -> "Field must be of type spock.lang.Snapshotter or a valid subtype");

SpecInfo spec = field.getParent().getBottomSpec();
spec.getAllFeatures().forEach(featureInfo -> featureInfo.addTestTag("snapshot"));
spec.addSetupInterceptor(new SnapshotInterceptor(annotation, field));
}

@Override
public void visitParameterAnnotation(Snapshot annotation, ParameterInfo parameter) {
Class<?> type = parameter.getReflection().getType();
Checks.checkArgument(Snapshotter.class.isAssignableFrom(type), () -> "Field must be of type spock.lang.Snapshotter or a valid subtype");

MethodInfo method = parameter.getParent();
method.getFeature().addTestTag("snapshot");
method.addInterceptor(new ParameterResolver.Interceptor<>(parameter, (IMethodInvocation invocation) -> createSnapshotter(invocation, type, annotation)));

}

private Snapshotter createSnapshotter(IMethodInvocation invocation, Class<?> type, Snapshot annotation) {
String extension = annotation.extension().equals("<default>")
? Checks.notNull(config.defaultExtension, () -> "'snapshot.defaultExtension' must not be null.")
: annotation.extension();
Snapshotter.Store snapshotStore = new Snapshotter.Store(
invocation.getMethod().getIteration(),
config.rootPath,
config.updateSnapshots,
config.writeActualSnapshotOnMismatch,
extension,
Charset.forName(annotation.charset()));
Checks.checkArgument(Snapshotter.class.isAssignableFrom(type), () -> "Target must be of type spock.lang.Snapshotter or a valid subtype");
return (Snapshotter) ReflectionUtil.newInstance(type, snapshotStore);
}

private class SnapshotInterceptor implements IMethodInterceptor {
private final FieldInfo field;
private final Snapshot annotation;

private SnapshotInterceptor(Snapshot annotation, FieldInfo field) {
this.field = Assert.notNull(field);
this.annotation = Assert.notNull(annotation);
}

@Override
public void intercept(IMethodInvocation invocation) throws Throwable {
field.writeValue(invocation.getInstance(), createSnapshotter(invocation, field.getType(), annotation));
invocation.proceed();
}
}
}
28 changes: 14 additions & 14 deletions spock-core/src/main/java/org/spockframework/util/Assert.java
@@ -1,17 +1,16 @@
/*
* Copyright 2009 the original author or authors.
* Copyright 2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* https://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.spockframework.util;
Expand All @@ -28,13 +27,14 @@
public abstract class Assert {
// IDEA: beef up error message ("please submit bug report", include caller in message in case stack trace is lost)
@Contract("null -> fail")
public static void notNull(Object obj) {
notNull(obj, "argument is null");
public static <T> T notNull(T obj) {
return notNull(obj, "argument is null");
}

@Contract("null, _, _ -> fail")
public static void notNull(Object obj, String msg, Object... values) {
public static <T> T notNull(T obj, String msg, Object... values) {
if (obj == null) throw new InternalSpockError(String.format(msg, values));
return obj;
}

@Contract("false -> fail")
Expand Down
59 changes: 40 additions & 19 deletions spock-core/src/main/java/org/spockframework/util/IoUtil.java
@@ -1,20 +1,28 @@
/*
* Copyright 2010 the original author or authors.
* Copyright 2023 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* https://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* https://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.spockframework.util;

import org.codehaus.groovy.runtime.IOGroovyMethods;
import org.jetbrains.annotations.NotNull;

import java.io.*;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.*;

import static java.util.Collections.emptyList;
Expand All @@ -27,25 +35,30 @@
*/
public static String getText(Reader reader) throws IOException {
try(BufferedReader buffered = new BufferedReader(reader)) {
StringBuilder source = new StringBuilder();
return readFully(buffered);

Check warning on line 38 in spock-core/src/main/java/org/spockframework/util/IoUtil.java

View check run for this annotation

Codecov / codecov/patch

spock-core/src/main/java/org/spockframework/util/IoUtil.java#L38

Added line #L38 was not covered by tests
}
}

String line = buffered.readLine();
private static String readFully(BufferedReader buffered) throws IOException {
// use Groovy's methods as they keep the correct line endings
return IOGroovyMethods.getText(buffered);
}

while (line != null) {
source.append(line);
source.append('\n');
line = buffered.readLine();
}
public static String getText(Path path) throws IOException {
return getText(path, StandardCharsets.UTF_8);

Check warning on line 48 in spock-core/src/main/java/org/spockframework/util/IoUtil.java

View check run for this annotation

Codecov / codecov/patch

spock-core/src/main/java/org/spockframework/util/IoUtil.java#L48

Added line #L48 was not covered by tests
}

return source.toString();
public static String getText(Path path, Charset charset) throws IOException {
try(BufferedReader buffered = Files.newBufferedReader(path, charset)) {
return readFully(buffered);
}
}

/**
* Returns the text read from the given file as a String.
*/
public static String getText(File path) throws IOException {
return getText(new FileReader(path));
return getText(path.toPath());

Check warning on line 61 in spock-core/src/main/java/org/spockframework/util/IoUtil.java

View check run for this annotation

Codecov / codecov/patch

spock-core/src/main/java/org/spockframework/util/IoUtil.java#L61

Added line #L61 was not covered by tests
}

/**
Expand All @@ -56,6 +69,14 @@
return getText(new InputStreamReader(stream));
}

public static void writeText(Path path, String content, Charset charset) {
try (BufferedWriter writer = Files.newBufferedWriter(path, charset)) {
writer.write(content);
} catch (Exception e) {
throw new RuntimeException(e);

Check warning on line 76 in spock-core/src/main/java/org/spockframework/util/IoUtil.java

View check run for this annotation

Codecov / codecov/patch

spock-core/src/main/java/org/spockframework/util/IoUtil.java#L75-L76

Added lines #L75 - L76 were not covered by tests
}
}

public static void createDirectory(File dir) throws IOException {
if (!dir.isDirectory() && !dir.mkdirs())
throw new IOException("Failed to create directory: " + dir);
Expand Down