Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Property test #306

Merged
merged 2 commits into from
Mar 22, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion api/src/main/java/com/mastercard/test/flow/Model.java
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ default Stream<Flow> flows() {
* sub-models
* @return <code>this</code>
*/
Model withListener( Listener l );
Model listener( Listener l );

/**
* Instrumentation for {@link Model} construction
Expand Down
10 changes: 6 additions & 4 deletions assert/assert-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Some aspects of assertion behaviour can be controlled by system properties:

## Flow Selection

The framework will default to exercising all flows in the model that are relevant to the system under test. It is possible to run a subset of flows by setting the `mctf.filter` system properties described above.
The framework will default to exercising all flows in the model that are relevant to the system under test. It is possible to run a subset of flows by setting the filter system properties described above.

Note that flows will automatically be brought into the execution order as required to satisfy flow dependencies in the system model.

Expand All @@ -66,8 +66,10 @@ Note that flows will automatically be brought into the execution order as requir
If a parent flow fails, it is likely that the descendants of that flow will fail in the same way.
The assert components will thus skip running them.
This speeds up test execution and avoids spamming the report with duplicates of the same failure.
This behaviour can be avoided by setting system property `mctf.suppress.basis` to `true`.

We'll also skip flows where the dependency flows suffered an error.
This behaviour can be avoided by setting system property `mctf.suppress.dependency` to `true`.

## Flow ordering

Expand All @@ -78,8 +80,8 @@ The assert components will work out a flow execution order that:

## Report generation

The results of flow execution can be (depending on how the Flocessor is configured) collated into a human-readable report that details observed system behaviour and the results of comparing that against the system model.
The location of the report can be controlled with the `mctf.dir` and `mctf.report.name` system properties.
The results of flow execution can be (depending on how the `Flocessor` is configured) collated into a human-readable report that details observed system behaviour and the results of comparing that against the system model.
The location of the report can be controlled with the `mctf.dir` and `mctf.report.dir` system properties.

## Report replay

Expand All @@ -94,7 +96,7 @@ While we _could_ run these iterative assertions against the same system that pro
Using a report as the source of data for assertion is likely to be more efficient than exercising the actual system, and is obviously far more convenient if the system is difficult to reliably access.

Set the `mctf.replay=path/to/report_directory` system property to activate replay mode - the specified report will be used as the source of observed behaviour rather than the actual system.
Setting `mctf.reply=latest` will cause the most recently-generated report in the `mctf` artifact directory to be replayed.
Setting `mctf.replay=latest` will cause the most recently-generated report in the `mctf` artifact directory to be replayed.

In your tests you can check the value of `Replay.isActive()` to detect if replay mode is active.
This allows you to avoid doing initialisation and teardown operations that are only relevant if you're testing against the actual system.
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ public Stream<Model> subModels() {
}

@Override
public Model withListener( Listener l ) {
public Model listener( Listener l ) {
throw new UnsupportedOperationException();
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ public Stream<Model> subModels() {
}

@Override
public Model withListener( Listener l ) {
public Model listener( Listener l ) {
throw new UnsupportedOperationException();
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ public Stream<Flow> flows( Set<String> include, Set<String> exclude ) {
}

@Override
public Model withListener( Listener l ) {
public Model listener( Listener l ) {
throw new UnsupportedOperationException();
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ public Stream<Flow> flows( Set<String> include, Set<String> exclude ) {
}

@Override
public Model withListener( Listener l ) {
public Model listener( Listener l ) {
throw new UnsupportedOperationException();
}

Expand Down
4 changes: 2 additions & 2 deletions doc/src/main/markdown/further.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This guide explores some features that will be useful as the system model grows
Adding a call to [`.reporting()`][AbstractFlocessor.reporting(Reporting)] to the construction chain of the `Flocessor` instance controls whether a HTML report of the test run is produced. The report will detail both the expected and observed system behaviour and the results of comparing the two.
The [`Reporting` enum value][Reporting] that you supply controls whether the report is generated and under what circumstances it is automatically opened in a browser.

By default the report will be saved to a timestamped directory under `target/mctf`, but the `mctf.report.name` system property offers control over the destination directory.
By default the report will be saved to a timestamped directory under `target/mctf`, but the `mctf.report.dir` system property offers control over the destination directory.

<!-- code_link_start -->

Expand All @@ -29,7 +29,7 @@ If you find yourself in the following circumstances:

then you can use the report replay feature to quickly iterate changes to your flows until the system model is accurate.

Run the same test that produced the report, but set system property `mctf.report.replay` to activate replay mode. The property can be set to the path to the report to read data from, or to `latest` to replay from the most recent report in `target/mctf`.
Run the same test that produced the report, but set system property `mctf.replay` to activate replay mode. The property can be set to the path to the report to read data from, or to `latest` to replay from the most recent report in `target/mctf`.

You can use [`Replay.isActive()`][Replay.isActive()] in your assertion components to avoid setup and teardown activities that serve no purpose when the source of data is a report rather than the actual system.

Expand Down
60 changes: 60 additions & 0 deletions doc/src/test/java/com/mastercard/test/flow/doc/PropertyTest.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
package com.mastercard.test.flow.doc;

import static org.junit.Assert.fail;
import static org.junit.jupiter.api.DynamicTest.dynamicTest;

import java.nio.file.Path;
import java.util.Set;
import java.util.TreeSet;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.Stream;

import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;

import com.mastercard.test.flow.assrt.AssertionOptions;
import com.mastercard.test.flow.assrt.filter.FilterOptions;
import com.mastercard.test.flow.report.QuietFiles;

/**
* Checks the correctness of things that look like system properties in our
* documentation
*/
@SuppressWarnings("static-method")
class PropertyTest {

private static final Pattern PROPERTY = Pattern.compile( "(mctf\\.[a-z._]+)" );

/**
* @return per-file tests that throw a wobbler if we find an invalid property
*/
@TestFactory
Stream<DynamicTest> markdown() {

Set<String> validProperties = new TreeSet<>();
Stream.of( AssertionOptions.values() ).forEach( o -> validProperties.add( o.property() ) );
Stream.of( FilterOptions.values() ).forEach( o -> validProperties.add( o.property() ) );

return Util.markdownFiles()
.map( mdFile -> dynamicTest(
mdFile.toString(),
() -> checkProperties( mdFile, validProperties ) ) );
}

private static void checkProperties( Path file, Set<String> validProperties ) {
QuietFiles.lines( file )
.forEach( line -> {
Matcher mtch = PROPERTY.matcher( line );
while( mtch.find() ) {
if( !validProperties.contains( mtch.group( 1 ) ) ) {
fail( String.format(
"Line\n '%s'\ncontains unknown property '%s'. Valid values are\n %s",
line, mtch.group( 1 ),
validProperties.stream().collect( Collectors.joining( "\n " ) ) ) );
}
}
} );
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ public Stream<Model> subModels() {
}

@Override
public Model withListener( Listener l ) {
children.forEach( c -> c.withListener( l ) );
public Model listener( Listener l ) {
children.forEach( c -> c.listener( l ) );
return this;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ else if( o instanceof Stream<?> ) {
}

@Override
public Model withListener( Listener l ) {
public Model listener( Listener l ) {
if( l != null ) {
l.count( this, 0, members.size() );
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ private <T extends EagerModel> Model instantiate( Class<T> type ) {
}
try {
T instance = (T) constructor.newInstance( parameters );
instance.withListener( listener.orElse( null ) );
instance.listener( listener.orElse( null ) );
instances.put( type, instance );
}
catch( Exception e ) {
Expand All @@ -162,9 +162,9 @@ public Stream<Model> subModels() {
}

@Override
public Model withListener( Listener l ) {
public Model listener( Listener l ) {
listener = Optional.ofNullable( l );
instances.values().forEach( i -> i.withListener( l ) );
instances.values().forEach( i -> i.listener( l ) );
return this;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ void listener() {
NoDeps.class,
Deps.class ) );
BuildListener bl = new BuildListener();
Model ret = gm.withListener( bl );
Model ret = gm.listener( bl );
assertSame( gm, ret );

assertEquals( ""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -193,13 +193,13 @@ void emptyModel() {

/**
* SHows that the
* {@link EagerModel#withListener(com.mastercard.test.flow.Model.Listener)}
* returns the same instance
* {@link EagerModel#listener(com.mastercard.test.flow.Model.Listener)} returns
* the same instance
*/
@Test
void withListener() {
void listener() {
Model m = new TaggedModel();
Model r = m.withListener( null );
Model r = m.listener( null );
assertSame( m, r );
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ void submodels() {
.with( Transitive.class );

BuildListener bl = new BuildListener();
lm.withListener( bl );
lm.listener( bl );

// request a flow to cause a single submodel to be built
assertEquals( 1, lm.flows( Collections.singleton( "a" ), Collections.emptySet() ).count() );
Expand Down Expand Up @@ -354,7 +354,7 @@ private static void assertModel( Model m,
Set<String> include, Set<String> exclude,
String tags, String flows, String builds ) {
BuildListener bl = new BuildListener();
assertSame( m, m.withListener( bl ) );
assertSame( m, m.listener( bl ) );

assertEquals( tags, m.tags().toString() );
assertEquals( "", bl.buildEvents(), "tag access does not build models" );
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ public Stream<Model> subModels() {
}

@Override
public Model withListener( Listener l ) {
public Model listener( Listener l ) {
throw new UnsupportedOperationException();
}
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ public Stream<Model> subModels() {
}

@Override
public Model withListener( Listener l ) {
public Model listener( Listener l ) {
throw new UnsupportedOperationException();
}
};
Expand Down