Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pretty-printing inputs and outputs #85

Closed
owickstrom opened this issue Jan 22, 2020 · 14 comments
Closed

Pretty-printing inputs and outputs #85

owickstrom opened this issue Jan 22, 2020 · 14 comments

Comments

@owickstrom
Copy link

owickstrom commented Jan 22, 2020

Testing Problem

I'm using jqwik with much delight to test Apache Beam pipelines, i.e. integration tests using PBT. The inputs and outputs are generally large data structures, and bugs are found with sequences of inputs that trigger certain behaviour (e.g. a sequence of input elements interspersed with clock commands that cause time to pass in the test).

When jqwik finds a failing example it prints the minimal example input (after shrinking) along with the original input, I guess using toString? In any case, with these large structures, often being lists of large structures, the output is very hard to digest.

In my case, it can look something like the following, but with the lines being way longer:

sample = [[ElementReceived(timestampedValue=TimestampedValue(SuperLargeThing(...), 1970-01-01T00:00:00.001Z), ProcessingTimePassed(duration=PT1800S)], ...]
original-sample = [... far more SuperLargeThings ... ]

Suggested Solution

I'd like to have structures pretty-printed over multiple lines in order to make them more readable and easy troubleshooting. I can't suggest any universal pretty-printing solution for the JVM, so I'm thinking that this should be up to the user. Use toString by default, and let the user override with a custom printer.

In Haskell Hedgehog (which I'm most familiar with), there's an alternative to forAll called forAllWith, that takes a function to convert your generated value to a string. In jqwik, this is handled with method arguments and annotations, so I'm not sure what would be a suitable counterpart. Maybe there's some convention on the JVM (or in Java, specifically) that could be used?

I'm leaving the solution part of this issue intentionally vague.

Discussion

Another concern, which might be a separate issue altogether, is pretty-printing in assertions. The same pretty-printing solution used for examples would be nice to use in AssertJ or whatever assertions used. Ideally, I'd like to have pretty-printed data structures and diffs (like git diff). Example from an article I wrote:

I'm not sure if there exists any library for doing pretty-printed diffs that could be used both for printing examples and in assertions.

Looking forward to getting some feedback on this!

@jlink
Copy link
Owner

jlink commented Jan 22, 2020

@owickstrom Can you give a specific example of how a structure of your choice should look when being pretty-printed? I'm especially interested in line breaks between different parameters and line breaks within a single parameter.

@owickstrom
Copy link
Author

owickstrom commented Jan 22, 2020

Sure! I can't show you the actual code as it's for a proprietary project, but I can sketch something out.

If we have a test looking something like this:

public void testExample(ForAll("inputs") List<Input<SuperLargeThing>> inputs, @ForAll Integer someNumber) { ... }

Then the Jqwik output might contain:

Sample
------

  inputs: [
    ElementReceived(
      timestampedValue=TimestampedValue(
        value=SuperLargeThing(
          foo=1,
          bar="",
          baz=null
        ),
        timestamp=1970-01-01T00:00:00.001Z
      )
    ),
    ProcessingTimePassed(
      duration=PT1800S
    ),
    ...
  ]
  someNumber: 0

Original Sample
---------------

(similar to above)

Note that I'd prefer every field/value pair to be on a separate line. If the pretty printer could compact small values on a single line then that's fine, but I rather have more lines than longer lines.

Also note that I used another formatting for "Sample" and "Original Sample" headings. That might not be in line with jqwik's style, so ignore that if you don't like it.

@jlink
Copy link
Owner

jlink commented Jan 22, 2020

@owickstrom Thanks. IMO there's some generic stuff that could be improved and there's the idea of pluggable output formatting. I'll be pondering a bit and get back here.

@owickstrom
Copy link
Author

owickstrom commented Jan 22, 2020

@jlink Sounds good! 👍

@jlink
Copy link
Owner

jlink commented Feb 4, 2020

Here's a first draft of what I could do...

Step 1: Make reporting configurable through jqwik.properties

reportingFormat=COMPACT|DETAILED

Step 2: Introduce DETAILED format, similar to

                              |-------------------jqwik-------------------
tries = 1                     | # of calls to property
...
seed = 8531870486618337326    | random seed to reproduce generated values

Sample
-------
aString (java.lang.String): "this is a string"
myObject (my.package.MyObject): <toString() or registered (multiline) printer>

Original Sample
----------------
aString (java.lang.String): "this is another string"
myObject (my.package.MyObject): <toString() or registered (multiline) printer>

Mind that in Java there's additional compiler configuration necessary (-parameters) to have real parameter names available. Otherwise they'll show up as arg0, arg1 etc.

Step 3: Allow registration of printers through Java's SPI mechanism with a few already registered default printers for lists, sets, arrays. Interface could be like

interface SamplePrinter<T> {
    boolean canPrintFor(TypeUsage targetType);
    List<String> print(T object);
}

What do you think?

@jlink
Copy link
Owner

jlink commented Apr 7, 2020

To account for the difference of compact and detailed printing I modify the suggested type:

interface SamplePrintingFormat {
  boolean canPrint(TypeUsage targetType);
  String printCompact(Object object, Function<Object, String> compactPrinter);
  List<String> printDetailed(Object object, Function<Object, List<String> detailedPrinter);
}

The compactPrinter and detailedPrinter functions are needed to print containedObjects.

I suggest an optional annotation @SamplePrinting(SamplePrintingMode)
and an addition configuration parameter:

defaultSamplePrintingMode = COMPACT|DETAILED|AUTO

Mode AUTO would use compact mode under a certain threshold, e.g. 60 chars.

@jlink
Copy link
Owner

jlink commented Jun 3, 2020

To make SamplePrintingFormat simpler I tend to just join the lines of detailed format with comma.
So here's my current suggestion:

interface SamplePrintingFormat {
  boolean canPrint(TypeUsage targetType);
  /**
   * @param printer Use to format contained objects
  **/
  List<String> print(Object object, Function<Object, List<String> printer);
}

I'll probably start working on it soon.

@jlink
Copy link
Owner

jlink commented Jun 11, 2020

Some progress made. Consider the following property:

@Property(afterFailure = AfterFailureMode.RANDOM_SEED)
void reportFalsifiedSamples(
	@ForAll int anInt,
	@ForAll List<Integer> listOfInts,
	@ForAll @Size(min = 3) Map<@AlphaChars @StringLength(3) String, Integer> aMap
) {
	Assertions.assertThat(anInt).isLessThan(10);
}

will now produce the following report:

SampleReportingExamples:reportFalsifiedSamples = 

java.lang.AssertionError: 
Expecting:
 <10>
to be less than:
 <10> 

                              |-------------------jqwik-------------------
tries = 1                     | # of calls to property
checks = 1                    | # of not rejected calls
generation = RANDOMIZED       | parameters are randomly generated
after-failure = RANDOM_SEED   | use a new random seed
edge-cases#mode = MIXIN       | edge cases are mixed in
edge-cases#total = 0          | # of all combined edge cases
edge-cases#tried = 0          | # of edge cases tried in current run
seed = 5636506477930535961    | random seed to reproduce generated values

Sample
------
  anInt: 10
  listOfInts: []
  aMap: {"aaa"=0, "aba"=0, "AAA"=0}

Original Sample
---------------
  anInt: 40
  listOfInts:
    [
      3472291, -117543, -102, 997, 280, -922, -1294, 27, -13279, 64, -2147483648, 291, -582, -741, 
      20, 4103137, 61097, 6136
    ]
  aMap:
    {
      "pAr"=-8, 
      "oAz"=-5879760, 
      "Zzk"=6533, 
      "ULA"=-52, 
      ...
      "hZn"=49000, 
      "ZtX"=785
    }

@jlink
Copy link
Owner

jlink commented Jun 14, 2020

More progress. Reporting arrays:

@Property(afterFailure = AfterFailureMode.RANDOM_SEED)
void reportFalsifiedArrays(
	@ForAll int anInt,
	@ForAll int[] arrayOfInts,
	@ForAll @Size(min = 2) @AlphaChars @StringLength(3) String[] arrayOfStrings
) {
	Assertions.assertThat(anInt).isLessThan(10);
}
Sample
------
  anInt: 10
  arrayOfInts: int[] []
  arrayOfStrings: String[] ["aaa", "aaa"]

Original Sample
---------------
  anInt: 195
  arrayOfInts: int[] [14114, 2120155713, 27, 2147483647, 2563]
  arrayOfStrings:
    String[] [
      "oJJ", "aza", "zrZ", "aAx", "Zaz", "SAz", "zlL", "ApK", "jAz", "oXA", "zam", "wZt", "XAa", 
      "XGA", "eZi"
    ]

Of course, available in "1.3.1-SNAPSHOT"

@owickstrom
Copy link
Author

owickstrom commented Jun 14, 2020

@jlink
Copy link
Owner

jlink commented Jun 17, 2020

More progress. One can register sample formats for any class:

Consider the following property:

@Property(afterFailure = AfterFailureMode.RANDOM_SEED)
@Report(Reporting.GENERATED)
void reportWithFormat(@ForAll("dates") LocalDate localDate) {
	Assertions.assertThat(localDate).isBefore(LocalDate.of(2000, 1, 1));
}

@Provide
Arbitrary<LocalDate> dates() {
	Arbitrary<Integer> years = Arbitraries.integers().between(1900, 2100);
	Arbitrary<Integer> months = Arbitraries.integers().between(1, 12);
	Arbitrary<Integer> days = Arbitraries.integers().between(1, 28);

	return Combinators.combine(years, months, days).as(LocalDate::of);
}

And register this format class:

public static class LocalDateFormat implements SampleReportingFormat {

	@Override
	public boolean appliesTo(final Object value) {
		return value instanceof LocalDate;
	}

	@Override
	public Object report(final Object value) {
		LocalDate date = (LocalDate) value;
		Map<String, Object> valueMap = new HashMap<>();
		valueMap.put("year", date.getYear());
		valueMap.put("month", date.getMonth());
		valueMap.put("day", date.getDayOfMonth());
		return valueMap;
	}

	@Override
	public Optional<String> label(final Object value) {
		return Optional.of("LocalDate ");
	}
}

Then the output will be:

Sample
------
  localDate: LocalDate {"month"=JANUARY, "year"=2000, "day"=1}

Original Sample
---------------
  localDate: LocalDate {"month"=APRIL, "year"=2025, "day"=27}

@jlink
Copy link
Owner

jlink commented Jun 17, 2020

here's the relevant section in the user guide: https://jqwik.net/docs/snapshot/user-guide.html#failure-reporting

@jlink
Copy link
Owner

jlink commented Jun 17, 2020

Available in "1.3.1-SNAPSHOT"

@jlink jlink closed this as completed Jun 17, 2020
@jlink jlink removed the in progress label Jun 17, 2020
@owickstrom
Copy link
Author

owickstrom commented Jun 18, 2020

Very cool, looking forward to trying this in our project at work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants