Close #92 - Test report does not have any message from test failure #93

kevin-lee · 2019-07-14T04:22:58Z

Close #92 - Test report does not have any message from test failure

When if fails, the test report may look like

<?xml version='1.0' encoding='UTF-8'?>
<testsuite hostname="my.local" name="io.kevinlee.property_based_testing.MyAppSpecWithHedgehog" tests="1" errors="0" failures="1" skipped="0" time="0.035" timestamp="2019-07-13T14:17:29">
  <properties>
    <!-- omitted -->
  </properties>
    <testcase classname="io.kevinlee.property_based_testing.MyAppSpecWithHedgehog" name="add(a, b) should return a + b" time="0.035">
      <failure message="x: 1
y: 1
=== Not Equal ===
2
3" type="java.lang.AssertionError">java.lang.AssertionError: x: 1
y: 1
=== Not Equal ===
2
3
  at hedgehog.sbt.Event$.fromReport(Framework.scala:103)
  at hedgehog.sbt.Task.$anonfun$execute$1(Framework.scala:76)
  at hedgehog.sbt.Task.$anonfun$execute$1$adapted(Framework.scala:72)
  at scala.collection.immutable.List.foreach(List.scala:388)
  at hedgehog.sbt.Task.execute(Framework.scala:72)
  at sbt.TestRunner.runTest$1(TestFramework.scala:113)
  at sbt.TestRunner.run(TestFramework.scala:124)
  at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282)
  at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246)
  at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
  at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
  at sbt.TestFunction.apply(TestFramework.scala:294)
  at sbt.Tests$.$anonfun$toTask$1(Tests.scala:309)
  at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46)
  at sbt.std.Transform$$anon$4.work(System.scala:67)
  at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
  at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
  at sbt.Execute.work(Execute.scala:278)
  at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
  at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
  at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:834)
    </failure>
  </testcase>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

To compare with other testing tools, I've also provided the reports from ScalaCheck and ScalaTest. It's not the same test though.

ScalaCheck

<?xml version='1.0' encoding='UTF-8'?>
<testsuite hostname="my.local" name="io.kevinlee.property_based_testing.MyAppSpecWithScalaCheck-0" tests="1" errors="0" failures="1" skipped="0" time="-0.001" timestamp="2019-07-12T21:09:33">
  <properties>
    <!-- omitted -->
  </properties>
  <testcase classname="io.kevinlee.property_based_testing.MyAppSpecWithScalaCheck-0" name="add(Int, Int): has identity element" time="-0.001">
    <failure message="Falsified after 0 passed tests.
&gt; ARG_0: 0
&gt; ARG_0_ORIGINAL: -1327215743" type="java.lang.Exception">java.lang.Exception: Falsified after 0 passed tests.
&gt; ARG_0: 0
&gt; ARG_0_ORIGINAL: -1327215743
  at org.scalacheck.ScalaCheckRunner$$anon$2$$anon$6.&lt;init&gt;(ScalaCheckFramework.scala:115)
  at org.scalacheck.ScalaCheckRunner$$anon$2.$anonfun$execute$5(ScalaCheckFramework.scala:104)
  at org.scalacheck.ScalaCheckRunner$$anon$2.$anonfun$execute$5$adapted(ScalaCheckFramework.scala:100)
  at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:788)
  at scala.collection.immutable.List.foreach(List.scala:388)
  at scala.collection.generic.TraversableForwarder.foreach(TraversableForwarder.scala:34)
  at scala.collection.generic.TraversableForwarder.foreach$(TraversableForwarder.scala:34)
  at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:43)
  at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:787)
  at org.scalacheck.ScalaCheckRunner$$anon$2.$anonfun$execute$3(ScalaCheckFramework.scala:100)
  at org.scalacheck.ScalaCheckRunner$$anon$2.$anonfun$execute$3$adapted(ScalaCheckFramework.scala:97)
  at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)
  at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
  at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:194)
  at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)
  at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)
  at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:194)
  at org.scalacheck.ScalaCheckRunner$$anon$2.execute(ScalaCheckFramework.scala:97)
  at sbt.TestRunner.runTest$1(TestFramework.scala:113)
  at sbt.TestRunner.run(TestFramework.scala:124)
  at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282)
  at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246)
  at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
  at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
  at sbt.TestFunction.apply(TestFramework.scala:294)
  at sbt.Tests$.$anonfun$toTask$1(Tests.scala:309)
  at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46)
  at sbt.std.Transform$$anon$4.work(System.scala:67)
  at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
  at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
  at sbt.Execute.work(Execute.scala:278)
  at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
  at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
  at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:834)
    </failure>
  </testcase>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

ScalaTest

<?xml version='1.0' encoding='UTF-8'?>
<testsuite hostname="my.local" name="io.kevinlee.property_based_testing.MyAppTest" tests="1" errors="0" failures="1" skipped="0" time="0.02" timestamp="2019-07-12T21:09:33">
  <properties>
    <!-- omitted -->
  </properties>
  <testcase classname="io.kevinlee.property_based_testing.MyAppTest" name="add(2, 2) should add both Int values and return 4" time="0.02">
    <failure message="4 was not equal to 41" type="org.scalatest.exceptions.TestFailedException">org.scalatest.exceptions.TestFailedException: 4 was not equal to 41
  at org.scalatest.MatchersHelper$.indicateFailure(MatchersHelper.scala:340)
  at org.scalatest.Matchers$ShouldMethodHelper$.shouldMatcher(Matchers.scala:6668)
  at org.scalatest.Matchers$AnyShouldWrapper.should(Matchers.scala:6704)
  at io.kevinlee.property_based_testing.MyAppTest.$anonfun$new$2(MyAppTest.scala:17)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  at org.scalatest.WordSpecLike$$anon$1.apply(WordSpecLike.scala:1078)
  at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
  at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
  at org.scalatest.WordSpec.withFixture(WordSpec.scala:1881)
  at org.scalatest.WordSpecLike.invokeWithFixture$1(WordSpecLike.scala:1076)
  at org.scalatest.WordSpecLike.$anonfun$runTest$1(WordSpecLike.scala:1088)
  at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
  at org.scalatest.WordSpecLike.runTest(WordSpecLike.scala:1088)
  at org.scalatest.WordSpecLike.runTest$(WordSpecLike.scala:1070)
  at org.scalatest.WordSpec.runTest(WordSpec.scala:1881)
  at org.scalatest.WordSpecLike.$anonfun$runTests$1(WordSpecLike.scala:1147)
  at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
  at scala.collection.immutable.List.foreach(List.scala:388)
  at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
  at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:373)
  at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:410)
  at scala.collection.immutable.List.foreach(List.scala:388)
  at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
  at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
  at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
  at org.scalatest.WordSpecLike.runTests(WordSpecLike.scala:1147)
  at org.scalatest.WordSpecLike.runTests$(WordSpecLike.scala:1146)
  at org.scalatest.WordSpec.runTests(WordSpec.scala:1881)
  at org.scalatest.Suite.run(Suite.scala:1147)
  at org.scalatest.Suite.run$(Suite.scala:1129)
  at org.scalatest.WordSpec.org$scalatest$WordSpecLike$$super$run(WordSpec.scala:1881)
  at org.scalatest.WordSpecLike.$anonfun$run$1(WordSpecLike.scala:1192)
  at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
  at org.scalatest.WordSpecLike.run(WordSpecLike.scala:1192)
  at org.scalatest.WordSpecLike.run$(WordSpecLike.scala:1190)
  at org.scalatest.WordSpec.run(WordSpec.scala:1881)
  at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
  at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
  at sbt.TestRunner.runTest$1(TestFramework.scala:113)
  at sbt.TestRunner.run(TestFramework.scala:124)
  at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:282)
  at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:246)
  at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
  at sbt.TestFramework$$anon$2$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:282)
  at sbt.TestFunction.apply(TestFramework.scala:294)
  at sbt.Tests$.$anonfun$toTask$1(Tests.scala:309)
  at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:46)
  at sbt.std.Transform$$anon$4.work(System.scala:67)
  at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
  at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
  at sbt.Execute.work(Execute.scala:278)
  at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
  at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
  at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:834)
    </failure>
  </testcase>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

As you can see, ScalaCheck uses java.lang.Exception whereas ScalaTest uses its own example named org.scalatest.exceptions.TestFailedException. I made Hedgehog use java.lang.AssertionError for failure message but not sure what it should be.
Do you have any suggestion?

charleso · 2019-07-16T22:53:09Z

Sorry for the delay @kevin-lee. Doesn't seem like there's much we can do. Here is the line of code that prints things:

https://github.com/sbt/sbt/blob/d4df289f2d6a0b8f6582346f331cd44408112c95/testing/src/main/scala/sbt/JUnitXmlTestsListener.scala#L128

The only option would be to override "printStackTrace" on a custom exception class. I would be tempted to do that to be honest. (Dodgy, I know)

Otherwise, or in any case, did you mind linking to the code above and I'll merge after that.

kevin-lee · 2019-07-17T00:00:54Z

@charleso Having a custom exception with printStackTrace muted seems fine at least for now. I'll do it. Thanks.

kevin-lee · 2019-07-17T13:38:06Z

@charleso I made hedgehog.sbt.StackTraceMutedException and now it looks like this. Any better name suggestion?

<?xml version='1.0' encoding='UTF-8'?>
<testsuite hostname="K-MP.local" name="io.kevinlee.property_based_testing.MyAppSpecWithHedgehog" tests="1" errors="0" failures="1" skipped="0" time="0.023" timestamp="2019-07-17T23:30:44">
  <properties>
    <!-- omitted -->
  </properties>
  <testcase classname="io.kevinlee.property_based_testing.MyAppSpecWithHedgehog" name="add(a, b) should return a + b" time="0.023">
    <failure message="x: 1
y: 1
=== Not Equal ===
2
3" type="hedgehog.sbt.StackTraceMutedException"></failure>
  </testcase>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

charleso · 2019-07-17T23:05:11Z

I'm easy, but I would be tempted to rename it MessageOnlyException which is the same as the one in sbt itself (except that we shouldn't use that one).

If you prefer StackTraceMutedException I'm happy to merge this.

charleso · 2019-07-17T23:07:38Z

sbt-test/src/main/scala/hedgehog/sbt/Framework.scala

+
+/** 
+  * This exception ignores printStackTrace with the given PrintStream or PrintWriter argument
+  * in order to avoid printing noisy and useless stacktrace


I would necessarily say "useless" here. It's more "misleading" because the stacktrace is coming from here and not the assertion. :)

Ah yeah you're right. I'll fix it.

kevin-lee · 2019-07-17T23:20:14Z

@charleso That's good. My first choice was MessageOnlyException but changed as it's used in sbt.
I'll change it. Thanks.

… failure

kevin-lee force-pushed the 92-fix-missing-failure-message-in-test-report branch from 3956221 to 5420948 Compare July 17, 2019 13:37

kevin-lee force-pushed the 92-fix-missing-failure-message-in-test-report branch from 5420948 to f5e5f38 Compare July 17, 2019 13:53

charleso reviewed Jul 17, 2019

View reviewed changes

Close hedgehogqa#92 - Test report does not have any message from test…

de8785c

… failure

kevin-lee force-pushed the 92-fix-missing-failure-message-in-test-report branch from f5e5f38 to de8785c Compare July 17, 2019 23:24

charleso merged commit 1aed18f into hedgehogqa:master Jul 17, 2019

kevin-lee deleted the 92-fix-missing-failure-message-in-test-report branch July 18, 2019 11:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Close #92 - Test report does not have any message from test failure #93

Close #92 - Test report does not have any message from test failure #93

kevin-lee commented Jul 14, 2019 •

edited

Loading

charleso commented Jul 16, 2019

kevin-lee commented Jul 17, 2019

kevin-lee commented Jul 17, 2019

charleso commented Jul 17, 2019

charleso Jul 17, 2019 •

edited

Loading

kevin-lee Jul 17, 2019

kevin-lee commented Jul 17, 2019

Close #92 - Test report does not have any message from test failure #93

Close #92 - Test report does not have any message from test failure #93

Conversation

kevin-lee commented Jul 14, 2019 • edited Loading

charleso commented Jul 16, 2019

kevin-lee commented Jul 17, 2019

kevin-lee commented Jul 17, 2019

charleso commented Jul 17, 2019

charleso Jul 17, 2019 • edited Loading

Choose a reason for hiding this comment

kevin-lee Jul 17, 2019

Choose a reason for hiding this comment

kevin-lee commented Jul 17, 2019

kevin-lee commented Jul 14, 2019 •

edited

Loading

charleso Jul 17, 2019 •

edited

Loading