Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Original error message is replaced when using TimeLimitedTests trait #1421

Open
NikolajsCmihuns opened this issue Oct 11, 2018 · 2 comments
Open

Comments

@NikolajsCmihuns
Copy link

Original error message replaced when using def org.scalatest.concurrent.Eventually.eventually and org.scalatest.concurrent.TimeLimitedTests trait.

Use this project as example: https://github.com/NikolajsCmihuns/test_scalatest
test files - /test_scalatest/specs/src/test/scala/*.scala

Output from TimeLimitedSpec (mixes in TimeLimitedTests trait):

[info] Custom error messages
[info] - must be preserved when test fails *** FAILED *** (2 seconds, 59 milliseconds)
[info] The test did not complete within the specified 2 seconds time limit. (TimeLimitedSpec.scala:9)
[info] org.scalatest.exceptions.TestFailedDueToTimeoutException:
[info] at org.scalatest.concurrent.Eventually.tryTryAgain$1(Eventually.scala:432)
[info] at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:439)
[info] at org.scalatest.concurrent.Eventually.eventually$(Eventually.scala:391)
[info] at TimeLimitedSpec.eventually(TimeLimitedSpec.scala:4)
........
[info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[info] at java.lang.Thread.run(Thread.java:748)
[info] Cause: org.scalatest.exceptions.TestFailedException: My error message
......

Output from WithoutTimeLimitSpec (does not mix in TimeLimitedTests trait):

[info] - must be preserved when test fails *** FAILED *** (2 seconds, 58 milliseconds)
[info] The code passed to eventually never returned normally. Attempted 14 times over 2.041572782 seconds. Last failure message: My error message. (WithoutTimeLimitSpec.scala:6)
[info] org.scalatest.exceptions.TestFailedDueToTimeoutException:
[info] at org.scalatest.concurrent.Eventually.tryTryAgain$1(Eventually.scala:432)
[info] at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:439)

TimeLimitedSpec has failing eventually { assert(1 == 2, "My error message") } assertion(retries 2 sec.) and timeLimit(5 sec.) which is not breached. BUT in output custom error message is displayed in stacktrace bottom and also message on top of output is The test did not complete within the specified 2 seconds time limit which is not true in this case - there was not time out breach and time out set is not even 2 seconds, it is 5 seconds.

Such behaviour also makes scalatest html report (see TimeLimitedSpec_html.png below) incorrect by hiding original message completely (this is especially bad when using scalatest to write automated selenium tests and custom error message are crucial)
TimeLimitedSpec_html_report.png
Message: in report should include My error message instead, which is the real cause

Look at WithoutTimeLimitSpec_html_report.png below
WithoutTimeLimitSpec_html_report.png
Message: in report includes My error message which is correct

Possible cause of decribed issue:
when using org.scalatest.concurrent.Eventually.eventually { ... } it throws TestFailedDueToTimeoutException exception. It then get's modified in

case e: org.scalatest.exceptions.ModifiableMessage[_] with TimeoutField =>

@NikolajsCmihuns NikolajsCmihuns changed the title Original error message replaced when using TimeLimitedTests trait Original error message is replaced when using TimeLimitedTests trait Oct 11, 2018
@NikolajsCmihuns
Copy link
Author

NikolajsCmihuns commented Oct 11, 2018

What about creating new custom exception type here (instead of TestFailedDueToTimeoutException let's say TestFailedDueToTimeLimitExceededException)

val e = new TestFailedDueToTimeoutException(

and then here match against it

case e: org.scalatest.exceptions.ModifiableMessage[_] with TimeoutField =>

Thus we will not update all exception that match against org.scalatest.exceptions.ModifiableMessage[_] with TimeoutField

@NikolajsCmihuns
Copy link
Author

@cheeseng @bvenners, please review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant