New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Set sorting reporter as complete if test run failed unexpectedly #2161
Set sorting reporter as complete if test run failed unexpectedly #2161
Conversation
Hi @nimataheri-hootsuite, we require contributors to sign our Contributor License Agreement, and we don't have yours on file. In order for us to review and merge your code, please access https://www.artima.com/cla/choose-type to sign our Contributor License Agreement. Your effort is highly appreciated. Thank you. |
If test runner is of sorted type, it buffers all events and eventually emits/prints them upon completion of the test. Now, in case test runner fails unexpectedly it does not flush buffered events until an internal timer kicks off; this won't work with SBT system as all loggers of a task are discarded by the time that internal timer is triggered.
bd68eb2
to
3e83214
Compare
@nimatrueway This PR looks good to me, thanks! @bvenners Unfortunately this PR contains a lot of whitespace diff, in order to filter out whitespace diff lines you may apply the filter like this: |
@cheeseng thanks for reviewing it! I can remove all whitespace changes if it helps but I figured this formatting will happen eventually. |
case e: Throwable => { | ||
suiteSortingReporter.completedTests(suite.suiteId) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we call this again after the timer expires. It sounds like it may not matter with running with sbt, but I wonder if that causes issues when running not with sbt.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey Bill!
completedTests
method invokes fireReadyEvents
which in turn calls cancelTimeoutTask
to cancel the subject timer. Whether SuiteSortingReporter#timeout task is thread-safe (in a race with completedTests
) is a more difficult question which I'm afraid I can't answer. However, that could happen regardless of this PR's change I believe. Namely, a test suite completes in the happy code path and simultaneously SuiteSortingReporter#timeout task gets fired after 2 sec.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bvenners @nimatrueway Framework.scala is called by sbt only, anyway, that also raises a question if we should do the similar when run with Runner, for Runner I think the similar spot is in SuiteRunner.scala:
I am not sure how we can call completedTests there, we probably shouldn't. Different from the sbt runner, the ScalaTest Runner will wait until the timeout and got the SuiteAborted eventually, I think.
If test runner reporter is of "sorted" type, it buffers all events and eventually emits/prints them upon completion of the test. Now, in case test runner fails unexpectedly it does not flush buffered events until an internal timer kicks off. This won't work with SBT system as all loggers of a task are already discarded when a task is finished.
Sample
Before this PR

After this PR
