Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple Errors in a chain of ScriptRunner Methods #975

aaronatball opened this issue Apr 12, 2019 · 3 comments

Multiple Errors in a chain of ScriptRunner Methods #975

aaronatball opened this issue Apr 12, 2019 · 3 comments


Copy link

@aaronatball aaronatball commented Apr 12, 2019

It looks like there is an minor issue with raising errors in ScriptRunner/TestRunner, specifically when an error happens in lower methods while ScriptRunnerFrame.@continue_after_error is false.

Basically, if something like a CheckError is raised, the exception_instrumentation in the instrumented code handles the error as intended, but then the error is raised up to the next level, and that line's exception_instrumentation handles the error again. It duplicates the error for each nested level, so if you are 2 levels down, you get the error 3 times.

That error is added to ScriptRunnerFrame.@Exceptions each time, so it looks like the error happened more times than intended.

This does not occur when @ScriptRunnerFrame.@continue_after_error is true (the default) because this error is not raised:

Here is a TestRunner Suite that replicates this behavior (when run with 'Continue Test Case after Error' not checked):

load 'cosmos/tools/test_runner/test.rb'

class ErrorRaiseTestSuite < Cosmos::TestSuite
  def initialize()

class MultiErrorRaise < Cosmos::Test
  def test_case_one

  def method_one

  def method_two

I think the fix is actually fairly simple, but I wanted to discuss it before making a PR in case there is a different way you want to handle it.

ScriptRunnerFrame.@@error will be the last error received, so in exception_instrumentation, if the error passed to the method has the same object_id of @@errors, then we know it's a chain of the same error bubbling up, and we can skip this line since it was already handled by a child method:

Copy link

@jasonatball jasonatball commented Jun 5, 2019

I'm not able to reproduce this. I ran your example and get the following Script Runner output:

2019/06/05 08:50:10.880 (SCRIPTRUNNER): Starting script: ErrorRaiseTestSuite_MultiErrorRaise_test_case_one
2019/06/05 08:50:12.094 (multi_error.rb:20):<R> ERROR: CHECK: 1==2 is FALSE
2019/06/05 08:50:12.141 (SCRIPTRUNNER): Script completed: ErrorRaiseTestSuite_MultiErrorRaise_test_case_one

And Test Runner output:

--- Test Report ---

Report Filename: C:/git/COSMOS/demo/outputs/logs/2019_06_05_08_50_11_testrunner_results.txt
Detailed Test Output Logged to: C:/git/COSMOS/demo/outputs/logs/2019_06_05_08_50_10_sr_ErrorRaiseTestSuite_MultiErrorRaise_test_case_one_messages.txt

USER_VERSION = Unofficial
RUBY_VERSION = 2.4.4p296
OPERATOR_NAME = Unspecified

Pause on Error = true
Continue Test Case after Error = false
Abort Testing after Error = false
Manual = true
Loop Testing = false
Break Loop after Error = false

2019/06/05 08:50:11.970: Executing ErrorRaiseTestSuite:MultiErrorRaise:test_case_one 
2019/06/05 08:50:12.128: MultiErrorRaise:test_case_one:PASS
2019/06/05 08:50:12.180: Completed ErrorRaiseTestSuite:MultiErrorRaise:test_case_one 

--- Test Summary ---

Run Time : 0.21 seconds
Total Tests : 1
Pass : 1
Skip : 0
Fail : 0

Copy link
Contributor Author

@aaronatball aaronatball commented Jun 17, 2019


Is that the output when 'Continue Test Case after Error' is disabled?

My output matches yours when it is enabled, but it gets the multiple error thing when it's disabled, and I think the bug in this case only applies when it's disabled.

Copy link

@jasonatball jasonatball commented Jun 26, 2019

I was able to reproduce when I had Continue Test Case after Error AND Pause on Error unchecked. When I had Pause on Error checked it didn't reproduce. I think that is due to this line:

@jasonatball jasonatball self-assigned this Jun 26, 2019
@ryanatball ryanatball added this to the v4.4.0 milestone Jul 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants