Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple Errors in a chain of ScriptRunner Methods #975

Closed
aaronatball opened this issue Apr 12, 2019 · 3 comments

Comments

Projects
None yet
3 participants
@aaronatball
Copy link
Contributor

commented Apr 12, 2019

It looks like there is an minor issue with raising errors in ScriptRunner/TestRunner, specifically when an error happens in lower methods while ScriptRunnerFrame.@continue_after_error is false.

Basically, if something like a CheckError is raised, the exception_instrumentation in the instrumented code handles the error as intended, but then the error is raised up to the next level, and that line's exception_instrumentation handles the error again. It duplicates the error for each nested level, so if you are 2 levels down, you get the error 3 times.

That error is added to ScriptRunnerFrame.@Exceptions each time, so it looks like the error happened more times than intended.

This does not occur when @ScriptRunnerFrame.@continue_after_error is true (the default) because this error is not raised: https://github.com/BallAerospace/COSMOS/blob/master/lib/cosmos/tools/script_runner/script_runner_frame.rb#L1639

Here is a TestRunner Suite that replicates this behavior (when run with 'Continue Test Case after Error' not checked):

load 'cosmos/tools/test_runner/test.rb'

class ErrorRaiseTestSuite < Cosmos::TestSuite
  def initialize()
    super()
    add_test("MultiErrorRaise")
  end
end

class MultiErrorRaise < Cosmos::Test
  def test_case_one
    method_one
  end

  def method_one
    method_two
  end

  def method_two
    check_expression("1==2")
  end
end

I think the fix is actually fairly simple, but I wanted to discuss it before making a PR in case there is a different way you want to handle it.

ScriptRunnerFrame.@@error will be the last error received, so in exception_instrumentation, if the error passed to the method has the same object_id of @@errors, then we know it's a chain of the same error bubbling up, and we can skip this line since it was already handled by a child method: https://github.com/BallAerospace/COSMOS/blob/master/lib/cosmos/tools/script_runner/script_runner_frame.rb#L679

@jasonatball

This comment has been minimized.

Copy link
Collaborator

commented Jun 5, 2019

I'm not able to reproduce this. I ran your example and get the following Script Runner output:

2019/06/05 08:50:10.880 (SCRIPTRUNNER): Starting script: ErrorRaiseTestSuite_MultiErrorRaise_test_case_one
2019/06/05 08:50:12.094 (multi_error.rb:20):<R> ERROR: CHECK: 1==2 is FALSE
2019/06/05 08:50:12.141 (SCRIPTRUNNER): Script completed: ErrorRaiseTestSuite_MultiErrorRaise_test_case_one

And Test Runner output:

--- Test Report ---

Files:
Report Filename: C:/git/COSMOS/demo/outputs/logs/2019_06_05_08_50_11_testrunner_results.txt
Detailed Test Output Logged to: C:/git/COSMOS/demo/outputs/logs/2019_06_05_08_50_10_sr_ErrorRaiseTestSuite_MultiErrorRaise_test_case_one_messages.txt

Metadata:
COSMOS_VERSION = 4.3.0
USER_VERSION = Unofficial
RUBY_VERSION = 2.4.4p296
OPERATOR_NAME = Unspecified

Settings:
Pause on Error = true
Continue Test Case after Error = false
Abort Testing after Error = false
Manual = true
Loop Testing = false
Break Loop after Error = false

Results:
2019/06/05 08:50:11.970: Executing ErrorRaiseTestSuite:MultiErrorRaise:test_case_one 
2019/06/05 08:50:12.128: MultiErrorRaise:test_case_one:PASS
2019/06/05 08:50:12.180: Completed ErrorRaiseTestSuite:MultiErrorRaise:test_case_one 

--- Test Summary ---

Run Time : 0.21 seconds
Total Tests : 1
Pass : 1
Skip : 0
Fail : 0
@aaronatball

This comment has been minimized.

Copy link
Contributor Author

commented Jun 17, 2019

@jasonatball

Is that the output when 'Continue Test Case after Error' is disabled?

My output matches yours when it is enabled, but it gets the multiple error thing when it's disabled, and I think the bug in this case only applies when it's disabled.

@jasonatball

This comment has been minimized.

Copy link
Collaborator

commented Jun 26, 2019

I was able to reproduce when I had Continue Test Case after Error AND Pause on Error unchecked. When I had Pause on Error checked it didn't reproduce. I think that is due to this line: https://github.com/BallAerospace/COSMOS/blob/master/lib/cosmos/tools/script_runner/script_runner_frame.rb#L1649.

@jasonatball jasonatball added bug and removed possible bug labels Jun 26, 2019

@jasonatball jasonatball self-assigned this Jun 26, 2019

@ryanatball ryanatball added this to the v4.4.0 milestone Jul 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.