You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is more a question than an issue, really, but I'm trying to understand why the patched version of stopTestalways calls addSuccess, regardless of the state of the actual test run.
In our (Datadog's CI Visibility integration of unittest) case, this makes us incorrectly report a test as passing even if it fails, since we consider any call of addSuccess in the test's run as an indication the test succeeded, and since flexmock wraps the original test run.
For us, I see reporting the last result update (whether it's addSuccess or any other) as more appropriate, since it'll properly deal with cases where someone decides to re-run a test until it succeeds.
We can come up with workarounds (generalized or more specific to flexmock) (eg: maybe ignore results if stopTest is in the call stack, since that shouldn't be providing results according to the original unittest case class), so I'm genuinely more curious about why addSuccess is called unconditionally.
The text was updated successfully, but these errors were encountered:
It seems that that change was added 12 years ago in this commit: 670609e
I'm not actually sure why addSuccess is called at all because the default implementation doesn't do anything (unittest.TestResult.addSuccess) and addSuccess should have been called already before stopTest is called. My guess is that it was added for nose integration, but nose support has been dropped so we could just remove the addSuccess call.
This is more a question than an issue, really, but I'm trying to understand why the patched version of
stopTest
always callsaddSuccess
, regardless of the state of the actual test run.In our (Datadog's CI Visibility integration of
unittest
) case, this makes us incorrectly report a test as passing even if it fails, since we consider any call ofaddSuccess
in the test's run as an indication the test succeeded, and since flexmock wraps the original test run.For us, I see reporting the last result update (whether it's
addSuccess
or any other) as more appropriate, since it'll properly deal with cases where someone decides to re-run a test until it succeeds.We can come up with workarounds (generalized or more specific to flexmock) (eg: maybe ignore results if
stopTest
is in the call stack, since that shouldn't be providing results according to the originalunittest
case class), so I'm genuinely more curious about whyaddSuccess
is called unconditionally.The text was updated successfully, but these errors were encountered: