Unexpected Successes do not result in overall failure

[electrofelix]

Issue description

It appears that tests that pass unexpectedly, when marked with @unittest.expectedFailure do not trigger an overall test suite failure.

Based on changes in python3.4 from the fix landing for https://bugs.python.org/issue20165 this is rather unexpected.

Expected behavior and actual behavior

Mark any successful test with the decorator @unittest.expectedFailure
Run the tests using stestr run --slowest
Exit code should be non-zero

Actual behaviour is that current exit code is zero (0)

Steps to reproduce the problem

Mark any successful test with the decorator @unittest.expectedFailure and run tests with stestr.

Specifications like the version of the project, operating system, or hardware

System information

stestr version (stestr --version): stestr 2.1.0

Python release (python --version): Python 3.5.2

pip packages (pip freeze):
-f /var/cache/pip
cliff==2.13.0
cmd2==0.9.2
colorama==0.3.9
extras==1.0.0
fixtures==3.0.0
flake8==3.5.0
future==0.16.0
linecache2==1.0.0

mccabe==0.6.1
pbr==4.1.0
pkg-resources==0.0.0
prettytable==0.7.2
pycodestyle==2.3.1
pyflakes==1.6.0
pyparsing==2.2.0
pyperclip==1.6.2
python-mimeparse==1.6.0
python-subunit==1.3.0
PyYAML==3.13
six==1.11.0
stestr==2.1.0
stevedore==1.28.0
testtools==2.3.0
traceback2==1.4.0
unittest2==1.1.0
voluptuous==0.11.1
wcwidth==0.1.7

Additional information

It would appear that testtools TestResult class contains a method that has the correct behaviour at https://github.com/testing-cabal/testtools/blob/master/testtools/testresult/real.py#L174 and it would be possible to fix this in stestr by changing from calling summary_result.wasSuccessful() at https://github.com/mtreinish/stestr/blob/master/stestr/commands/load.py#L222 to call testtools.TestResult.wasSuccessful(summary_result) instead to resolve. There would also be a few other places that would need updating.

Based on looking at the tests in testing-cabal/testtools It may be that testtools is maintaining backwards compatibility with unittest.TestResult prior to the fix landing in python 3.4 in the StreamSummary class and downstream projects at expected to handle this instead. Will log an issue there as well referencing this issue to try and determine where is the correct place to submit a fix for this.

[mtreinish]

I'm fine with switching over all the wasSuccessful() calls to use the TestResult method now. Even if that is treated as bug in testtools, the time to update StreamResult.wasSuccessful() and get that in a release is probably going to be a while. So in the meantime I'd like to fix this on the stestr side and get a bugfix release out the door. @electrofelix do you want to push a patch for switching things over?

If/when testtools pushes out a release with a fix for StreamResult we can go back and switch back all the calls.

[electrofelix]

@mtreinish I'll try to put together a PR sometime tomorrow or Monday