Tests pass although analysis crashes

[yaelbh]

Insert

div0 = 6 / 0

at the beginning of RBAnalysis._generate_fit_guesses. Then:

env1) hrlquantum:System$ stestr run -n test/randomized_benchmarking/test_rb_analysis.py 
{0} test.randomized_benchmarking.test_rb_analysis.TestInterleavedRBAnalysis.test_interleaved_rb_analysis_test [0.170752s] ... ok
Analysis callback <function BaseAnalysis.run.<locals>.run_analysis at 0x7f0392195e50> failed:
Traceback (most recent call last):
  File "/home/hrlquantum/work/experiments/System/qiskit_experiments/database_service/db_experiment_data.py", line 298, in _wrapped_callback
    callback(self, **kwargs)
  File "/home/hrlquantum/work/experiments/System/qiskit_experiments/framework/base_analysis.py", line 105, in run_analysis
    results, figures = self._run_analysis(expdata, **analysis_options)
  File "/home/hrlquantum/work/experiments/System/qiskit_experiments/curve_analysis/curve_analysis.py", line 935, in _run_analysis
    fit_options = self._generate_fit_guesses(default_fit_opt)
  File "/home/hrlquantum/work/experiments/System/qiskit_experiments/library/randomized_benchmarking/rb_analysis.py", line 105, in _generate_fit_guesses
    div0 = 6 / 0
ZeroDivisionError: division by zero

Possibly incomplete analysis results: an analysis callback raised an error.
Analysis callback <function BaseAnalysis.run.<locals>.run_analysis at 0x7f03920a2a60> failed:
Traceback (most recent call last):
  File "/home/hrlquantum/work/experiments/System/qiskit_experiments/database_service/db_experiment_data.py", line 298, in _wrapped_callback
    callback(self, **kwargs)
  File "/home/hrlquantum/work/experiments/System/qiskit_experiments/framework/base_analysis.py", line 105, in run_analysis
    results, figures = self._run_analysis(expdata, **analysis_options)
  File "/home/hrlquantum/work/experiments/System/qiskit_experiments/curve_analysis/curve_analysis.py", line 935, in _run_analysis
    fit_options = self._generate_fit_guesses(default_fit_opt)
  File "/home/hrlquantum/work/experiments/System/qiskit_experiments/library/randomized_benchmarking/rb_analysis.py", line 105, in _generate_fit_guesses
    div0 = 6 / 0
ZeroDivisionError: division by zero

Possibly incomplete analysis results: an analysis callback raised an error.
{0} test.randomized_benchmarking.test_rb_analysis.TestStandardRBAnalysis.test_standard_rb_analysis_test [0.022537s] ... ok

======
Totals
======
Ran: 2 tests in 0.1946 sec.
 - Passed: 2
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 0.1933 sec.

==============
Worker Balance
==============
 - Worker 0 (2 tests) => 0:00:00.194626

[chriseclectic]

Catching broad exceptions is by design so that experiments are run like jobs, but this is not good for catching test failures.

I think what we need here is a test case added to the base QiskitExperimentsTestCase class that checks.

def assertSuccess(self, experiment_data):
    experiment_data.block_for_results()
    assertEqual(experiment_data.status(), "DONE", msg=experiment_data.errors())

The in each unit test you need to do:

...
expdata = experiment.run(...)
self.assertSuccess(expdata)
# rest of test checks...

Qiskit-Aer does something similar for all its unittests to check that the simulation result was a success before checking the specific result checks for each unit test case.