Cross-resonance integration tests fail in my environment

[yaelbh]

(env1) hrlquantum:System$ stestr run -n test/test_cross_resonance_hamiltonian.py 
{0} test.test_cross_resonance_hamiltonian.TestCrossResonanceHamiltonian.test_circuit_generation [0.041296s] ... ok
{0} test.test_cross_resonance_hamiltonian.TestCrossResonanceHamiltonian.test_experiment_config [0.006548s] ... ok
{0} test.test_cross_resonance_hamiltonian.TestCrossResonanceHamiltonian.test_integration_1__1000000_0__2000000_0__1000_0___3000000_0___2000000_0__10000_0_ [604.556753s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/hrlquantum/miniconda3/envs/env1/lib/python3.9/site-packages/ddt.py", line 182, in wrapper
    return func(self, *args, **kwargs)

      File "/home/hrlquantum/work/experiments/System/test/test_cross_resonance_hamiltonian.py", line 250, in test_integration
    self.assertExperimentDone(exp_data, timeout=600)

      File "/home/hrlquantum/work/experiments/System/test/base.py", line 56, in assertExperimentDone
    msg="All threads are executed but status is not DONE. " + experiment_data.errors(),

      File "/home/hrlquantum/work/experiments/System/qiskit_experiments/database_service/db_experiment_data.py", line 1400, in errors
    return self.job_errors() + self.analysis_errors()

      File "/home/hrlquantum/work/experiments/System/qiskit_experiments/database_service/db_experiment_data.py", line 1382, in analysis_errors
    ex = fut.exception()

    AttributeError: 'NoneType' object has no attribute 'exception'

{0} test.test_cross_resonance_hamiltonian.TestCrossResonanceHamiltonian.test_integration_2___1000000_0___2000000_0__1000_0__3000000_0__2000000_0__10000_0_ [504.584740s] ... ok
{0} test.test_cross_resonance_hamiltonian.TestCrossResonanceHamiltonian.test_integration_3__10000_0__20000_0__1000_0__5000000_0__1000000_0__2000_0_ [257.341428s] ... ok
{0} test.test_cross_resonance_hamiltonian.TestCrossResonanceHamiltonian.test_integration_4__10000_0___1000_0__1000_0__500000_0__1000_0___1000_0_ [188.185892s] ... ok
{0} test.test_cross_resonance_hamiltonian.TestCrossResonanceHamiltonian.test_integration_5___100000_0__120000_0__1000_0__150000_0___110000_0___1000_0_ [96.004021s] ... ok
{0} test.test_cross_resonance_hamiltonian.TestCrossResonanceHamiltonian.test_roundtrip_serializable [0.006110s] ... ok

==============================
Failed 1 tests - output below:
==============================

test.test_cross_resonance_hamiltonian.TestCrossResonanceHamiltonian.test_integration_1__1000000_0__2000000_0__1000_0___3000000_0___2000000_0__10000_0_
------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/hrlquantum/miniconda3/envs/env1/lib/python3.9/site-packages/ddt.py", line 182, in wrapper
    return func(self, *args, **kwargs)

      File "/home/hrlquantum/work/experiments/System/test/test_cross_resonance_hamiltonian.py", line 250, in test_integration
    self.assertExperimentDone(exp_data, timeout=600)

      File "/home/hrlquantum/work/experiments/System/test/base.py", line 56, in assertExperimentDone
    msg="All threads are executed but status is not DONE. " + experiment_data.errors(),

      File "/home/hrlquantum/work/experiments/System/qiskit_experiments/database_service/db_experiment_data.py", line 1400, in errors
    return self.job_errors() + self.analysis_errors()

      File "/home/hrlquantum/work/experiments/System/qiskit_experiments/database_service/db_experiment_data.py", line 1382, in analysis_errors
    ex = fut.exception()

    AttributeError: 'NoneType' object has no attribute 'exception'



======
Totals
======
Ran: 8 tests in 1650.7602 sec.
 - Passed: 7
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 1
Sum of execute time for each test: 1650.7268 sec.

==============
Worker Balance
==============
 - Worker 0 (8 tests) => 0:27:30.760168

Comments:

1. This happens consistently.
2. Note the long run times also in the tests that pass.
3. More cross-resonance integration tests fail when I run the entire test suite (stestr run) .
4. I guess that I run into timeout, and that there's a bug in the exception raised by the timeout.

[yaelbh]

Happens also in the CI: https://github.com/Qiskit/qiskit-experiments/runs/5217453192?check_suite_focus=true

[nkanazawa1989]

Yeah probably I underestimated the execution time. This is much shorter in my laptop. I know the bottleneck of the current implementation. Probably I'll open the PR to fix this but meanwhile you can just increase this number
https://github.com/Qiskit/qiskit-experiments/blob/317b964b8cbde0febcda8696fd4bd5cce3490e0a/test/test_cross_resonance_hamiltonian.py#L250

[yaelbh]

@nkanazawa1989 Can we close this issue?

[nkanazawa1989]

No, still we need group analysis to fix the performance issue. I'm working on it.