Cannot recreate RB experiment with job ID

[catornow]

Informations

• Qiskit Experiments version: current main
• Python version: 3.8.8
• Operating system: MacOS

What is the current behavior?

Running an RB experiment on the hardware

lengths = np.arange(1, 1000, 100)
num_samples = 10
seed = 1010
qubits = [3]

# Run an RB experiment on qubit 3
exp = StandardRB(qubits, lengths, num_samples=num_samples, seed=seed)

exp_data = exp.run(backend)

and retrieving the job using the job_id to recreate the RB experiment and perform the RB analysis

job_id = '6156ddf28fd55fe123fa9923'

job = backend.retrieve_job(job_id)

exp_data_retrieved = ExperimentData(exp)
exp_data_retrieved.add_data(job.result())

analysis = RBAnalysis()
analysis.run(exp_data_retrieved)

gives the following error message:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/Documents/qiskit-experiments/qiskit_experiments/curve_analysis/curve_analysis.py in _run_analysis(self, experiment_data, **options)
    865             try:
--> 866                 meas_level = run_options["meas_level"]
    867             except KeyError as ex:

KeyError: 'meas_level'

The above exception was the direct cause of the following exception:

DataProcessorError                        Traceback (most recent call last)
<ipython-input-6-2fc6fbd4f629> in <module>
      1 analysis = RBAnalysis()
----> 2 analysis.run(exp_data_retrieved)

~/Documents/qiskit-experiments/qiskit_experiments/framework/base_analysis.py in run(self, experiment_data, **options)
     87 
     88         # Run analysis
---> 89         results, figures = self._run_analysis(experiment_data, **analysis_options)
     90 
     91         # Add components

~/Documents/qiskit-experiments/qiskit_experiments/curve_analysis/curve_analysis.py in _run_analysis(self, experiment_data, **options)
    866                 meas_level = run_options["meas_level"]
    867             except KeyError as ex:
--> 868                 raise DataProcessorError(
    869                     f"Cannot process data without knowing the measurement level: {str(ex)}."
    870                 ) from ex

DataProcessorError: "Cannot process data without knowing the measurement level: 'meas_level'."

What is the expected behavior?

The RB analysis runs after recreating the RB experiment using the job ID.

[nkanazawa1989]

Thank you for the bug report. This is actually somewhat of expected behavior -- but we should support this at some point. The problem is you don't pass analysis options to run method. Usually, run method needs to receive **kwargs which are not type hinted. For now, you should use

exp.run_analysis(exp_data_retrieved)

This extracts default analysis options associated with the experiment instance, and passes the options to the analysis run method behind the scene. This means analysis class cannot run itself due to missing options. This looks bit awkward. Probably we should remove **kwargs from analysis.run and let the analysis retrieve options from experiment data. There is metadata in the experiment data that contains serialized analysis options.

[eggerdj]

Here are some additional limitations that I am seeing. I'll take RB for testing purposes but I imagine that other experiments will have the same issues. First, I create an experiment, run and save it.

import numpy as np

from qiskit import IBMQ
from qiskit_experiments.library import StandardRB

IBMQ.load_account()
provider = IBMQ.get_provider(hub=..., group=..., project=...)
backend = provider.get_backend(...)

lengths = np.arange(1, 1000, 100)
num_samples = 10
seed = 1010
qubits = [0]

# Run an RB experiment
exp = StandardRB(qubits, lengths, num_samples=num_samples, seed=seed)

exp_data = exp.run(backend).block_for_results()

exp_data.save()

At this point I'm done for the day and close everything up. I would expect that tomorrow I can reload the experiment and analyze it once again (perhaps with different options). Here is what I do:

import numpy as np

from qiskit import IBMQ
from qiskit_experiments.library import StandardRB
from qiskit_experiments.database_service import DbExperimentDataV1 as DbExperimentData

IBMQ.load_account()
provider = IBMQ.get_provider(hub=..., group=..., project=...)
backend = provider.get_backend(...)

lengths = np.arange(1, 1000, 100)
num_samples = 10
seed = 1010
qubits = [0]

load_exp = DbExperimentData.load("b8e1bfc0-1e93-406d-b691-e087163d9cd3", provider.service("experiment"))
exp2 = StandardRB(qubits, lengths, num_samples=num_samples, seed=seed)
exp2.run_analysis(load_exp)

The last step returns

QiskitError: 'Invalid experiment data type, expected ExperimentData but received DbExperimentDataV1'

Is this a bug, or did I miss a step? If this is a bug we should fix it. If I missed something I'd be happy to see that step explained in the tutorial: https://github.com/Qiskit/qiskit-experiments/blob/main/docs/tutorials/experiment_cloud_service.ipynb

[nkanazawa1989]

It seems like a bug. We check experiment data type for consistency, and that refers to the one in the class attribute. The problem here is attached class is the ExperimentData, which is a subclass of db one. So the loaded experiment data will be rejected (isinstance only checks superclass). Probably this is an issue of loader and we need metadata representing an experiment data subclass.

[chriseclectic]

Using fixes in PR #421 and #423 I can get the data to load and analysis to run in some cases, but others return analysis errors due to missing analysis options or metadata.

# Experiment I ran and saved data from
from qiskit_experiments.library import StandardRB

qubits = [0]
lengths = [2 ** i for i in range(10)]
exp = StandardRB(qubits, lengths=lengths)

Loading from service works re-running analysis

from qiskit import IBMQ

IBMQ.load_account()
provider = IBMQ.get_provider(group="deployed")
service = provider.service("experiment")

exp_id = "<some id>"
data = ExperimentData.load(exp_id, service=service)
data = data._copy_metadata() # hack to remove saved analysis results so we can try and rerun
exp.run_analysis(data) # Ok

Creating ExperimentData with original experiment and running analysis now works:

backend = provider.get_backend("ibmq_bogota")
job_id = "<some id>"
job = backend.retrieve_job(job_id)

data = ExperimentData(exp)
data.add_data(job)
data.block_for_results()
exp.run_analysis(data)

Loading ExperimentData directly from job id (without experiment) gives an analysis error:

data = ExperimentData(backend=backend, job_ids=[job_ids])
exp.run_analysis(data) # Error!

------------------------------------------------------------------------
TypeError                              Traceback (most recent call last)
<ipython-input-17-94cf07b004fc> in <module>
----> 1 exp.run_analysis(data1)

~/git/qiskit/qiskit-experiments/qiskit_experiments/framework/base_experiment.py in run_analysis(self, experiment_data, **options)
    204         # Run analysis
    205         analysis = self.analysis()
--> 206         analysis.run(experiment_data, **analysis_options)
    207         return experiment_data
    208 

~/git/qiskit/qiskit-experiments/qiskit_experiments/framework/base_analysis.py in run(self, experiment_data, **options)
     87 
     88         # Run analysis
---> 89         results, figures = self._run_analysis(experiment_data, **analysis_options)
     90 
     91         # Add components

~/git/qiskit/qiskit-experiments/qiskit_experiments/curve_analysis/curve_analysis.py in _run_analysis(self, experiment_data, **options)
    898         )
    899 
--> 900         fit_options = self._generate_fit_guesses(default_fit_opt)
    901         if isinstance(fit_options, FitOptions):
    902             fit_options = [fit_options]

~/git/qiskit/qiskit-experiments/qiskit_experiments/library/randomized_benchmarking/rb_analysis.py in _generate_fit_guesses(self, user_opt)
    111         )
    112 
--> 113         return self._initial_guess(user_opt, curve_data.x, curve_data.y, self._num_qubits)
    114 
    115     @staticmethod

~/git/qiskit/qiskit-experiments/qiskit_experiments/library/randomized_benchmarking/rb_analysis.py in _initial_guess(opt, x_values, y_values, num_qubits)
    118     ) -> curve.FitOptions:
    119         """Create initial guess with experiment data."""
--> 120         opt.p0.set_if_empty(b=1 / 2 ** num_qubits)
    121 
    122         # Use the first two points to guess the decay param

TypeError: unsupported operand type(s) for ** or pow(): 'int' and 'NoneType'

My guess this is an issue with the RB Analysis class itself having missing default values that can work for the fitter.

Finally, trying to use the RBAnalysis class directly, instead of exp.run_analysis gives the original error relating to meas_level in data processing in this issue.

data = ExperimentData(exp)
data.add_data(job)
data.block_for_results()
exp.analysis().run(data)

I guess this one is from a bug in DataProcessor / CurveAnalysis that could be fixed

[nkanazawa1989]

Meas level is an issue of data processor. However, the RB analysis requires physical qubits (number of qubits) that cannot be hardcoded to the analysis class. If you really want to hardcode everything necessary for RB analysis (as default options), we need RB1Q, RB2Q, RB1QAnalysis, RB2QAnalysis, etc... I think this is not clean solution. Probably we need to upgrade initial guess not to depend on qubit numbers. Also EPG calculation should be done in the same way with calibration updader.

[chriseclectic]

For RB num qubits could easily be inferred from the count_ops circuit metadata during analysis. Currently it actually stores physical_qubits too so you could get it from there as well, but storing this maybe unnecessary in the first place.

[yaelbh]

@gadial I see that you've assigned yourself on this issue. What's the status? Do you intend to work on it?

[chriseclectic]

Closing this to replace with #644 since I think most of the remaining issues are covered by that general case.