metriq-gym is a Python framework for implementing and running standard quantum benchmarks on different quantum devices by different providers.
- Open – Open-source since its inception and fully developed in public.
- Transparent – All benchmark parameters are defined in a schema file and the benchmark code is reviewable by the community.
- Cross-platform – Supports running benchmarks on multiple quantum hardware providers (integration powered by qBraid-SDK)
- User-friendly – Provides a simple command-line interface for dispatching, monitoring, and polling benchmark jobs (you can go on with your life while your job waits in the queue).
Data generated by metriq-gym is intended to be published on https://metriq.info.
Join the conversation!
- For code, repo, or theory questions, especially those requiring more detailed responses, submit a Discussion.
- For casual or time sensitive questions, chat with us on Discord's
#metriqchannel.
You will require Python 3.12 or above, and poetry.
When cloning the metriq-gym repository use:
git clone --recurse-submodules https://github.com/unitaryfoundation/metriq-gym.gitThis allows you to fetch qiskit-device-benchmarking as a git submodule for a set of some of the IBM benchmarks.
Once you have poetry installed and the repository cloned, run:
poetry installfrom the root folder of the project, in order to install the project dependencies. We recommend doing this in an isolated virtual environment. See Poetry documentation for more information on managing virtual environments.
If you use pyenv, here is a quick start guide to set up the environment and install all dependencies:
pyenv install 3.13
pyenv local 3.13
poetry install
eval $(poetry env activate)All Python commands below should be run in the virtual environment.
To run on hardware, each hardware provider offers API tokens that are required to interact with their quantum devices. In order to run on these devices, you will need to follow the instructions on the respective pages of the providers and obtain API keys from them.
The .env.example file illustrates how to specify the API keys once you have acquired them. You will need to create a
.env file in the same directory as .env.example and populate the values of these variables accordingly.
You can dispatch a job by specifying the parameters of the job you wish to launch in a configuration file.
mgym dispatch <BENCHMARK_JSON> --provider <PROVIDER> --device <DEVICE>Refer to the schemas/ directory for example schema files for other supported benchmarks.
If running on quantum cloud hardware, the job will be added to a polling queue. The status of the queue can be checked with
mgym poll --job_id <METRIQ_GYM_JOB_ID>where <METRIQ_GYM_JOB_ID> is the assigned job ID of the job that was dispatched as provided by metriq-gym.
Alternatively, the poll action can be used without the --job_id flag to view all dispatched jobs,
and select the one that is of interest.
mgym pollIn order to export results to a JSON file, you can use the --json flag with the poll action.
mgym poll --job_id <METRIQ_GYM_JOB_ID> --jsonThis will create a JSON file with the results and the metadata of the job identified by <METRIQ_GYM_JOB_ID>.
By default, the JSON file will be saved in the current working directory with the name <METRIQ_GYM_JOB_ID>.json.
You can view all the jobs that have been dispatched by using the view action.
This will display basic information about each job, including its ID, backend, job type, provider, and device.
mgym viewIn order to view the details of a specific job (e.g., the parameters the job was launched with),
you can use the view action with the --job_id flag or select the job by index from the list of all dispatched jobs.
mgym --job_id <METRIQ_GYM_JOB_ID>The following example is for IBM, but the general workflow is applicable to any of the supported providers and benchmarks.
To run on IBM hardware, you will also require an IBM token. To obtain this, please visit the IBM Quantum
Platform and include the API token in the local .env file as previously described.
The schemas/examples/ directory houses example JSON configuration files that define the benchmark to run. In this
case, we use the bseq_example.json file as we want to run a BSEQ job. The following dispatches a
job on the ibm-sherbrooke device for BSEQ.
mgym dispatch metriq_gym/schemas/examples/bseq.example.json --provider ibm --device ibm_sherbrookeWe should see logging information in our terminal to indicate that the dispatch action is taking place:
INFO - Starting job dispatch...
INFO - Dispatching BSEQ benchmark job on ibm_sherbrooke device...
...
INFO - Job dispatched with ID: 93a06a18-41d8-475a-a030-339fbf3accb9We can confirm that the job has indeed been dispatched and retrieve the associated metriq-gym job ID (along with other pieces of metadata).
+--------------------------------------+------------+------------------------------------------------------+----------------+----------------------------+
| Metriq-gym Job Id | Provider | Device | Type | Dispatch time (UTC) |
+======================================+============+======================================================+================+============================+
| 93a06a18-41d8-475a-a030-339fbf3accb9 | ibm | ibm_sherbrooke | BSEQ | 2025-03-05T10:21:18.333703 |
+--------------------------------------+------------+------------------------------------------------------+----------------+----------------------------+We can use the "poll" action to check the status of our job:
mgym poll --job_id 93a06a18-41d8-475a-a030-339fbf3accb9Doing so gives us the results of our job (if it has completed):
INFO - Polling job...
BSEQResult(largest_connected_size=100, fraction_connected=0.7874015748031497)In the event where the job has not completed, we would receive the following message instead
INFO - Polling job...
INFO - Job is not yet completed. Current status:
INFO - - d0wtyfhvx7bg008203b0: QUEUED (position 3)
INFO - Please try again later.As a convenience, while we could supply the metriq-gym job ID, we can also poll the job by running mgym poll and then selecting the job to poll by index from our local metriq-gym jobs database.
Available jobs:
+----+--------------------------------------+------------+------------------------------------------------------+----------------+-----------------------------+
| | Metriq-gym Job Id | Provider | Device | Type | Dispatch time (UTC) |
+====+======================================+============+======================================================+================+=============================+
| 0 | 93a06a18-41d8-475a-a030-339fbf3accb9 | ibm | ibm_sherbrooke | BSEQ | 2025-03-05T10:21:18.333703 |
+----+--------------------------------------+------------+------------------------------------------------------+----------------+-----------------------------+
Select a job index: Entering the index (in this case, 0), polls the same job.
Select a job index: 0
INFO - Polling job...First, follow the Setup instructions above.
To pull the latest changes from the submodule’s repository:
cd submodules/qiskit-device-benchmarking
git pull origin mainThen, commit the updated submodule reference in your main repository.
We don't have a style guide per se, but we recommend that both linter and formatter are run before each commit. In order to guarantee that, please install the pre-commit hook with
poetry run pre-commit installimmediately upon cloning the repository.
The suite of unit tests can be run with
poetry run pytestThe project uses mypy for static type checking. To run mypy, use the following command:
poetry run mypyThe project uses Sphinx to generate documentation. To build the HTML documentation:
1.Navigate to the docs/ directory:
cd docs/Run the following command to build the HTML files:
make htmlOpen the generated index.html file located in the _build/html/ directory to view the documentation.