Skip to content

Commit

Permalink
Slight speed up to notebook tests (#4147)
Browse files Browse the repository at this point in the history
Shaves a bit of time off notebook tests by

1. Fixing typos in two replacement filenames (so the replacements get applied)
2. Adding two flags to papermill: `--no-request-save-on-cell-execute` and `--autosave-cell-every 0`.

From the [papermill docs](https://papermill.readthedocs.io/en/latest/usage-cli.html):

```
  --request-save-on-cell-execute / --no-request-save-on-cell-execute
                                  Request save notebook after each cell
                                  execution

  --autosave-cell-every INTEGER   How often in seconds to autosave the
                                  notebook during long cell executions (0 to
                                  disable)
```

Sample times and notebooks over 10s from `check/pytest -n auto -m slow dev_tools/notebooks/notebook_test.py --durations=0` on my computer are below.

<details>
<summary> On master: 144.78s </summary>

```
89.85s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/cirq-core/cirq/contrib/quimb/Cirq-to-Tensor-Networks.ipynb]
56.99s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/educators/intro.ipynb]
37.92s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/quantum_walks.ipynb]
30.20s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/qcvv/isolated_xeb.ipynb]
23.58s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/qcvv/xeb_coherent_noise.ipynb]
22.69s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/qcvv/parallel_xeb.ipynb]
20.16s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/educators/textbook_algorithms.ipynb]
17.15s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/operators_and_observables.ipynb]
16.72s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/cirq-core/cirq/contrib/quimb/Contract-a-Grid-Circuit.ipynb]
14.05s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/educators/qaoa_ising.ipynb]
13.93s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/qaoa.ipynb]
13.91s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/hidden_linear_function.ipynb]
12.91s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/rabi_oscillations.ipynb]
12.10s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/noise.ipynb]
12.02s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/qcvv/xeb_theory.ipynb]
11.75s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/interop.ipynb]
10.44s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/basics.ipynb]
10.42s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/examples/advanced/quantum_volume_errors.ipynb]
10.03s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/fourier_checking.ipynb]
```
</details>

<details>
<summary> With replacement file typo fixes: 127.23s </summary>

```
57.27s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/educators/intro.ipynb]
37.01s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/quantum_walks.ipynb]
30.10s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/qcvv/isolated_xeb.ipynb]
22.76s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/qcvv/xeb_coherent_noise.ipynb]
22.62s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/qcvv/parallel_xeb.ipynb]
16.75s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/cirq-core/cirq/contrib/quimb/Contract-a-Grid-Circuit.ipynb]
15.04s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/educators/textbook_algorithms.ipynb]
14.99s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/operators_and_observables.ipynb]
14.92s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/cirq-core/cirq/contrib/quimb/Cirq-to-Tensor-Networks.ipynb]
13.98s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/educators/qaoa_ising.ipynb]
13.87s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/qaoa.ipynb]
12.60s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/examples/advanced/quantum_volume_errors.ipynb]
12.44s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/rabi_oscillations.ipynb]
11.88s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/hidden_linear_function.ipynb]
11.79s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/interop.ipynb]
11.69s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/noise.ipynb]
10.95s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/qcvv/xeb_theory.ipynb]
10.47s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/shor.ipynb]
```
</details>

<details>
<summary> With replacement file typo fixes & papermill options: 98.97s </summary>

```
34.08s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/quantum_walks.ipynb]
29.59s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/qcvv/isolated_xeb.ipynb]
22.62s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/qcvv/xeb_coherent_noise.ipynb]
21.39s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/qcvv/parallel_xeb.ipynb]
13.76s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/cirq-core/cirq/contrib/quimb/Cirq-to-Tensor-Networks.ipynb]
13.49s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/cirq-core/cirq/contrib/quimb/Contract-a-Grid-Circuit.ipynb]
12.59s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/examples/advanced/quantum_volume_errors.ipynb]
12.29s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/qaoa.ipynb]
10.46s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/tutorials/rabi_oscillations.ipynb]
10.05s call     dev_tools/notebooks/notebook_test.py::test_notebooks_against_released_cirq[Cirq/docs/interop.ipynb]
```
</details>

I think I see a little speedup in CI...
  • Loading branch information
rmlarose committed May 28, 2021
1 parent fec7710 commit 705f37b
Show file tree
Hide file tree
Showing 3 changed files with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion dev_tools/notebooks/notebook_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,9 @@ def test_notebooks_against_released_cirq(notebook_path):
notebook_rel_dir = os.path.dirname(os.path.relpath(notebook_path, "."))
out_path = f"out/{notebook_rel_dir}/{notebook_file[:-6]}.out.ipynb"
rewritten_notebook_descriptor, rewritten_notebook_path = rewrite_notebook(notebook_path)
papermill_flags = "--no-request-save-on-cell-execute --autosave-cell-every 0"
cmd = f"""mkdir -p out/{notebook_rel_dir}
papermill {rewritten_notebook_path} {out_path}"""
papermill {rewritten_notebook_path} {out_path} {papermill_flags}"""

_, stderr, status = shell_tools.run_shell(
cmd=cmd,
Expand Down

0 comments on commit 705f37b

Please sign in to comment.