Reorder Pull request CI steps

[frankharkins]

This PR re-orders the main PR CI steps by how likely they are to fail. This should reduce CI time and help contributors get feedback faster. It also pushes the Playwright and ImageMagick install steps to just before they're required, rather than at the beginning.

Here are the failure reasons for the last 500 failed runs:

Script used

import subprocess
from collections import defaultdict
from time import sleep

LIMIT = 500

run_ids = (
    subprocess.run(
        f"gh run list --workflow='Pull request' --status=failure --limit {LIMIT} --json databaseId | jq '.[] | .databaseId'",
        shell=True,
        check=True,
        capture_output=True,
    )
    .stdout.decode()
    .strip()
    .split("\n")
)


def get_failure_step(run_id: str) -> str:
    reason = (
        subprocess.run(
            f'gh run view {run_id} --json jobs | jq \'.jobs[] | select( .name == "Lint" ) | select( .conclusion == "failure" ) | .steps[] | select( .conclusion == "failure" ) | .name\'',
            shell=True,
            check=True,
            capture_output=True,
        )
        .stdout.decode()
        .strip()
        .strip('"')
    )
    return reason

sums = defaultdict(int)
for index, num in enumerate(run_ids):
    sleep(1)
    print(f"Getting run {index}/{len(run_ids)}")
    try:
        reason = get_failure_step(num)
        if reason == "":
            continue
        sums[reason] += 1
    except Exception as err:
        print(f"Error: {err}")
        pass

items = [(count, name) for (name, count) in sums.items()]
for count, name in sorted(items, key=lambda x: -x[0]):
    print(f"{str(count).rjust(5)} {name}")

107 Internal link checker
 97 Lint notebooks
 59 Check stale images
 56 Check markdown
 27 Spellcheck
 17 Check Qiskit bot config
 12 Formatting
 12 Install ImageMagick
 10 Check orphaned pages
  7 Check that pages render
  7 Check all notebooks are listed in the config file
  7 Infrastructure tests
  4 Pull preview image
  4 Install Node.js dependencies
  3 Check for orphan pages
  1 Get changed files artifact

While "Lint notebooks" comes second, I think it still makes sense to run it last because it feels neater to keep all the Python checks together, and it requires Python/ImageMagick setup.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[arnaucasau]

Great idea!! 💡