Implement continuous profiling with Codspeed

[sydney-runkle]

We should use Codspeed for continuous profiling of things we care about like import time, as we do on pydantic-core.

See #7947 (comment) for some more context.

We'd love some help on this 🙏!

[lambertsbennett]

I would be into working on this!

[lambertsbennett]

OK so I think I have the basic setup as follows:

name: codspeed

on:
  push:
    branches:
      - main
  pull_request:
  # `workflow_dispatch` allows CodSpeed to trigger backtest
  # performance analysis in order to generate initial data.
  workflow_dispatch:

jobs:
  codspeed-profiling:
    name: codspeed profiling
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: pdm-project/setup-pdm@v3
        with:
          python-version: '3.10'
          cache: true

      - name: install deps
        run: |
          pdm venv create --with-pip --force $PYTHON
          pdm install -G testing -G testing-extra -G email

      - name: Run CodSpeed benchmarks
        uses: CodSpeedHQ/action@v1
        with:
          run: pdm run pytest ./tests/benchmarks --codspeed

This runs, but then I get an error that it does not detect a repository or the token is invalid. I see on the codspeed website that normally in the with block you include a token like

token: ${{ secrets.CODSPEED_TOKEN }}

I don't have access to this token to test it fully, but otherwise the benchmarks run fine.

[samuelcolvin]

it's shouldn't be just on main, we want codspeed to run on all PRs to notify us of performance regressions, as it does on pydantic-core or jiter.

[lambertsbennett]

@samuelcolvin ok no problem, I'll make that change, but there still seems to be an issue with a token for codspeed?