Bug: crash on performance collection: InvalidPerfEventSize

[woodruffw]

Hi there! I think this is the right repo to file this against, but apologies if not.

I'm using Codspeed (it's great, thank you!) to do some benchmarking of zizmor. I recently tried to enable some online benchmarks in CI, but was met with this uncaught error, causing a panic:

  ============================= test session starts ==============================
  platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0
  codspeed: 4.2.0 (enabled, mode: walltime, callgraph: enabled, timer_resolution: 1.0ns)
  rootdir: /home/runner/work/zizmor/zizmor
  configfile: pyproject.toml
  plugins: codspeed-4.2.0
  collected 7 items
  
  bench/test_bench_basic.py ..                                             [ 28%]
  bench/test_bench_offline.py ..                                           [ 57%]
  bench/test_bench_online.py ...                                           [100%]
  
                                 Benchmark Results                                
  ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━┓
  ┃                                                ┃  Time ┃ Rel. ┃   Run ┃      ┃
  ┃                                      Benchmark ┃ (bes… ┃ Std… ┃  time ┃ Ite… ┃
  ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━┩
  │                            test_zizmor_startup │ 2,42… │ 6.7% │ 3.02s │ 1,0… │
  │                               test_zizmor_help │ 2,69… │ 5.8% │ 3.02s │ 1,0… │
  │         test_zizmor_offline_grafana_9f212d11d0 │ 350,… │ 1.1% │ 2.87s │    8 │
  │       test_zizmor_offline_cpython_48f88310044c │ 122,… │ 1.3% │ 2.91s │   23 │
  │          test_zizmor_online_gha_hazmat_da3c3cd │ 176,… │ 0.0% │ 17.6… │   10 │
  │        test_zizmor_online_cpython_48f88310044c │ 275,… │ 0.0% │ 27.6… │   10 │
  │ test_zizmor_online_collect_workflows_fast_path │ 196,… │ 0.0% │ 19.6… │   10 │
  └────────────────────────────────────────────────┴───────┴──────┴───────┴──────┘
  
  ================================ 7 benchmarked =================================
  
  ======================== 7 passed in 229.89s (0:03:49) =========================
  
  thread 'main' panicked at src/executor/wall_time/perf/parse_perf_file.rs:68:70:
  called `Result::unwrap()` on an `Err` value: InvalidPerfEventSize
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  Error: Process completed with exit code 101.

I don't know the Linux perf subsystem super well, but this suggests a mismatch between that the perf event header actually contains and what Codspeed expected.

Full workflow run here: https://github.com/zizmorcore/zizmor/actions/runs/24426548664/job/71361672276?pr=1912

Some other potentially relevant bits:

• Running on ubuntu-latest on GitHub Actions, which is 24.04 at the moment
• According to GitHub's docs, ubuntu-latest is also kernel 6.17.0-1010-azure
• Actions runner version is 2.333.1

Suspiciously, it looks like GitHub also bumped the kernel version recently (~12 hours ago): actions/runner-images@d8fb498 -- the bump appears to be a build number however and not an actual kernel version change, so that may be a red herring.

[woodruffw]

This appears to be the crashing line as well:

codspeed/src/executor/wall_time/perf/parse_perf_file.rs

Line 68 in fe49a28

while let Some(record) = record_iter.next_record(&mut perf_file).unwrap() {

[GuillaumeLagrange]

Thanks for the report, indeed this is the proper repository, we've seen similar issues before and will have a look ad what's happening here.

[GuillaumeLagrange]

We're going to merge #298, which does not fully fix the issue, but at least we'll be able to retrieve corrupted perf.pipedata to understand what's up.

We have already observed some similar stuff on things that seemed to be an upstream perf bug when handling large payloads in perf samples with compression enabled, which ended up corrupted. We have a potential repro and fix but we have not yet taken the time to sanitize it for it to be upstreamed, and for now we're just making sure the version of perf on the macro runners is carefully chosen, but we do not have that luxury on github actions runner.

With the partial fix, the symptom will be some missing profiles that we'll then be able to investigate further. Thanks for the report!

[woodruffw]

Awesome, thank you @GuillaumeLagrange! I'm happy to help debug however I can/however would be helpful.