Comparative benchmarks

[penelopeysm]

This is the PR you all probably wanted but never got round to doing. It changes the benchmark workflow so that it runs benchmarks on the current PR, on the base of the PR, and prints a table side by side.

No doubt I'll have messed the CI part up, so we'll see.

[github-actions]

Benchmark Report

• this PR's head: 12f6cc5b4e01ef0bfb11671995ec196c855bb123
• base branch: 8e359cec781d023bf9fea8639624c0c29a88d4c3

Computer Information

Julia Version 1.11.7
Commit f2b3dbda30a (2025-09-08 12:10 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × AMD EPYC 7763 64-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

Benchmark Results

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────────────────┬──────────────────────────┐
│                       │       │             │                   │        │      t(eval) / t(ref)      │    t(grad) / t(eval)     │
│                       │       │             │                   │        │ ─────┬───────────┬──────── │ ─────┬─────────┬──────── │
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ base │   this PR │ speedup │ base │ this PR │ speedup │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼──────┼───────────┼─────────┼──────┼─────────┼─────────┤
│               Dynamic │    10 │    mooncake │             typed │   true │  err │    430.18 │     err │  err │   11.01 │     err │
│                   LDA │    12 │ reversediff │             typed │   true │  err │   2906.19 │     err │  err │    2.12 │     err │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │  err │ 156925.99 │     err │  err │    5.41 │     err │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │  err │  15889.80 │     err │  err │    5.14 │     err │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │  err │  31578.66 │     err │  err │    9.91 │     err │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │  err │   4195.83 │     err │  err │    7.72 │     err │
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │  err │     19.76 │     err │  err │    1.72 │     err │
│           Smorgasbord │   201 │      enzyme │             typed │   true │  err │   2645.29 │     err │  err │    4.62 │     err │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │  err │  23917.21 │     err │  err │   29.83 │     err │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │  err │   1073.73 │     err │  err │   81.99 │     err │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │  err │   2667.90 │     err │  err │   46.12 │     err │
│           Smorgasbord │   201 │ forwarddiff │      typed_vector │   true │  err │   2685.71 │     err │  err │   45.56 │     err │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │  err │   2357.46 │     err │  err │   47.07 │     err │
│           Smorgasbord │   201 │ forwarddiff │    untyped_vector │   true │  err │   2388.74 │     err │  err │   45.60 │     err │
│           Smorgasbord │   201 │    mooncake │             typed │   true │  err │   2643.80 │     err │  err │    5.65 │     err │
│           Smorgasbord │   201 │ reversediff │             typed │   true │  err │   2727.77 │     err │  err │   54.74 │     err │
│              Submodel │     1 │    mooncake │             typed │   true │  err │     26.23 │     err │  err │    5.37 │     err │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴──────┴───────────┴─────────┴──────┴─────────┴─────────┘

[penelopeysm]

OK, I literally can't prove that this works on this PR, because it relies on the benchmarks outputting a JSON file and that doesn't happen on the main branch. Going to have to merge before I can actually test it.

That's not true, I am a grown up programmer and I can gracefully handle files that don't exist.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.58%. Comparing base (8e359ce) to head (12f6cc5).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1143   +/-   ##
=======================================
  Coverage   81.58%   81.58%           
=======================================
  Files          40       40           
  Lines        3845     3845           
=======================================
  Hits         3137     3137           
  Misses        708      708           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yebai]

AirspeedVelocity.jl from @MilesCranmer looks great for benchmarking purposes.

[penelopeysm]

Ah, yes, that looks good. I'm sure somebody can investigate switching over to it when they find the time.

[github-actions]

DynamicPPL.jl documentation for PR #1143 is available at:
https://TuringLang.github.io/DynamicPPL.jl/previews/PR1143/

[penelopeysm]

I'm going to merge this to see if it actually works correctly with another PR

[mhauru]

Yesssss