Towards Public Benchmarks

[cburroughs]

Fast is good! People are endeared to fast tools. Faster is a feature and makes new things possible. Fast makes it feasible to compose commands into more useful workflows.

Historically most reasoning about performance for the project has centered around either the Pants repository itself, or private ones. These aren't bad approaches, but each of these has downsides.

Pants itself has meaningful differences from Pants users such as:

• Is maximal in backend use. Most users don't use every language or backend.
• Has few "deployable" targets.
• Is narrowly constrained in 3rdparty dependencies (no giant Python ML libraries)
• Is as large as it needs to be; not among the biggest or smallest repositories.
• Always bootstraps at HEAD. Can't (easily) use Pants with older Pants versions to compare performance.

Relying on internal repositories has the great virtue of measuring real world use, but the obvious downside that they can not be shared. There is also always uncertainty in judging how representative an internal process is. Is Pants slow at X? Or is X something absurd where the answer for any tool would be "don't do that"?

In connection with #18911 @jriddy did some great work to try to create a synthetic repository that preserved the structure of an internal repository, but with all of the symbols renamed and bodies of functions removed. This wasn't perfect -- the original pathological behavior wasn't replicated -- but it was still super useful and I think this is a promising model.

So: a benchmark derived from real use that is not totally synthetic or artificial. Ultra processed benchmarks perhaps?

As an aspiration, I'd imagine something like The Python Performance Benchmark Suite where there are a set of benchmarks and a tool to help consistently run them and aggregate results.

Like all parts of a community project, which benchmarks exist is partly a function of who is motivated to make one and what they are interested in. I think https://github.com/cburroughs/clam-diggers/ is great for Python dependency inference. Some other areas I might be interested in are: visibility, check hot cache path, changed-since.

Some Open & Discussion Questions

• How to structure the relationship between the harness and the benchmarks? Inline? Sub-modules? Have a checkout script? A lot of tradeoffs here.

• I'm assuming on a scale between "macro" and "micro" benchmarks, that everyone is going to say "macro" is better. But pyperformance is a somewhat idiosyncratic mix. I can think of a few cases where something bugs me for how slow it is, but to get it to clearly stand out I'd have to make a pretty odd repository.

• Does anyone care about absolute/consistent numbers, or is it always about change by change relative performance? "How was your Python compiled?" makes a huge difference for example, and one could sink a lot of time in environment consistency.

• I suspect it is going to be easier to get consistent results for CPU/memory heavy problems vs IO. But "doing anything with this giant pile of dependencies is slow" is a common -- and very reasonable -- compliant.

• I'm super excited about the embarrassment of riches for improvements from the perf support. I don't have a good sense of how valuable "Run the benchmark in docker with everything properly compiled" is. Or is there some equivalent mac thing people have been using?

• How could open source Pants use like https://github.com/StackStorm/st2/ fit into this?

[cburroughs]

@cognifloyd Curios if you have any thoughts re st2. Should we spend more time making your workflows faster? ;-)

and @tobni I know you have been doing perf work recently. Would any of this be useful, or do you have everything you need by going down the list of slowness in your internal repo?

[cognifloyd]

I would love more help with the pants integration for st2. Sadly, a large work project has sucked up all my time for several months--far exceeding the time I expected to spend on it. So, I haven't had the mental bandwidth to work on finishing the st2 transition to pants. I really hope to get back to that soon.

As far as benchmarks go, faster is awesome, but I'm not sure what goes into building good benchmarks. Using st2 to create a benchmark would be great, I'm just not sure what using st2 to benchmark pants looks like.

[Affanmir]

I recently put together benchmarks for lockfile generation (pex resolver vs uv resolver) in #23212 and ran into some of the challenges you're describing

I used hyperfine with 87 top-level requirements (Trio test deps + common ecosystem packages same baseline uv uses) resolving to 169 packages.
A few observations that might be useful for this effort:

1. Engine overhead dominates some stats. The raw resolver comparison showed major speedup in my PR about 57x (uv vs pex/pip), but end-to-end through Pants it was only 1.48x because ~20s of engine init, interpreter discovery, and PEX CLI bootstrap is constant. Any benchmark suite should probably measure both the isolated operation and the full pants invocation to separate tool speed from engine overhead.

2. Cache state is the hardest variable to control. I ended up defining three tiers: "cold" (all caches cleared), "semi-warm" (wheel caches retained, LMDB cleared), and "warm" (everything cached). The results were quite different across tiers. Having the benchmark suite define standard cache states would help

Regarding benchmark categories: Lockfile generation (generate-lockfiles) would be a great addition to the suite. The Trio-based requirements set works well as a public baseline since it's what uv itself benchmarks against.

[cburroughs]

Engine overhead dominates some stats

Interesting. Did you compare to the Pex command itself? That is with Pants the lockfile took X to generate, but with the same Pex command it was Y?

[Affanmir]

Yup, I ran pex3 lock create and uv pip compile directly against the same 87-requirement file, then the same workload through pants generate-lockfiles. Both resolvers scoped to my current platform / Python 3.11 + Hardware listed below (pex3 --style=strict matches uv pip compile's default).

Resolver only

Cache	pex3 lock create	uv pip compile	Ratio
Cold	119.8 s ± 15.3 s (n=3)	3.63 s ± 0.47 s (n=5)	33×
Warm	34.7 s ± 14.1 s (n=5)	0.40 s ± 0.08 s (n=5)	88×

Pants e2e, semi-warm (LMDB + lockfile cleared each run, wheel caches kept)

Resolver	Mean	Range	n
--python-lockfile-resolver=pex	31.2 s ± 11.9 s	21.7 – 49.2 s	5
--python-lockfile-resolver=uv	24.9 s ± 0.4 s	24.4 – 25.3 s	5

So: ~33×/88× on the resolver collapses to 1.25× e2e the remaining time is a ~20 s floor of interpreter discovery, PEX CLI bootstrap, and rule graph setup that sits under both paths. Pex's ±12 s e2e stddev vs uv's ±0.4 s on identical inputs is the other thing worth noting; a mean-only benchmark would hide it.

Also we might benefit from having benchmarking scope definitions e.g resolution time with warm cache vs cold cache

Env: Apple M4, macOS 15, Python 3.11.13, pex 2.92.0, uv 0.11.4, hyperfine 1.20.0, pants feature/uv-lockfile-resolver. Full hyperfine invocations in #23212.

[cburroughs]

changed-since.

https://github.com/altana-ai/pants-changed-dependents-perf-repro

[cburroughs]

Mostly a self breadcrumb for an extremely early prototype https://github.com/cburroughs/pants-benchmark-ii-hyper-fighting