Stop the bundled benchmarks comparing daisy against itself

[rhoadesScholar]

Why

The gerbera -> daisy rename (2c1a382) collapsed both arms of every comparison in benchmarks/ onto the same package. The damage is mechanical and easy to miss:

• each script defines bench_daisy twice (bench_dependency_graph.py L19/L47, bench_worker_scaling.py L23/L53). The second shadows the first, so one arm of every comparison is silently discarded and both reported columns run the same code;
• bench_worker_scaling.py builds its result dicts with the key "daisy_s" twice, and bench_dependency_graph.py with "daisy" twice. The later value wins, so one arm never even reaches the JSON.

Running the scripts unchanged on current v2.0 shows what that produces — a "speedup" of 1.0x, i.e. the shape of one implementation measured against itself:

$ python benchmarks/bench_dependency_graph.py     # on origin/v2.0, unmodified
  daisy:    1000000 blocks,   1 levels, build=0.0000s  iter=0.7433s  total=0.7434s
  daisy:  1000000 blocks,   1 levels, build=0.0000s  iter=0.7541s  total=0.7541s
  speedup: 1.0x
...
  daisy:     970299 blocks,   8 levels, build=0.0000s  iter=2.0755s  total=2.0755s
  daisy:   970299 blocks,   8 levels, build=0.0000s  iter=1.9518s  total=1.9518s
  speedup: 1.1x

$ python benchmarks/bench_worker_scaling.py       # on origin/v2.0, unmodified
 workers |      daisy |      daisy |    daisy-wf |  speedup
-----------------------------------------------------------
       1 |     0.075s |     0.052s |      0.873s |     1.4x
       2 |     1.209s |     1.231s |      0.823s |     1.0x
       4 |     0.634s |     0.662s |      0.331s |     1.0x
       8 |     0.388s |     0.383s |      0.232s |     1.0x
      16 |     0.278s |     0.282s |      0.219s |     1.0x
      32 |     0.333s |     0.259s |      0.226s |     1.3x

The five committed artifacts encode the same thing, and two of them prove they were text-substituted by the rename rather than regenerated — a Python dict literal cannot emit a duplicate key, but the committed JSON has one:

$ python -c "
import json, collections
def hook(pairs):
    dup=[k for k,c in collections.Counter(k for k,_ in pairs).items() if c>1]
    if dup: print('DUPLICATE KEYS:', dup, '->  kept value wins silently')
    return dict(pairs)
json.loads(open('benchmarks/dep_graph_results.json').read(), object_pairs_hook=hook)
json.loads(open('benchmarks/worker_scaling_results.json').read(), object_pairs_hook=hook)"
DUPLICATE KEYS: ['daisy'] ->  kept value wins silently
DUPLICATE KEYS: ['daisy'] ->  kept value wins silently
DUPLICATE KEYS: ['daisy'] ->  kept value wins silently
DUPLICATE KEYS: ['daisy'] ->  kept value wins silently
DUPLICATE KEYS: ['daisy_s'] ->  kept value wins silently
DUPLICATE KEYS: ['daisy_s'] ->  kept value wins silently
DUPLICATE KEYS: ['daisy_s'] ->  kept value wins silently
DUPLICATE KEYS: ['daisy_s'] ->  kept value wins silently
DUPLICATE KEYS: ['daisy_s'] ->  kept value wins silently
DUPLICATE KEYS: ['daisy_s'] ->  kept value wins silently

And block_scaling_results.json advertised a 28.44x speedup on the 100-block row, which is inter-run noise on a ~0.2s measurement of the same code path.

What this does — drop the vestigial arm, don't restore it

git show 2c1a382^:benchmarks/bench_dependency_graph.py settles the question: the first bench_* in each file was the old pure-Python daisy 1.x package, the second was gerbera, the Rust rewrite that became this repository. There is no second implementation here to compare against and no way to obtain one from this tree, so option (a):

• bench_dependency_graph.py is now a single-implementation scaling benchmark — how enumeration cost grows with block count, conflict levels and context. One arm, no ratio, flat result records (blocks, levels, build_s, iter_s, total_s, blocks_per_s).
• bench_worker_scaling.py keeps the two process_function modalities that genuinely still exist and drops the phantom third series from the plot: 1-arg block-fn (the worker subprocess drives Client.acquire_block()) vs 0-arg worker-fn (the subprocess calls the function, which drives its own loop). Both run in worker subprocesses since aae3e6d, so this is one execution model, two callback shapes.

Three further corrections were required for the regenerated numbers to mean anything. Each is called out because each moves a number:

1. The worker-scaling curve now uses the distributed Server at every worker count. It previously switched to the in-process SerialServer at workers=1 for the block-fn arm only — so that point measured a different execution model than the rest of its own line, and than the worker-fn arm, which always used Server. This is why block-fn at workers=1 moves from 0.075s to ~2.2s: the old value was in-process serial execution with no TCP and no subprocess, mislabelled as "1 worker".

2. progress=False. tqdm rendering was inside a timed region whose docstring claims to isolate coordination overhead.

3. Success is completed_count == total_block_count, not is_done(). TaskState.is_done() means "the counters balance", which is also true when every block failed. A run that completed nothing passed the old assertion and reported a fast time for it — observed while setting this up, before funlib.geometry was present in the venv:

   block-fn 1000 blocks / 4 workers: (0.3281273050233722, True, 0)
   

   0.33s, done=True, 0 of 1000 blocks completed, and assert g_done was satisfied.

Also updated because they describe exactly what changed: ARCHITECTURE.md's one-line description of benchmarks/ ("Throughput comparisons vs daisy 1.x" — no longer true), a CHANGELOG entry, and the [tool.ruff] extend-exclude = ["benchmarks"] escape hatch, whose comment ("they need a rewrite, not a lint pass") asked for precisely this.

Ruff, before and after

Before, on origin/v2.0 (annotation bodies elided for length; --select F811,F601 was passed explicitly because base excludes benchmarks/ from lint entirely):

$ ruff check --select F811,F601 benchmarks
F811 [*] Redefinition of unused `DaisyGraph` from line 11
  --> benchmarks/bench_dependency_graph.py:15:47
F811 [*] Redefinition of unused `DaisyRoi` from line 12
  --> benchmarks/bench_dependency_graph.py:12:26
F811 Redefinition of unused `bench_daisy` from line 19
  --> benchmarks/bench_dependency_graph.py:47:5
F601 Dictionary key literal `"daisy"` repeated
   --> benchmarks/bench_dependency_graph.py:109:13
F811 [*] Redefinition of unused `daisy` from line 18
  --> benchmarks/bench_worker_scaling.py:18:8
F811 Redefinition of unused `bench_daisy` from line 23
  --> benchmarks/bench_worker_scaling.py:53:5
F601 Dictionary key literal `"daisy_s"` repeated
   --> benchmarks/bench_worker_scaling.py:147:13
F601 Dictionary key literal `"daisy_s"` repeated
   --> benchmarks/bench_worker_scaling.py:183:13
Found 8 errors.
[*] 3 fixable with the `--fix` option.

After, with the exclusion removed so the project's own rule set applies:

$ ruff check --no-cache --select F811,F601 benchmarks
All checks passed!
$ ruff check --no-cache benchmarks
All checks passed!
$ ruff format --check --no-cache benchmarks
2 files already formatted

ruff check . does not pass on this branch — and does not pass on origin/v2.0 either. A pristine origin/v2.0 worktree with the pinned ruff==0.16.1 reports 28 I001 errors, all in tests/, none in benchmarks/, and this branch reports the identical 28. This PR does not touch tests/ and neither introduces nor fixes them; removing the benchmarks exclusion contributes none of them. Worth a separate PR.

Regenerated artifacts

All five were produced by running the fixed scripts — nothing was hand-edited. Verbatim output of the run that produced the committed files:

$ python benchmarks/bench_dependency_graph.py

============================================================
  1M blocks, no conflict
  total=(1000, 1000, 1000) block=(10, 10, 10) context=0 conflict=False
============================================================
   1000000 blocks,   1 levels, build=0.0000s  iter=0.8423s  total=0.8423s  (1,187,170 blocks/s)

============================================================
  1M blocks, with conflict
  total=(1000, 1000, 1000) block=(10, 10, 10) context=2 conflict=True
============================================================
    970299 blocks,   8 levels, build=0.0000s  iter=2.3310s  total=2.3310s  (416,262 blocks/s)

============================================================
  125K blocks, small chunks
  total=(200, 200, 200) block=(4, 4, 4) context=0 conflict=False
============================================================
    125000 blocks,   1 levels, build=0.0000s  iter=0.0677s  total=0.0678s  (1,844,424 blocks/s)

============================================================
  1M blocks, small context
  total=(500, 500, 500) block=(5, 5, 5) context=1 conflict=True
============================================================
    970299 blocks,   8 levels, build=0.0000s  iter=2.1713s  total=2.1714s  (446,862 blocks/s)

Saved benchmarks/dep_graph_benchmark.png

$ python benchmarks/bench_worker_scaling.py
Blocks: 10000
Worker counts: [1, 2, 4, 8, 16, 32]
 workers |   block-fn |   worker-fn |    ratio
----------------------------------------------
       1 |     2.206s |      0.959s |    2.30x
       2 |     1.153s |      0.554s |    2.08x
       4 |     0.656s |      0.312s |    2.10x
       8 |     0.428s |      0.237s |    1.81x
      16 |     0.308s |      0.174s |    1.76x
      32 |     0.271s |      0.218s |    1.24x

Block scaling (workers=4)
  blocks |   block-fn |   worker-fn |    ratio
----------------------------------------------
     100 |     0.085s |      0.096s |    0.88x
    1000 |     0.147s |      0.119s |    1.24x
   10000 |     0.664s |      0.289s |    2.30x
  100000 |     5.792s |      2.303s |    2.52x

Saved benchmarks/worker_scaling_benchmark.png

Both scripts are cheap: 15s and 17s wall respectively. The JSON now round-trips with no duplicate keys:

$ python -c "... same object_pairs_hook duplicate check as above ..."
benchmarks/dep_graph_results.json no duplicate keys
benchmarks/worker_scaling_results.json no duplicate keys
benchmarks/block_scaling_results.json no duplicate keys

The numbers are now monotone in worker count and consistent between the two block-scaling views, which the old self-comparing ones were not. Worker-fn beating block-fn by 1.2-2.3x is a real, reproducible difference between the two callback shapes (it held across three runs); I have deliberately not speculated in the docstrings about why, since I did not profile it. Note both arms go through the v1-compat Block boundary — v1_compat monkey-patches Client.acquire_block — so compat wrapping is not the explanation.

Environment: AMD EPYC 9454 (96 CPUs), Linux 6.8, CPython 3.12.3, maturin develop --release, daisy 2.0.0 at 542c3c2. Timings are machine-specific; the docstring now says so.

Running the benchmarks needs matplotlib, cloudpickle and — less obviously — funlib.geometry, which is not a declared dependency of daisy anywhere but is imported by v1_compat.Block.read_roi, so every distributed 1-arg block function fails without it. Reported separately; not fixed here.

Does this change any documented conclusion?

No. The report that prompted this suspected these scripts were the evidence base for the subprocess-vs-thread defaults in daisy-py/python/daisy/_worker_processes.py ("threads were 1.7x slower at 10% python glue, 28x at 100%", "~15%"). They are not. git log -S "python glue" traces that text to 7e0ebc8, which names its source explicitly:

Benchmark evidence (bench_workload_mix, 16 workers, 96 x ~100ms blocks, BLAS pinned to 1 thread) [...] It loses 1.7x with just 10% python glue, 8x at 30%, 28x at 100% python.

bench_workload_mix has never existed in this repository (git log --all --diff-filter=A -- '*workload_mix*' is empty), so those figures are unreproducible from the tree either way — a separate gap, not this PR's to close. The 28x / 28.44x near-collision with block_scaling_results.json is a coincidence: different workload, different worker count, different quantity.

Skeptical pass

• One change: make the bundled benchmarks measure daisy v2 honestly, and regenerate the artifacts from the fixed code. The ARCHITECTURE.md line, CHANGELOG entry and ruff-exclusion removal all describe or gate exactly that.
• Nothing riding along: no library code touched (daisy-py/, daisy-core/ untouched), no tests touched, no dependencies added.
• The artifacts are regenerated, never hand-edited. If you would rather they were untracked, that is a one-line .gitignore change on top — but they regenerate in ~30s total, so keeping them is cheap and they are the only committed record of the shapes.

[pattonw]

Oh good catch. I haven't revisited these benchmarks in a little while. Thanks for cleaning them up. I'm curious why the block function is slower than the worker function, they should be pretty identical at this point. But that can be something for future investigation.
I would also like to add something like a daisy version comparison so we can add the current daisy on pypi as a comparison target and see how the speeds change. I'd be surprised if the original daisy was anywhere near 400k blocks per second on any of those tasks.