fix(run-stats): count blocks server-side so blocks_processed is correct in every worker mode - #74
Closed
rhoadesScholar wants to merge 1 commit into
Closed
fix(run-stats): count blocks server-side so blocks_processed is correct in every worker mode#74rhoadesScholar wants to merge 1 commit into
rhoadesScholar wants to merge 1 commit into
Conversation
…ct in every worker mode
The Resource Utilization report showed "blocks 0" per task for subprocess-mode runs
-- the default -- and empty per-worker counts, so tests/test_run_stats.py had to pin
itself to thread mode. This was the documented gap in RUN_STATS.md.
Cause: WorkerStats.blocks_processed was incremented only inside the in-process worker
thread's block loop. Subprocess shim workers and external cluster workers process
blocks where the stats layer never looked.
Fix: move counting to the server, where every worker's block return already arrives
over TCP.
- New protocol message Register { task_id, worker_id }, sent by Client::connect right
after the connection opens. Appended as the LAST enum variant, so the wire
discriminants of the existing messages are unchanged.
- The bookkeeper records each connection's registered identity; every VALID block
return (ReleaseBlock or BlockFailed) increments a RunTally counter per task and per
worker.
- build_run_stats merges the per-worker counts into the exit-channel WorkerStats by
(task_id, worker_id); registered workers with no thread in the server process
(fully external workers) get synthetic entries carrying their block counts.
Semantics: counts are now identical across thread, subprocess-shim and external
workers. Reclaimed attempts (block timeout, dead client) were never valid returns and
are no longer counted -- slightly more honest than the old thread-loop counter.
Failed-but-returned blocks still count, matching the old accounting. A client running
an older daisy that never registers still yields correct per-task counts; only its
per-worker attribution is absent.
Registration doubles as a liveness signal: registering clears any stale
closed-connection marker left on an OS-recycled ephemeral port, and disconnecting
drops the registration, so recycled ports cannot inherit stale identities or get their
in-flight blocks falsely reclaimed.
Internal API change: Client::connect and _rs.SyncClient now take a worker_id (both
internal; daisy.Client reads it from DAISY_CONTEXT as before).
Also fixes tests/test_run_stats.py::test_slowing_workload_reports_positive_slope,
which fails reproducibly on v2.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rhoadesScholar
pushed a commit
that referenced
this pull request
Aug 3, 2026
Pure `ruff --fix` pass over the Python sources with --select I001,F401,F811,RUF100,UP035,PYI029,PYI041. No behaviour changes: import sorting, unused-import removals, duplicate-import dedups, redundant noqa removals, typing.Callable -> collections.abc.Callable, and two .pyi stub cleanups (redundant __repr__ declarations; float|int -> float). Rebased from v2.0_patch onto v2.0 now that #70 is closed in favour of #72/#73/#74. Regenerated rather than cherry-picked, so the two hunks that only existed via #70 (build_wrapper.py, tests/test_worker_serialization.py) are simply absent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 3, 2026
Collaborator
|
Thanks for the report. This was a problem and was due to the same values being counted in multiple places (blocks processed). There should really only be one source of ground truth so that this sort of thing doesn't happen. resource tracking was in a fairly proof-of-concept state anyway so I fleshed it out. I removed the duplicated block done tracking, and made the resource tracking optional and more generally applicable. It now fires in the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of three, replacing #70. This is the substantial one — it changes the wire protocol and an
internal Rust API — and it was the piece hardest to see under the other two in #70.
Targets
v2.0directly. It shares no files with #73, so the two can be reviewed in either order.See the note at the bottom about running the reproduction, though.
Symptom
The Resource Utilization report showed
blocks 0per task for subprocess-mode runs — the default —and empty per-worker counts.
tests/test_run_stats.pyhad to pin itself to thread mode to assertanything. This was the "known gap" section in
docs/source/design/RUN_STATS.md.Both runs complete 6/6 blocks — the scheduler was always fine, only the accounting was wrong.
On
v2.0:On this branch:
Root cause
WorkerStats.blocks_processedwas incremented only inside the in-process worker thread's block loop.Subprocess shim workers and external cluster workers process blocks where the stats layer never looked.
Fix — count where every worker already reports
Register { task_id, worker_id }, sent byClient::connectright after theconnection opens. Appended as the last enum variant, so the wire discriminants of the six existing
messages are unchanged.
(
ReleaseBlockorBlockFailed) increments aRunTallycounter per task and per worker.build_run_statsmerges the per-worker counts into the exit-channelWorkerStatsby(task_id, worker_id). Registered workers with no thread in the server process — fully externalworkers — get synthetic entries carrying their block counts.
Semantics
with the same worker id as its babysitter thread).
— slightly more honest than the old thread-loop counter.
its per-worker attribution is absent.
explicitly rather than leaving it implicit: that assumption does not hold for every consumer — volara,
for instance, spawns
volara-cliworkers that resolve daisy from their own environment, so a mixedinstall is possible there.
left on an OS-recycled ephemeral port, and disconnecting drops the registration, so recycled ports
cannot inherit stale identities or get their in-flight blocks falsely reclaimed.
Internal API change:
Client::connectand_rs.SyncClientnow take aworker_id. Both areinternal;
daisy.Clientreads it fromDAISY_CONTEXTas before.docs/source/design/RUN_STATS.mdis updated — the "known gap" section is replaced by the Registerdesign, with
daisy.profile_blockkept as the future path for per-block CPU/RSS.Tests
Rust: unit tests for
RunTallycounting and thebuild_run_statsmerge (including syntheticexternal-worker entries and the unregistered-client fallback), bookkeeper registration lifecycle and
recycled-port hygiene;
integration_tcp.rsnow asserts externally-connected registered workers appear inrun stats. Python: the previously thread-pinned tests now run the default subprocess mode and their
blocks_processedassertions pass; a new pinned test keeps thread-mode counting covered.This also fixes
tests/test_run_stats.py::test_slowing_workload_reports_positive_slope, which failsreproducibly on
v2.0(3/3 runs on a tree that touches nothing inrun_stats). I flagged it aspre-existing on #73; it is fixed here.
One note on reproducing the MWE
The block function above deliberately does not reference the
daisymodule. If you write the morenatural
block.status = daisy.BlockStatus.SUCCESSunder an editable install, you will hitTypeError: cannot pickle '_thread._local' objectbefore reaching the stats — that is the separate bugfixed in #73. Either apply #73 first, or keep the block function free of module globals as above.