Metrics and Diagnosis

Metrics and Diagnosis

PoolWatch analyzes intervals, not terminal progress characters or textual logs. For each observed test attempt, the interval begins at the earliest pytest phase start and ends at the latest phase finish.

Timing source

Pytest report timestamps are authoritative when they exist. This matters for concurrent schedulers that emit pytest_runtest_logstart or pytest_runtest_logfinish after a coroutine has already completed. Lifecycle hooks remain a fallback for interrupted or incomplete attempts.

The phases are:

• setup
• call
• teardown

Aggregate phase totals may exceed wall-clock duration because concurrent tests overlap.

Timeline construction

Every test interval contributes two boundaries:

(started_at, +1)
(finished_at, -1)

PoolWatch groups equal timestamps and sweeps the sorted boundaries. Completions are applied before starts, then the state for the following segment is measured. This produces active-test and queued-test step functions.

Target capacity

The target is the expected maximum number of concurrently active tests. It can be explicit, detected from a supported scheduler, or unknown.

A known target is required for a definitive underfill diagnosis. When PoolWatch has only an observed peak, it reports a useful baseline but does not claim that the scheduler left known capacity unused.

Peak active tests

The maximum value of the active-test timeline.

Peak proves that a concurrency level was reached at least once; it does not show whether that level was sustained.

Average active tests

Let each timeline segment have active count aᵢ and duration Δtᵢ:

active area = Σ(aᵢ × Δtᵢ)
average active = active area / observed duration

The average is time-weighted. A brief high-concurrency spike cannot dominate a long underfilled period.

Concurrency utilization

For target T and observed duration D:

utilization = active area / (T × D)

Examples:

• 1.0 means all configured slots were occupied for the full observed interval.
• 0.75 means the suite used 75% of available slot-time.
• a value above 1.0 indicates that the supplied target was lower than observed concurrency and should be corrected.

Utilization includes natural suite drain. Use underfill windows to distinguish drain from a scheduler failing to refill slots.

Idle slot-seconds

For each segment:

idle slot-seconds = Σ(max(0, T - aᵢ) × Δtᵢ)

This is lost capacity area, not elapsed time. Four idle slots for ten seconds contribute forty idle slot-seconds.

Idle slot-seconds include normal drain after the queue empties.

Queue and peak queued tests

Queued tests are collected tests that have not started their first observed attempt. The queue decreases when a node ID starts for the first time.

Peak queued tests is the largest queue value represented after a timeline event.

Scheduler underfill

A timeline segment is scheduler underfill only when all conditions are true:

target is known
queued > 0
active < target
segment duration > 0

Adjacent underfilled segments are merged. Each window records:

• start and finish offsets;
• minimum active tests;
• maximum queued tests;
• idle slot-seconds inside the window.

Windows shorter than poolwatch_underfill_threshold are excluded from the underfill duration and report table.

Natural drain

When queued == 0, reduced concurrency is expected: no unstarted work remains to refill empty slots. PoolWatch still includes those empty slots in utilization and idle slot-seconds, but does not diagnose scheduler underfill.

What PoolWatch cannot infer

Version 0.1 cannot determine whether active async tests are making progress, waiting on external services, or blocking an event loop. An active test occupies one observed test slot in all three cases.

PoolWatch therefore avoids claiming event-loop responsiveness or root-cause attribution. See Roadmap for possible future measurements.