flake: TestProvisionerJobQueueWaitMetric

[flake-investigator]

CI Run: https://github.com/coder/coder/actions/runs/21974897464 (job: test-go-pg (macos-latest))
Commit: 2cc8cc59faf16c82d6abb58114023915c1d13483 (chore: tell claude to use react-query) by ケイラ
Failure time: 2026-02-13 04:52:59Z

Error:
TestProvisionerJobQueueWaitMetric failed at coderd/workspaces_test.go:5940
"-0.020158" is not greater than "0"
Message: import job queue wait should be non-zero

Log snippet:

Error Trace: /Users/runner/work/coder/coder/coderd/workspaces_test.go:5940
Error:       "-0.020158" is not greater than "0"
Test:        TestProvisionerJobQueueWaitMetric
Messages:    import job queue wait should be non-zero

Root cause classification: Flaky test (metric sample sum went negative/zero; timing/measurement issue on macOS).
Race/Panic/OOM evidence: none observed in logs.

Assignment analysis:

• git log --oneline -10 --follow coderd/workspaces_test.go
  • 5f3be6b feat: add provisioner job queue wait time histogram and jobs enqueued counter (Callum Styan)
    Most recent meaningful change added the metric and assertions in this test, so assigning to cstyan.

Related issues: none found (searched TestProvisionerJobQueueWaitMetric, queue wait, provisioner job queue wait metric).

Reproduction: not known; intermittent on macOS CI.

[zedkipp]

I took a first pass at this one. I don't have a smoking gun, but I do have a possible theory.

The job queue wait time metrics are recorded here: https://github.com/coder/coder/blob/ebd7ab11cbb5ff7dc6eacfd50e6d72e49eb85afd/coderd/provisionerdserver/provisionerdserver.go#L843
queueWaitSeconds := job.StartedAt.Time.Sub(job.CreatedAt).Seconds()

From a monotonic time perspective, I can't see how the start time would be before the creation time. That said, job.StartedAt.Time and job.CreatedAt go through a PostgreSQL round-trip. After the database round-trip, the time.Time values have no monotonic clock component; they're pure wall-clock times. Go's time.Sub() depends on whether both times have monotonic readings, so without monotonic readings, the time.Sub() calculation is susceptible to NTP adjustments. A 20ms step adjustment from NTP seems possible to me, but my understanding is generally for smaller adjustments NTP will slew the clock instead. That removes some confidence this is NTP related.

I would consider flooring the calculation with max(job.StartedAt.Time.Sub(job.CreatedAt).Seconds(), 0) and checking for >= 0 in the test. This would make sense to me because 1) negative values don't make sense for the metric 2) the job spent very little time in the queue anyway if we encounter this 3) the metric was indeed recorded.