Same-repository PR batches can livelock on optimistic repository observation conflicts

[ewhauser]

Summary

In v0.5.1, a large refresh fanout for pull requests in the same repository can repeatedly fail with apply repository: observation superseded instead of converging.

The pull-request coordinator prepares multiple PR observations and applies an incidental parent-repository observation for each repository represented in the batch. If that repository observation loses its optimistic version race, the coordinator records a repository-level failure and returns that error for every prepared PR from the repository. Concurrent batches therefore invalidate otherwise independent PR work.

Failure sequence

1. Multiple branch-refresh pages begin concurrently for one repository.
2. Each batch snapshots the same repository observation version.
3. One batch successfully advances the repository version.
4. Other batches receive ErrObservationSuperseded.
5. The repository-level error causes every PR in each losing batch to fail.
6. retrySupersededObservation retries immediately, up to three times, without backoff or jitter, so competing batches can remain synchronized.
7. River then retries the larger jobs, producing a retry stampede and very little completed PR work.

This can also affect branch-page work that skips its own repository observation if the coordinator batch contains another candidate that does request the incidental repository refresh.

Expected behavior

A conflict while refreshing incidental repository metadata should not invalidate independently authoritative PR observations for the entire batch. Concurrent same-repository fanout should converge without synchronized retries.

Possible direction

Any of these approaches could break the failure amplification:

• Coalesce or serialize incidental repository refreshes by repository.
• Detach the parent-repository refresh from the PR batch and enqueue it independently.
• Treat a repository CAS conflict as non-fatal for independently applicable PR observations.
• If a refetch is required, use bounded jitter/backoff or singleflight rather than immediate synchronized retries.

Acceptance criteria

• Exercise at least four concurrent branch-refresh pages of 25 PRs from one repository, including sweep-originated PR work.
• A repository observation conflict does not fail all PR observations in the losing batch.
• The workload converges without a retry stampede.
• Event-to-refresh latency remains bounded under same-repository fanout.

Affected version: v0.5.1