feat(joblifecycle): the queue journal speaks every transition once - #335
Merged
Conversation
Latency questions were answered by correlating GARM journals by hand. The journal already holds each intent's current state with the time it was entered; export-job-lifecycle samples it and emits one record per observed transition into gha_fleet_job_lifecycle -- the whole queued-acquiring-acquired-assigned-running ladder with the journal's own timestamps and waited_seconds from first queue time, a completed record when the intent reaches the terminal map, and a vanished record when it disappears without one, which is the orphan this fleet has been burned by. Watermarks advance only after a successful delivery, so a failed export retries rather than losing records; the journal is read tolerantly so a schema addition never turns a read-only exporter into a two-deploy dance. Claude-Session: https://claude.ai/code/session_01LsGid6U5RrQdFvJmvYdGCF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the module half of the estate ledger item
job-lifecycle-otel-stream:internal/joblifecycle(transition differ over a tolerant journal read + OpenObserve bulk exporter) andgha-fleet export-job-lifecycle(plan mode prints pending records;--applydelivers then advances watermarks, so failures retry the same records).The stream is keyed by intent/repository/run with runner and scale-set identity:
queued/acquiring/acquired/assigned/runningcarry the journal's ownstate_entered_atandwaited_seconds;completedcloses via the terminal map;vanishednames an intent that disappeared without one — the exact orphan class from the correlation-leak incident.internal/joblifecycleis not in the provider dependency set — no derivative release. Estate follow-up (rides the estate wave): systemd unit+timer on services, dashboards over the stream.https://claude.ai/code/session_01LsGid6U5RrQdFvJmvYdGCF