Skip to content

feat: monitor tasks, Prometheus metrics, and seictl v0.0.24#33

Merged
bdchatham merged 3 commits intomainfrom
claude/dreamy-williamson
Mar 28, 2026
Merged

feat: monitor tasks, Prometheus metrics, and seictl v0.0.24#33
bdchatham merged 3 commits intomainfrom
claude/dreamy-williamson

Conversation

@bdchatham
Copy link
Copy Markdown
Collaborator

Summary

  • Adds monitor task infrastructure — a third task category that the controller submits once in Running phase, polls each reconcile, and reacts to on completion with Conditions, Events, and Prometheus metrics
  • First monitor task: result-export comparison mode — sidecar compares block results against a canonical RPC and completes on app-hash divergence
  • Sidecar-lost-task detection via ErrNotFound after synchronous submission
  • ResultExportComplete Condition with reasons DivergenceDetected, TaskFailed, TaskLost
  • Prometheus metrics: sei_controller_monitor_task_completed_total (counter) and sei_controller_monitor_task_status (gauge)
  • Renames PlannedTaskStatusTaskStatus for cross-context clarity
  • Removes dead scheduled-mode result-export code (CanonicalRPC is now required)
  • Bumps seictl to v0.0.24 and sidecar image to sha256:63860a7c

Test plan

  • ensureMonitorTask: submit once, idempotent skip, submit error
  • pollMonitorTasks: completed, failed, still running, skip completed, sidecar lost task, invalid UUID, transient error, unknown error
  • reconcileRuntimeTasks: monitor mode submission, replayer without ResultExport, poll requeue, conflict early return
  • Planner: ResultExportMonitorTask returns/nil for all node types
  • Conditions verified on all terminal paths
  • make manifests generate — CRDs regenerated
  • golangci-lint run — 0 issues
  • Full test suite passes

Add counter and gauge metrics for monitor task lifecycle:
- sei_controller_monitor_task_completed_total: terminal state counter
  with reason labels (DivergenceDetected, TaskFailed, TaskLost)
- sei_controller_monitor_task_status: gauge showing current status
  per task type (Pending/Complete/Failed)

Also updates shadow replayer manifest description to reflect the
comparison mode export model.
@bdchatham bdchatham force-pushed the claude/dreamy-williamson branch from 2a6cbf6 to 125be8b Compare March 28, 2026 00:44
@bdchatham bdchatham force-pushed the claude/dreamy-williamson branch from 125be8b to 99dc37b Compare March 28, 2026 00:47
@bdchatham bdchatham marked this pull request as ready for review March 28, 2026 00:48
@bdchatham bdchatham merged commit 118977c into main Mar 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant