fix(alertmanager): route pod health alerts - #634
Conversation
1 similar comment
|
Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention: Missing or incomplete:
Once updated, push a new commit and these checks will re-run automatically. — commitperclip |
|
@ally please review this config-only Alertmanager routing change for BLO-14625. Focus: whether the new pod-health class routes correctly mirror the BLO-12202 issueRouteMap pattern, preserve instance override behavior, and ensure alert issues get a non-null assigneeAgentId at creation without changing owner override precedence. |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Looks good
This is a clean, purely additive config change. Verified:
- All six new
classlabel values (pod_pending,pod_init_stuck,pod_crashloop,pod_create_error,pod_config_error,pod_image_pull) match exactly what onprem-k8s's Prometheus rules (monitoring/prometheus-configmap.yaml) actually emit — no naming drift, which is the exact failure mode this PR is fixing (a silent route miss). mergeIssueRouteMap/resolveIssueRouteinworker.ts/issue-route-resolver.tscorrectly pick up the new bundled defaults while still letting instance config override per-class.- The new
it.eachtest block asserts the actualhandleWebhookoutput (projectId/goalId/status/assigneeAgentId/assigneeUserId), matching production behavior — not a tautological check against the map itself. - CI is green at
afbe4494(build, typecheck, all test shards, e2e, policy, security-review).
Suggestions (1)
- [code]
packages/plugins/paperclip-plugin-alertmanager/src/constants.ts— The existingphysical_infra_*classes are dual-mapped in bothDEFAULT_OWNER_MAP(email fallback) andDEFAULT_ISSUE_ROUTE_MAP(agent assignment), but the six newpod_*classes are only added toDEFAULT_ISSUE_ROUTE_MAP. Not a functional bug today —issueRouteMapresolution takes precedence overownerMapwhen both apply, so pod-health issues are assigned correctly. But if a future edit ever removes/renames apod_*entry fromDEFAULT_ISSUE_ROUTE_MAPwithout a correspondingDEFAULT_OWNER_MAPfallback, these classes silently regress to unassigned — the exact bug this PR fixes. Consider adding matchingpod_*→support@blockcast.netentries toDEFAULT_OWNER_MAPfor defense-in-depth, following thephysical_infra_*pattern.
Recommended Action
Non-blocking suggestion only — safe to merge as-is.
Distinct-reviewer token unavailable for formal approval on this repo (read-only permission); review posted as comment, formal approval requires a human.
Thinking Path
Linked Issues or Issue Description
No GitHub issue exists for this; tracked in the Blockcast Paperclip board as BLO-14625 (refs BLO-14465 baseline, BLO-12202 pattern). Inline issue description per the bug report template:
What happened?
Prometheus pod-health alerts (
classlabelspod_pending,pod_init_stuck,pod_crashloop,pod_create_error,pod_config_error,pod_image_pull) delivered through paperclip-plugin-alertmanager create issues with no assignee, because these classes are missing from the bundledDEFAULT_ISSUE_ROUTE_MAP. A live severity-critical alert issue (board id BLO-14465) sat unowned withassigneeAgentId: nulluntil it self-cancelled — nothing woke an owner, and the org's stalled-run escalation machinery never engages for unassigned issues.Expected behavior
Pod-health alert issues are created pre-assigned to the Platform/SRE owner (same treatment the bundled map already gives the
physical_infra_*classes), so the owner is woken at creation and escalation machinery is reachable.Steps to reproduce
issueRouteMapoverride).class: pod_pending(e.g. PodPendingTooLong).assigneeAgentId: nulland statusbacklog, instead of an assignedtodoissue.What Changed
DEFAULT_ISSUE_ROUTE_MAPentries forpod_pending,pod_init_stuck,pod_crashloop,pod_create_error,pod_config_error, andpod_image_pull.Verification
pnpm --filter paperclip-plugin-alertmanager testpnpm --filter paperclip-plugin-alertmanager typecheckRisks
todoissues instead of unassigned issues.assigneeAgentId.Model Used
openai/gpt-5.5) with tool use in the Paperclip OpenCode runtime.Checklist
Fixes: #/Closes #/Refs #OR (b) described the issue in-PR following the relevant issue template