Skip to content

fix(alertmanager): route pod health alerts - #634

Merged
kkroo merged 1 commit into
masterfrom
blo-14625-pod-health-alert-routes
Jul 11, 2026
Merged

fix(alertmanager): route pod health alerts#634
kkroo merged 1 commit into
masterfrom
blo-14625-pod-health-alert-routes

Conversation

@allyblockcast

@allyblockcast allyblockcast Bot commented Jul 11, 2026

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work.
  • The Alertmanager plugin converts firing Prometheus alerts into Paperclip issues.
  • The plugin already has bundled class-based issue routes for Blockcast physical infrastructure alerts.
  • Pod-health alerts emitted by onprem-k8s used class labels but lacked bundled routes, so new alert issues could be created unassigned.
  • This pull request extends the existing class-route map rather than changing webhook control flow.
  • The benefit is that fresh deploys assign pod-health alert issues to the Platform/SRE owner at creation while instance config can still override routes.

Linked Issues or Issue Description

No GitHub issue exists for this; tracked in the Blockcast Paperclip board as BLO-14625 (refs BLO-14465 baseline, BLO-12202 pattern). Inline issue description per the bug report template:

What happened?

Prometheus pod-health alerts (class labels pod_pending, pod_init_stuck, pod_crashloop, pod_create_error, pod_config_error, pod_image_pull) delivered through paperclip-plugin-alertmanager create issues with no assignee, because these classes are missing from the bundled DEFAULT_ISSUE_ROUTE_MAP. A live severity-critical alert issue (board id BLO-14465) sat unowned with assigneeAgentId: null until it self-cancelled — nothing woke an owner, and the org's stalled-run escalation machinery never engages for unassigned issues.

Expected behavior

Pod-health alert issues are created pre-assigned to the Platform/SRE owner (same treatment the bundled map already gives the physical_infra_* classes), so the owner is woken at creation and escalation machinery is reachable.

Steps to reproduce

  1. Configure Alertmanager to deliver to the plugin webhook with the bundled default config (no instance issueRouteMap override).
  2. Fire any alert carrying class: pod_pending (e.g. PodPendingTooLong).
  3. Observe the created issue has assigneeAgentId: null and status backlog, instead of an assigned todo issue.

What Changed

  • Added bundled DEFAULT_ISSUE_ROUTE_MAP entries for pod_pending, pod_init_stuck, pod_crashloop, pod_create_error, pod_config_error, and pod_image_pull.
  • Each new class route targets the existing Blockcast physical-infra project/goal and Staff Engineer agent assignee.
  • Added worker coverage proving all six pod-health classes create issues assigned to the Platform/SRE agent route.

Verification

  • pnpm --filter paperclip-plugin-alertmanager test
  • pnpm --filter paperclip-plugin-alertmanager typecheck

Risks

  • Low risk: config-only default route expansion.
  • Behavioral shift is intentional: matching pod-health alerts now create assigned todo issues instead of unassigned issues.
  • Runtime verification still requires the first post-deploy pod-health alert-created issue to show non-null assigneeAgentId.

Model Used

  • OpenAI GPT-5.5 (openai/gpt-5.5) with tool use in the Paperclip OpenCode runtime.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • If this change affects the UI, I have included before/after screenshots
  • I have updated relevant documentation to reflect my changes
  • I have considered and documented any risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups
  • I will address all Greptile and reviewer comments before requesting merge

@allyblockcast

allyblockcast Bot commented Jul 11, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-14465
🔗 Paperclip issue: BLO-12202
🔗 Paperclip issue: BLO-14625

1 similar comment
@allyblockcast

allyblockcast Bot commented Jul 11, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-14465
🔗 Paperclip issue: BLO-12202
🔗 Paperclip issue: BLO-14625

@allyblockcast

allyblockcast Bot commented Jul 11, 2026

Copy link
Copy Markdown
Author

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

@allyblockcast

allyblockcast Bot commented Jul 11, 2026

Copy link
Copy Markdown
Author

@ally please review this config-only Alertmanager routing change for BLO-14625. Focus: whether the new pod-health class routes correctly mirror the BLO-12202 issueRouteMap pattern, preserve instance override behavior, and ensure alert issues get a non-null assigneeAgentId at creation without changing owner override precedence.

@kkroo
kkroo merged commit 872d326 into master Jul 11, 2026
18 of 19 checks passed

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.

Looks good

This is a clean, purely additive config change. Verified:

  • All six new class label values (pod_pending, pod_init_stuck, pod_crashloop, pod_create_error, pod_config_error, pod_image_pull) match exactly what onprem-k8s's Prometheus rules (monitoring/prometheus-configmap.yaml) actually emit — no naming drift, which is the exact failure mode this PR is fixing (a silent route miss).
  • mergeIssueRouteMap/resolveIssueRoute in worker.ts / issue-route-resolver.ts correctly pick up the new bundled defaults while still letting instance config override per-class.
  • The new it.each test block asserts the actual handleWebhook output (projectId/goalId/status/assigneeAgentId/assigneeUserId), matching production behavior — not a tautological check against the map itself.
  • CI is green at afbe4494 (build, typecheck, all test shards, e2e, policy, security-review).

Suggestions (1)

  • [code] packages/plugins/paperclip-plugin-alertmanager/src/constants.ts — The existing physical_infra_* classes are dual-mapped in both DEFAULT_OWNER_MAP (email fallback) and DEFAULT_ISSUE_ROUTE_MAP (agent assignment), but the six new pod_* classes are only added to DEFAULT_ISSUE_ROUTE_MAP. Not a functional bug today — issueRouteMap resolution takes precedence over ownerMap when both apply, so pod-health issues are assigned correctly. But if a future edit ever removes/renames a pod_* entry from DEFAULT_ISSUE_ROUTE_MAP without a corresponding DEFAULT_OWNER_MAP fallback, these classes silently regress to unassigned — the exact bug this PR fixes. Consider adding matching pod_*support@blockcast.net entries to DEFAULT_OWNER_MAP for defense-in-depth, following the physical_infra_* pattern.

Recommended Action

Non-blocking suggestion only — safe to merge as-is.

Distinct-reviewer token unavailable for formal approval on this repo (read-only permission); review posted as comment, formal approval requires a human.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant