Alert when a Crossplane managed resource stops reconciling (Ready=True hides Synced=False)

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

17 of the 20 Repository managed resources in github-config sat in a failing sync state for weeks with no signal. Measured on the live cluster 2026-07-27:

• Every one of the 17 reports Synced: False.
• Every one of the 17 simultaneously reports Ready: True.

That second line is why nothing surfaced it. Any health check keyed on Ready — which is the obvious one — reads this fleet as entirely healthy. Meanwhile no declared setting was reaching GitHub at all, including the visibility pins two repositories rely on. The control was voided silently, which is worse than not having it, because the config file still reads as the truth.

This is the detection half of devantler-tech/.github#123. The trigger for that specific outage is fixed (devantler-tech/.github#125), but nothing would surface the next one.

Problem

There is currently no mechanism that could raise this. docs/dr/alerting.md records that Coroot replaced the kube-prometheus-stack assembly, and with it went the PrometheusRule / ServiceMonitor / PodMonitor CRDs — there is no prometheus-operator in the cluster. Coroot's built-in alerting is SLO-based plus automatic inspections (node down, OOM kills, crashloops, disk filling, deployment issues, saturation). Those cover workload health well, and none of them observes a Crossplane managed resource's Synced condition.

So "alert on Synced: False" is not a rule we can simply author — the usual surface for it does not exist here. That is the actual work in this issue, and it is why it is worth its own item rather than a line on the parent.

Expected behaviour

A managed resource that stops reconciling raises a signal to the same Slack destination Coroot already delivers to, rather than sitting quietly. Scope it to the condition, not to one provider: this failure mode is generic to every Crossplane managed resource, not specific to GitHub repositories.

Worth weighing (not a decision):

1. A small exporter that surfaces managed-resource conditions as metrics Coroot's bundled Prometheus can scrape, then a Coroot alert on it.
2. A Kyverno policy or admission-time check that reports resources stuck in Synced: False beyond a threshold.
3. A scheduled job that queries the conditions directly and posts to the existing webhook.

Option 1 generalises best and reuses the delivery path already wired declaratively; options 2 and 3 are cheaper.

Acceptance criteria

• A managed resource held in Synced: False beyond a short grace period produces an alert, delivered through the existing Coroot → Slack path.
• The check keys on Synced, not on Ready — the two disagreed on 17 resources here, and Ready alone is what made this invisible.
• Applies to any Crossplane managed resource, not just Repository.
• RED-proved: a deliberately broken resource fires it, and a healthy fleet does not (no standing false alarm).

Rough size: M.

Part of devantler-tech/.github#123

[coderabbitai]

🔗 Related PRs

#2445 - feat(kubescape): route runtime-detection alerts to Headlamp, Slack, and Coroot [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[devantler]

🤖 Generated by the Agentic Engineer

Measured design finding: the obvious grace-period implementation would be exactly inverted

I took this off the queue and built the detection query against admin@prod before writing any manifest. The query works, but live data killed the design I was about to ship — recording it here because it is a landmine for whoever picks this up next (including me).

The natural implementation is: list managed resources, keep those with Synced != True, and suppress anything younger than a grace period using the condition's lastTransitionTime. That is wrong here, in both directions at once.

1. lastTransitionTime is rewritten on every reconcile, so a persistently failing resource is permanently "brand new"

The 7 GitHub Repository resources that are genuinely broken (reason=ReconcileError) — the exact fleet this issue was opened about — measured at 16:41:00Z:

resource	reason	age of lastTransitionTime
agent-plugins	ReconcileError	50s
agent-skills	ReconcileError	15s
aws	ReconcileError	5s
doggy-countdown	ReconcileError	2s
kyverno-policies	ReconcileError	27s
platform-tenant-template	ReconcileError	38s
provider-upjet-unifi	ReconcileError	57s

Confirmed by re-sampling one resource 75s apart, with a control:

• agent-plugins (erroring): 16:41:12Z → 16:42:13Z — it moves every reconcile (~60s)
• record/admin-vault (paused) control: 2026-07-04T20:26:00Z → unchanged, 32 days

So any threshold above ~1 minute never fires for a resource that has been failing for weeks. The alert would be a silent no-op on precisely the failure mode this issue exists to catch — a control that reads as working while surfacing nothing, which is the same class of defect as the Ready=True mask.

2. Meanwhile the only resources that do age past a grace period are the deliberately paused ones

9 unifi resources sit at Synced=False reason=ReconcilePaused, stable for 764h. A lastTransitionTime threshold selects exactly these and nothing else — a standing false alarm, which acceptance criterion 4 explicitly rules out.

Net: threshold-on-lastTransitionTime alerts on all the deliberate states and none of the real failures.

What this changes

• reason must be part of the filter, not just status. ReconcilePaused is a declared intent (crossplane.io/paused), not a fault, and must be excluded.
• Persistence cannot be read off the resource. There is no "unsynced since" timestamp that survives a retry, so the grace period has to come from the observer — two consecutive observations (small ConfigMap of last-seen keys), or an exported metric whose alert rule owns the for: duration. This is what makes option 1 (exporter + Coroot alert) materially stronger than option 3 (job posts straight to the webhook): the for: duration belongs in the alerting engine, which already has one, rather than being reinvented as job state.

Also worth recording

• kubectl get managed -A is a sound generic surface: 109 resources across 7 API groups, and it does cover the Repository fleet (45 of them). Scoping to a provider is unnecessary.
• RBAC needs no new ClusterRole: the existing crossplane-view is aggregated (rbac.crossplane.io/aggregate-to-view), so it picks up new providers automatically, and I verified it grants no core secrets access. Bind a ServiceAccount to it.

Unassigning myself so this is free to pick up — the remaining work is the persistence mechanism, which is a real design choice rather than a typing exercise.

[devantler]

🤖 Generated by the Agentic Engineer

The delivery path is settled — except one hop, which is where this would silently no-op a third time

Picking this up after the lastTransitionTime finding above. That finding killed the detection
design; this one is about the delivery design, and it has the same shape: the obvious
implementation looks correct and reaches nobody.

1. Option 1 is real — Coroot CE has native PromQL alerting rules, for: included

The design finding above argued option 1 was stronger because "the for: duration belongs in the
alerting engine, which already has one". That engine is real and richer than the issue body assumed.
Read from the live CRD (coroots.coroot.com, admin@prod), spec.projects[].alertingRules[]:

field	value
source.type	check | log_patterns | kubernetes_events | promql
source.promql.expression	"PromQL expression that triggers the alert when it returns results"
for / keepFiringFor	^[0-9]+[smhdwy]$
severity	warning | critical
selector.type	all | category | applications
notificationCategory	"Override the notification category for this rule"

So the persistence problem the earlier finding raised — "there is no unsynced since timestamp that
survives a retry, so the grace period has to come from the observer" — is solved by for:, with no
ConfigMap of last-seen keys and no reinvented job state. Rules declared in the CR are locked in the
UI, so they cannot be edited or deleted out from under the manifest.

2. But the existing Coroot → Slack path does not carry alerts at all

Acceptance criterion 1 says the alert is "delivered through the existing Coroot → Slack path". That
path is deliberately muted for alerts. From patches/enable-ha.yaml:

notificationIntegrations:
  webhook:
    incidents: true
    alerts: false        # <-- built-in inspection ALERTS stay in the UI only

That is not an oversight — it is a reviewed decision, because #notifications was carrying ~250+
[warning] alerts per window against effectively zero incidents, and the fan-out was buried. Only
incidents (SLO breaches) reach Slack today.

A custom PromQL rule is an alert, not an incident. So the natural implementation — exporter,
plus the PromQL rule above — fires correctly, shows up in the UI, and notifies no one. That is
the same failure class as Ready: True masking the fleet and as the lastTransitionTime threshold:
a control that reads as working while surfacing nothing. Third instance on this one issue.

3. The one unresolved question, stated precisely

Per-category routing exists and is per-integration — verified in the same CRD,
applicationCategories[].notificationSettings:

{incidents, deployments, alerts} × {slack, teams, pagerduty, opsgenie, webhook}.enabled

So the intended fix is a dedicated category for this signal with alerts.webhook.enabled: true,
leaving every other category muted. Whether that works depends on an undocumented precedence:

Does integration-level webhook.alerts: false act as a hard global gate, or is the category-level
notificationSettings.alerts.webhook.enabled: true evaluated independently?

Coroot's own docs describe the category gate ("Coroot determines the application's category and
checks the category's notification settings… the notification is silently skipped") but state no
precedence against the integration-level toggle. The webhook page does not document the
integration-level toggles at all.

I am not guessing this one. If the integration toggle wins, the whole feature ships dead, and
the wording — silently skipped — means it fails exactly as quietly as the two designs before it.
Resolving it needs one live probe against the Coroot API (in-cluster, authAnonymousRole: Admin,
no credential required) or a single UI check; a port-forward is outside my current grant, so I
could not settle it in this run.

4. The exporter: no bespoke image needed, and no new ClusterRole

Worth recording so the next run does not start by writing Go:

• clusterAgent.kubeStateMetrics exposes only image — no CustomResourceState config — so the
  bundled cluster-agent cannot be configured to emit these metrics.
• There are no ServiceMonitor/PodMonitor CRDs in the cluster (prometheus-operator is gone
  with the kube-prometheus-stack), so the usual scrape-config surface does not exist either.
• The supported surface is pod annotations, which the cluster-agent discovers:
  coroot.com/scrape-metrics: "true" + coroot.com/metrics-port, optional -path / -scheme.
  Nothing in k8s/ uses these yet — this would be the first, alongside the existing
  coroot.com/postgres-scrape integration on umami-db / backstage-db.
• platform's Go module is yaml.v3 only — there is no image build/publish pipeline here, so a
  bespoke exporter is a project in itself, not an increment. Upstream kube-state-metrics with a
  CustomResourceState ConfigMap gets the same metrics from a pinned upstream image with purely
  declarative config, and keeps this industry-standard rather than bespoke. Its one real cost is
  that CRS is per-GVK, and kubectl get managed -A spans 7 API groups.
• RBAC still needs nothing new: the earlier finding verified crossplane-view is aggregated and
  grants no core secrets. Binding a ServiceAccount to it avoids authoring a ClusterRole — which
  also avoids tripping the EKS authorization-surface fingerprint gate.

Decomposing

The routing unknown blocks the alert but not the metric, and the metric is a hard precondition for
all three options in the issue body. Splitting on that seam so the unblocked half can move:

1. Expose Crossplane managed-resource sync state as metrics — KSM + CustomResourceState, scraped
   via the annotations above, reason carried as a label so ReconcilePaused is excludable.
   Independently useful: today the fleet is not queryable at all. RED-provable against live data
   (7 resources at reason=ReconcileError right now; the 9 paused unifi resources are the control).
2. Alert on it and route it to Slack — the PromQL rule plus the dedicated category, gated on
   settling the precedence question in §3 first.

Filing both as sub-issues of this one. Acceptance criteria 2–4 on this issue are unchanged and are
inherited by child 1; criterion 1 belongs to child 2 and is what the precedence question decides.