fix(jobs): CHAOS F1 — unexpected_skip no longer silently marks propagation APPLIED

[mastermanas805]

Summary

CHAOS-DRILL-2026-05-20 P0 finding #1: propagation_runner.go handleTierElevation treated (Applied=false, SkipReason=<not-in-allowed-list>) as success, silently stamping applied_at on a propagation row whose regrade never landed. Real prod trigger: customer's postgres pod missing the postgres-admin Secret (legacy free-tier pods, mid-deprovisioning races, namespace teardown that beats the regrade).

A paying customer's tier-elevation regrade never landed — no retry, no dead-letter, no alert. Pre-fix, the only signal was a WARN log buried under thousands of routine messages.

Fix

Any non-allowed SkipReason now returns propagationUnexpectedSkipErr (errors.Is matches errPropagationUnexpectedSkipSentinel). The runner's markRetry path detects the sentinel and emits a distinct propagation.unexpected_skip audit row (NOT propagation.applied). The row retries per the existing backoff schedule and dead-letters at propagationMaxAttempts via the standard markDeadLettered path that operators already alert on.

New Prometheus counter

instant_propagation_unexpected_skip_total{kind,resource_type,skip_reason} — bounded skip_reason cardinality via bucketSkipReason(). Leading indicator for the dead-letter alert.

Suggested NR alert:

sum(rate(instant_propagation_unexpected_skip_total[15m])) > 0 for 30m → P2 page

Tests

All 5 F1 tests pass. make gate green (build + vet + go test ./... -short -count=1).

TestPropagation_UnexpectedSkip_DoesNotMarkApplied             PASS
TestPropagation_UnexpectedSkip_DeadLettersAtMaxAttempts       PASS
TestIsPropagationAllowedSkip_Coverage                         PASS
TestPropagationUnexpectedSkipErr_IsMatches                    PASS
TestBucketSkipReason_BoundsCardinality                        PASS

Coverage block (CLAUDE.md rule 17)

Symptom:        propagation.applied audit row + applied_at stamp on a
                row whose regrade never landed
Enumeration:    rg -F 'unexpected_skip'   (worker, provisioner, api)
Sites found:    1 emit site (handleTierElevation only)
Sites touched:  1
Coverage test:  TestIsPropagationAllowedSkip_Coverage iterates the
                propagationAllowedSkipSubstrings registry + asserts
                every known-failure SkipReason is treated as unexpected
Live verified:  pending — synthetic pending_propagations row pointing
                at non-existent team_id post-deploy

🤖 Generated with Claude Code