Design: CRS Escalation Feature (PR #770)

[ChakshuGautam]

CRS Escalation — design hub

This thread is the design hub for the CCRS escalation feature. The canonical, self-sufficient design doc lives in the repo (link below) — this body is the index and status board, kept current so the thread never carries a stale copy again.

Status (2026-06-11): implemented and live-verified on the Bomet deployment, including opt-in role-level escalation across every resolution path, OTEL provenance, and a UI-driven operator e2e. Foundation PR #770 → PR #775 (state mapping) → PR #815 (PRD alignment + role escalation + operator UI) await review.

📖 The canonical design doc

docs/escalation-feature-design.md (2,627 lines, consolidated 2026-06-11 — the former role-escalation-design.md satellite is folded in; a redirect stub keeps old links working). It is design and method: requirements traceability, full architecture, every schema, the testing methodology, and the operational gotchas earned in production.

Section	What it answers
Terminology cheatsheet	Read-from-line-1 glossary strip
Executive summary	What shipped, where, in which PR
Requirements traceability	PRD P1–P12 + BRD → shipped / deferred / open, with evidence
Architecture	Five-source SLA cascade, the state-mapping gate, all six MDMS schemas, scheduler walk, pre-breach warnings, trace-back, audit
Role-level escalation (opt-in)	The PRD's primary journey: R1 pin → R2 ladder → R3 consensus, skip-don't-guess, provenance, hardening, rollout
Configurator UI	The operator surface: SLA Matrix, Escalation Settings, verify card, copy principles
Operational gotchas	MDMS write redelivery, SYSTEM-role bypass, persister async, x-ref recovery, #1674 root cause
Testing strategy	Testing methodology (ten incident-grounded rules) + five layers + the live verification matrix
Open questions	Everything deliberately not solved, each with an owner path

Live verification (all on the Bomet deployment, persistent fixture tenants ke.etoeroles/ke.etoebeta): pgr-escalation-full-flow 10 ✅ · pgr-escalation-role-flow 8 ✅ · pgr-escalation-r2r3-flow 9 ✅ (incl. the cross-tenant single-scan proof) · escalation-settings-flow (UI-driven) 6 ✅.

💬 Discussion prompts — status board

#	Prompt	Status
1	Configurator UI for CRS.WorkflowStateMapping	✅ Closed — Escalation Settings page ships the editor (PR #815)
2	Workflow ASSIGN-assignee persistence	✅ Root-caused + fixed: misspelled @JsonProperty("assignes") silently dropping the correct spelling — eGovStack/core-services#1674; @JsonAlias fix deployed on Bomet, chain verified end-to-end
3	Category taxonomy (constrained picker)	Roadmap G1 — PR #789 / #790
4	Path routing rules	Roadmap G2 — PR #783 / #785
5	Submission form customisation (Strategy A)	Roadmap G8 — PR #782 / #787
6	Generic CRS.ConfigAuditLog	Roadmap G4 — PR #786 / #788
7	v0 EscalationConfig deletion	Follow-up — precondition documented (CRS.EscalationPolicy is the post-v0 home for maxDepth/levels)
8	mdms-v2 oneOf validator	✅ Filed upstream — eGovStack/core-services#1675
9	Role-level escalation (PRD P4)	✅ Implemented (opt-in) + live-verified across R1/R2/R3 + cross-tenant; remaining decision: default-on for new-tenant seeding
10	Inbox ownership transfer + N+1 visibility (P5/P11)	Open — upstream workflow/inbox concern
11	Business SLA clock (P6 second half)	Open — unmodeled; BRD's single-SLA column is the likely shape
12	State-based vs level-based SLA model	Sign-off pending — both ship and coexist; per-row levels outrank state cells

New findings worth this group's attention: MDMS write redelivery (Kafka→persister re-applied an acked write ~60–80s after a verified restore — "write then verify once" is unsafe) and SYSTEM transitions bypass action-role lists in workflow-v2 (confirmed mutating live; upstream question — say the word and it gets filed alongside #1674/#1675). Both documented in the gotchas chapter.

---

Revision note: this body previously embedded a full copy of the design doc; it went stale within days of active development. As of 2026-06-11 the body is an index — the repo file is the single source of truth.

[ChakshuGautam]

Requirements traceability + 4 new discussion prompts (PR #815)

The design doc now has a Requirements traceability section mapping the CMS Escalation PRD (Draft v3.0, April 2026) and the Mozambique BRD v4.0 to shipped/deferred/open status — previously the design only cited the BRD, and several PRD requirements were unaddressed without record. PR #815 (stacked on #775) closes the implementable gaps: per-level SLAs (CRS.CategorySLA.slaHoursByLevel + CRS.EscalationPolicy), pre-breach warning detection, dryRun on /escalation/_trigger, enriched escalation audit comment, configurable manual-ESCALATE comment rule, and an SLA-clock reset fix without which decreasing per-level SLAs cascaded straight to maxDepth.

Four product decisions surfaced by the traceability pass need owners — adding them as prompts 9–12:

9. Role-level escalation (PRD §1, primary user journey). The PRD's main scenario is a complaint sitting in a role inbox (all GROs/LMEs, no named assignee) escalating to the role's direct supervisor. The scheduler currently skips these as NO_ASSIGNEES — i.e. the PRD's primary journey never escalates, independent of the upstream ASSIGN-persistence bug. The multi-department case is marked TBD in the PRD itself. Who owns deciding the resolution model?

10. Inbox ownership transfer + visibility scope (PRD §3/§4). On escalation the PRD requires: removed from subordinate inbox (all role inboxes if role-assigned), appears in supervisor's inbox, subordinate keeps search access; N+1 sees direct reports' complaints from filing, N+2+ must search. These live in egov-workflow-v2/inbox, not pgr-services. Upstream ask or CRS-side overlay?

11. Business SLA clock (PRD §3). The PRD distinguishes the per-state clock (resets on escalation — now implemented) from a business SLA clock that "continues uninterrupted." The latter is unmodeled; the BRD's Appendix-A single SLA column is arguably this. Do we model it, and if so where does it surface (dashboards? citizen ETA)?

12. State-based vs level-based SLA model sign-off. PR #815 makes the two coexist (per-row level config takes precedence over state cells). Formal sign-off requested that this precedence — and shipping both models — is the intended product position, vs deprecating one.

Prompts 1–8 unchanged. The canonical mapping table lives in docs/escalation-feature-design.md § Requirements traceability on the PR #815 branch.

[ChakshuGautam]

Status update: prompts 1, 2 and 8 — plus stacked-PR review notes

Prompt 1 (no configurator UI for CRS.WorkflowStateMapping) — CLOSED. The Escalation Settings page (/manage/escalation-settings, on the PR #815 branch, live on Bomet) now edits the status mapping with operator-labelled selects, unique-name validation and a standard-set merge — plus the escalation policy (max depth, level SLAs, pre-breach, comment rule), a live SLA-source cascade overview, and an in-product "test scan" verification card. G1 no longer needs to absorb this.

Prompt 2 (workflow ASSIGN-assignee persistence) — upstream issue FILED: eGovStack/core-services#1674 (egov-workflow-v2), with a fresh 2026-06-10 reproduction (ASSIGN process instance present, eg_wf_assignee_v2 join rows = 0).

Prompt 8 (mdms-v2 oneOf validator) — upstream issue FILED: eGovStack/core-services#1675 (egov-mdms-service / MDMS v2), including the empty-x-unique ClassCastException sibling quirk.

Stacked-PR review-ability: re-pointing PR bases to parent branches is not possible here — the stack's head branches live on the fork and a PR base must be a branch of the base repository. Instead, every stacked PR (#775, #776, #777, #779, #780, #782, #783, #786, #789, #791, #794, #796, #797) now carries a header note directing reviewers to its own commits. PR #774 (image pin) was closed as superseded by #815's local-setup compose pin.

[ChakshuGautam]

Prompt 9 (role-level escalation): design posted — and its prerequisite just got fixed

Design: docs/role-escalation-design.md (on the PR #815 branch), panel-reviewed (PRD-fidelity / DIGIT-feasibility / ops lenses). Shape: opt-in via EscalationPolicy.roleEscalation (disabled = byte-identical to today); per-state acting-role map; resolution R1 explicit pin (CRS.RoleSupervisors) → R2 role-ladder → R3 reportingTo-consensus — always exactly one individual or an actionable skip (3 new skip reasons; ambiguity is surfaced, never guessed — the PRD itself marks the multi-supervisor case TBD); maxPerScan blast-radius cap; full provenance (strategy/candidateCount/departmentFiltered) on outcomes, Kafka and OTEL. Seeking sign-off on the skip-don't-guess stance and the opt-in default.

Prerequisite resolved: the design review correctly flagged that this feature's eligibility gate reads the same eg_wf_assignee_v2 table the ASSIGN bug fails to populate — so the bug fix had to come first. We root-caused it ourselves today (it is not a persistence bug): the workflow API binds @JsonProperty("assignes") and lenient deserialization silently drops the correctly-spelled "assignees" key clients actually send. Corrected analysis on eGovStack/core-services#1674; one-line @JsonAlias fix deployed to Bomet (egov-workflow-v2:maven-jdk21-43f925c2, built from the exact production commit, 311/311 tests). Verified live: ASSIGN with the natural spelling now persists assignees, and the auto-escalation chain ran end-to-end with no manual SQL fixup for the first time.

[ChakshuGautam]

Prompt 9 closed: role-level escalation IMPLEMENTED + live-verified (opt-in)

The design posted earlier is now implemented on the PR #815 branch and verified end-to-end on Bomet: an unassigned complaint escalated via the R1 pin to the role's supervisor, with full OTEL provenance asserted from Tempo (escalation.complaint child spans), dryRun parity, and the policy restored byte-identical afterward. Disabled config = byte-identical behavior (serialization-pinned). Details on the PR. The remaining P4 question for this thread is now narrower: when (if ever) to make it default-on for new-tenant seeding — the design doc sketches the path.

[ChakshuGautam]

Design-hub update: full verification matrix + testing methodology now part of the design docs

Role-level escalation (prompt 9) is now verified across every resolution path, live, on persistent fixture tenants built for this purpose (ke.etoeroles/ke.etoebeta on the Bomet test box, rebuilt idempotently by tests/integration-tests/scripts/setup-role-fixture.mjs):

Suite	Proves
pgr-escalation-full-flow (10)	Named-assignee chain + per-level SLA + OTEL child spans
pgr-escalation-role-flow (8)	R1 pin, dryRun parity, policy snapshot/restore, Tempo provenance
pgr-escalation-r2r3-flow (9)	R2 exactly-one (real), R2 ambiguous, R3 consensus (real), R3 split, cross-tenant memo proof — one scan over two tenants escalated each complaint to its own tenant's holder
escalation-settings-flow (6, UI-driven)	The real operator journey: enable → ladder config → save → test scan → disable, with API asserts that the UI persisted exactly what the scheduler reads

The design doc now also carries a Testing methodology section — ten binding rules each grounded in a real incident from this work (regression tests proven failing without the fix; surgical never-global test config; cron interference measured via sentinels; shared-singleton restores verified across the MDMS redelivery window; dry-run-first; honest residue accounting) — so the docs describe how this feature is kept correct, not just how it was designed.

Two platform findings worth this group's attention (now in the design doc's Operational gotchas):

1. MDMS write redelivery: Kafka→persister re-applied an acked write ~60–80s after a later verified write. Any "write then verify once" pattern against shared config is unsafe on this stack.
2. SYSTEM transitions bypass action-role lists in workflow-v2 (confirmed mutating, live): the escalation cron's ESCALATE succeeds at states whose actions list only GRO/AUTO_ESCALATE/PGR_VIEWER. Intended-per-PRD in effect, but the role-grant model question deserves an upstream answer — happy to file it against eGovStack/core-services alongside #1674/#1675 if there's agreement.

[subhashini-egov]

Is this tied to a complaint type/sub-type hierarchy? What if we moved to a dynamic multi-level hierarchy (upto 4 levels)?