[Prompt lifecycle] Add eval-gated prompt profile canary and rollback controls

[azalio]

Source

• User-provided local extract: /Users/azalio/.codex/attachments/a8832a13-5476-465b-b9bb-bbac3a7292fe/pasted-text.txt
• Source article: Rajesh Kumar, "Stop Hardcoding Your Prompts: The Case for Externalized LLM Configuration", Medium, 2026-07-11: https://medium.com/@rajesh.k.nitk/stop-hardcoding-your-prompts-the-case-for-externalized-llm-configuration-9af7f1b3e169
• Source-specific idea used here: prompts should be treated as versioned operational configuration, not as static application strings. The implementable MAP lesson is not a remote prompt gateway, but a local-first lifecycle for MAP prompt/skill surfaces: versioned prompt profiles, eval before promotion, canary cohorts, and fast rollback to the prior active profile.

Relevant source takeaways

The article argues that embedded prompts create operational friction:

• Prompt tweaks are fluid and frequent, but embedding them in application code forces them through the same CI/CD path as code changes.
• Product/domain/prompt stakeholders often iterate on prompt text, but code-bound prompts turn engineers into copy-paste proxies.
• Runtime safety is weak when a prompt update requires a full application rollout. A bad prompt should be rolled back by changing a configuration pointer, not by reverting code or redeploying an image.
• The proposed architecture externalizes prompt configuration behind a prompt id/version: runtime consumers reference a stable id while an external layer chooses the active approved version.
• Useful controls include hot fixes, canary/A-B routing, token/latency/hallucination monitoring, cross-environment sync, and instant rollback.

For MAP, the source should be translated conservatively. MAP is local-first and provider-surface based, so this should not become a hosted prompt gateway. The practical slice is to make MAP's own shipped prompt/skill/agent surfaces easier to trial, compare, promote, and roll back without editing generated files ad hoc or cutting a package release for every experiment.

Repo evidence

MAP already has strong prompt/template infrastructure, but not a prompt-lifecycle rollout control:

• README.md:169-177 documents calibrated workflow effort, mutation boundaries, XML prompt envelopes, token/ROI reports, clean retry, run-health reports, compact recovery, and Skill IR audit. These are runtime/prompt quality controls, not profile rollout controls.
• README.md:193 explains that provider-facing surfaces are installed as .claude/skills/map-*/SKILL.md for Claude Code and .agents/skills/, .codex/agents/, .codex/config.toml, hooks, and .map/scripts/ for Codex.
• README.md:207-222 documents .map/config.yaml knobs for minimality and context compression. MAP has runtime config, but those knobs tune selected behavior, not alternate prompt/skill versions.
• docs/ARCHITECTURE.md:19-30 lists MAP's in-scope provider scaffolding, settings, hooks, templates, run artifacts, SkillIR checks, and skill trigger evaluation.
• docs/ARCHITECTURE.md:73-82 names generated provider surfaces and .map/eval-runs/<skill>/ as on-disk artifacts; generated surfaces are the runtime spec consumed by providers.
• docs/ARCHITECTURE.md:86-92 says mapify init copies provider prompts/skills/scripts and mapify skill-eval evaluates trigger/cost behavior. There is no command described for selecting a prompt profile/channel or rolling back a changed prompt surface.
• docs/ARCHITECTURE.md:94-104 identifies CLI templates and delivery code as source of truth for what mapify init installs. That means normal prompt/skill changes are release-bound through template source, not locally canaryable as configuration.
• docs/SKILL-EVAL.md:8-20 shows the shipped eval engine is trigger accuracy and cost tuning for skill activation only. It explicitly tunes only the skill description: field, not the skill body/agent prompt lifecycle.
• docs/SKILL-EVAL.md:51-58 shows --apply patches the winning description into templates_src/... and re-renders; it is a template-source patch path, not a runtime profile/promotion path.
• docs/whole-skill-optimization-flow.md:10-17 defines body/outcome evaluation as human-in-the-loop measurement with deterministic gates plus an LLM judge.
• docs/whole-skill-optimization-flow.md:78-86 recommends one conceptual body edit, spot-check, full regression, held-out checks, and saved hypotheses. This is the right evaluation discipline, but it is a procedure, not a first-class canary/rollback mechanism.
• src/mapify_cli/templates_src/skills/README.md.jinja:49-68 documents skill authoring and template sync, including make render-templates and tests. It reinforces that prompt/skill changes are currently handled as source edits plus render checks.

Existing issue search

Live duplicate checks performed on 2026-07-12:

• gh issue list --repo azalio/map-framework --state all --limit 120 --search "externalized prompt OR prompt registry OR prompt gateway OR prompt version OR prompt canary OR prompt rollback OR hotfix prompts" returned no matches.
• gh issue list --repo azalio/map-framework --state all --limit 120 --search "skill prompt OR agent prompt OR SKILL.md OR templates_src OR template rollout OR canary OR rollback" returned no direct matches.
• Fallback: reviewed gh issue list --repo azalio/map-framework --state all --limit 200 --json number,title,state,url,labels.

Close related issues checked:

• SpecKit-style preset composition: layered template resolution with prepend/append/wrap strategies #291 SpecKit-style preset composition: layered template resolution with prepend/append/wrap strategies is open. It covers layered template resolution, presets, extension hooks, priority, enable/disable, and resolve. It does not define an eval-gated prompt profile lifecycle, canary cohorts, promotion criteria, or rollback pointer for MAP prompt/skill/agent surfaces. This issue should reuse or compose with SpecKit-style preset composition: layered template resolution with prepend/append/wrap strategies #291 if that lands, but it is not the same slice.
• [Claude Code Prompt Library] Add MAP prompt-pattern catalog and prompt-improvement audit #326 [Claude Code Prompt Library] Add MAP prompt-pattern catalog and prompt-improvement audit is closed. It covered user-facing prompt recipes and an audit of prompt patterns. It explicitly guarded against changing internal agent/skill prompt bodies without eval. It did not add runtime/profile rollout controls.
• [Governance] Add behavior-shaping asset manifest and audit report #342 [Governance] Add behavior-shaping asset manifest and audit report is closed. It inventories prompts, skills, evals, hooks, and access rules as behavior-shaping assets. It does not let maintainers canary or roll back prompt versions.
• [arXiv:2607.06624] Add trajectory-level outcome eval with side-by-side regression reports #351 [arXiv:2607.06624] Add trajectory-level outcome eval with side-by-side regression reports is open. It should provide the outcome/regression evidence needed before promotion, but it does not define how MAP selects, installs, canaries, or rolls back alternate prompt profiles.
• [agentry] Add install manifest/lock for MAP-managed provider surfaces #313 [agentry] Add install manifest/lock for MAP-managed provider surfaces is closed. It covers aggregate install locks and reproducibility of generated provider surfaces, not lifecycle promotion of alternate prompt versions.

No direct duplicate was found.

Why this is not already covered

MAP's current model is strong for source-controlled template quality, but weak for operational prompt experimentation:

• Core prompt/skill/agent changes are tied to templates_src edits, render checks, and package/source changes.
• Project-local overrides are not first-class prompt profiles with versions, promotion state, baseline linkage, or rollback metadata.
• skill-eval can tune trigger descriptions, and whole-skill optimization can measure outcomes, but neither provides a safe activation mechanism for a candidate prompt surface.
• Existing governance/reporting issues inventory behavior-shaping assets, but do not control which version is active.

The missing surface is a local-first prompt lifecycle controller: candidate profiles can be installed, evaluated, activated for a bounded cohort, compared to a baseline, and rolled back with a deterministic pointer change.

Problem

MAP maintainers and adopters need to improve load-bearing prompt/skill/agent surfaces without either:

1. editing generated files directly in a target repo;
2. forking MAP templates permanently;
3. cutting a full release for every prompt experiment;
4. relying on memory/manual notes to remember which prompt variant is active;
5. discovering regressions only after a prompt/profile becomes the default.

This is especially risky because MAP prompts govern workflow sequencing, mutation boundaries, verification, reporting trust, and escalation. A small wording change can improve one scenario while degrading another.

Proposed slice

Add an opt-in, local-first prompt profile lifecycle for MAP provider surfaces.

Possible first implementation shape:

• Add a prompt/profile manifest format, for example .map/prompt-profiles/<profile-id>/manifest.json, that records:
  • profile id, title, owner, creation time;
  • target surfaces (skill:map-efficient, agent:actor, agent:monitor, reference:map-xml-prompt-envelopes, etc.);
  • base template hash / source hash;
  • changed files or patch references;
  • intended behavior change and rollback notes;
  • eval requirements before promotion.
• Add a current-selection pointer, for example .map/prompt-profiles/active.json, with active profile/channel per provider or per skill/agent. Changing the pointer should be the rollback primitive.
• Add commands such as:
  • mapify prompt-profile list
  • mapify prompt-profile diff <profile>
  • mapify prompt-profile activate <profile> --scope <skill-or-agent> --dry-run
  • mapify prompt-profile rollback --scope <skill-or-agent>
  • mapify prompt-profile report <profile>
• Keep the first slice local and file-based. Do not add a hosted service or remote prompt gateway.
• Integrate with existing evaluators:
  • trigger-only changes can require mapify skill-eval run/optimize evidence;
  • body/agent prompt changes can require the future [arXiv:2607.06624] Add trajectory-level outcome eval with side-by-side regression reports #351 trajectory evaluator or the documented whole-skill eval harness;
  • report activation state and eval evidence in the governance report from [Governance] Add behavior-shaping asset manifest and audit report #342 if available.
• Support a conservative canary model suitable for local MAP:
  • a profile may be activated only for selected skills/agents, selected target repo, or selected eval fixture cohort;
  • no random production traffic routing in this slice;
  • promotion to default should require explicit command plus passing configured gates.
• Preserve generated-template invariants: do not edit generated trees as the source of truth. If activation materializes files into .claude/, .codex/, or .agents/skills/, it must be traceable to the profile manifest and checkable by mapify check / install-manifest logic.

Acceptance criteria

• A maintainer can create or register a prompt profile for at least one MAP skill or agent without editing generated files directly.
• mapify prompt-profile list/diff/activate/rollback or equivalent documented commands exist and are covered by tests.
• Activation writes or materializes provider-facing files deterministically, records the active profile pointer, and can roll back to the previous baseline without requiring a package release.
• Dry-run mode shows exactly which provider surfaces would change and which eval evidence is required/missing.
• A profile manifest records base hashes and target surfaces so mapify check or a dedicated check can detect active-profile drift.
• At least one fixture/test demonstrates a candidate profile activation, failed eval/promotion block, and rollback back to baseline.
• Documentation explains when to use prompt profiles versus normal template-source edits, SpecKit-style preset composition: layered template resolution with prepend/append/wrap strategies #291 preset composition, trigger-only skill-eval, and [arXiv:2607.06624] Add trajectory-level outcome eval with side-by-side regression reports #351 trajectory eval.

Guardrails

• Do not build a SaaS prompt gateway or remote registry in this slice. MAP is local-first.
• Do not bypass template/source invariants by treating generated .claude/, .codex/, or .agents/skills/ files as the durable authoring source.
• Do not make canary mean random live user traffic routing. Start with explicit local/eval cohorts and target-repo scoped activation.
• Do not promote prompt profiles based only on subjective preference. Require deterministic checks and/or trajectory eval evidence appropriate to the surface being changed.
• Do not store secrets, private transcripts, or local absolute paths in committed profile manifests.
• Do not duplicate SpecKit-style preset composition: layered template resolution with prepend/append/wrap strategies #291. If preset composition lands first, prompt profiles should use it as the composition/rendering substrate rather than adding a second layering engine.

[azalio]

Taking this into work.

Implementing slice 1: manifest format + mapify prompt-profile list.

Key design decisions for slice 1:

• .map/prompt-profiles/<id>/manifest.json — required fields: id, title, version; optional: description, owner, targets (list of {"surface": "skill:map-efficient"} etc.), eval_requirements, rollback_notes
• .map/prompt-profiles/active.json — {"active": "<id>" | null} pointer for rollback primitive
• mapify prompt-profile list [--project-path <path>] [--json] — shows all profiles, marks which is active
• Guardrail: no editing generated trees; profile manifests are metadata only in slice 1

Later slices will add: diff, activate, rollback, report commands plus integration with #291 preset composition as the rendering substrate.

---

Generated by Claude Code

[azalio]

Slice 1 implemented in PR #374.

What's in slice 1:

• mapify prompt-profile list [--project-path <path>] [--json] command
• Manifest format: .map/prompt-profiles/<id>/manifest.json (required: id, title, version; optional: description, owner, targets, eval_requirements, rollback_notes)
• Active pointer: .map/prompt-profiles/active.json with {"active": "<id>"|null} as rollback primitive
• Graceful degradation for missing/malformed manifests
• Stale active-pointer warning
• 20 tests in tests/test_prompt_profile_commands.py

Future slices (not in this PR): diff, activate, rollback, report commands + integration with #291 preset composition as the rendering substrate for profile materialization.

---

Generated by Claude Code