fix: prompt-surface correctness (P1/P2/P3/B/C5)

[damienen]

Summary

• P1: Prompt-surface benchmarks no longer hard-FAIL on coverage violations — coverage is reported but doesn't veto the verdict for surface: prompt
• P2: Each prompt task is now scored against its own capability's criteria (not all against caps[0]) — GeneratedTask.capabilityId field added, generation tags it, runner resolves per-task criteria via resolveCriteriaForTask
• P3: Empty criteria no longer produce a vacuous 1.0 score — evaluator sets noActiveCriteria: true, score: 0; runner surfaces an actionable SKILL.md error
• Bug B: openai/ model IDs (e.g. gpt-5.4) are now exempt from dot→hyphen rewriting in fix.ts, matching openrouter/ exemption already present
• C5: Dead src/discovery/prompt.ts (and its test file) deleted — active module is src/project/discover-prompt.ts

Test Plan

• npm run build — clean
• npm run typecheck — clean
• npm run lint — clean
• npm test — all suites pass including new smoke-verdict-prompt, smoke-prompt-criteria, smoke-changelog-coverage
• npx tsx src/cli.ts --help — CLI still works

New test files

• tests/smoke-verdict-prompt.ts — 5 scenarios: P1 regression guard, P2 caps[0]-collapse guard, P3 noActiveCriteria guard, verdict floor math, coverage exemption
• tests/smoke-prompt-criteria.ts — unit tests for resolveCriteriaForTask (match, distinct per cap, throws on unknown, throws on missing)
• tests/smoke-changelog-coverage.ts — release hygiene: every CHANGELOG Fixed/Added item must have a matching test file reference

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR fixes several correctness issues in the prompt-surface benchmark pipeline: verdict computation no longer hard-fails on coverage for prompt runs, prompt tasks are scored against the criteria for their own capability (via a new capabilityId), and empty criteria no longer yield a vacuous passing score.

Changes:

• Update prompt-surface scoring to resolve per-task evaluation criteria using capabilityId, and treat empty criteria as an evaluation error (noActiveCriteria, score 0).
• Adjust verdict logic so scopeCoverage.coverageViolation does not veto prompt-surface runs (still enforced for other surfaces).
• Add/extend smoke tests for verdict behavior, prompt criteria resolution, model-id rewriting exemptions, and changelog/test linkage; remove dead prompt discovery module/tests.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/smoke-verdict-prompt.ts	New smoke coverage for prompt-surface verdict policy + per-cap criteria + noActiveCriteria behavior
tests/smoke-release.ts	Adds release hygiene check: CHANGELOG must include current package version header
tests/smoke-prompt-evaluator.ts	Updates evaluator expectations for noActiveCriteria and non-vacuous empty-criteria scoring
tests/smoke-prompt-criteria.ts	New unit smoke tests for per-task criteria resolution (resolveCriteriaForTask)
tests/smoke-model-ids.ts	Adds defense-in-depth test ensuring openai/ IDs are not dot→hyphen rewritten in applyFixes
tests/smoke-generation.ts	Adds prompt-surface generation/grounding tests for capabilityId tagging and rejection paths
tests/smoke-discovery-prompt.ts	Removes obsolete smoke tests for deleted src/discovery/prompt.ts
tests/smoke-changelog-coverage.ts	New “CHANGELOG entries must have matching test token” guard
src/tasks/types.ts	Extends generated-task type with optional capabilityId (prompt surface)
src/tasks/ground.ts	Enforces capabilityId presence/validity for prompt-surface tasks during grounding
src/tasks/generate.ts	Prompts for and parses capabilityId on prompt-surface task generation
src/project/fix.ts	Exempts openai/ IDs from dot→hyphen rewriting (matching openrouter/ exemption)
src/discovery/prompt.ts	Deletes dead prompt discovery module
src/benchmark/scoring.ts	Makes coverage violation non-blocking for prompt-surface verdicts
src/benchmark/runner.ts	Resolves prompt evaluation criteria per-task via capabilityId; handles noActiveCriteria as failure
src/benchmark/prompt-evaluator.ts	Adds noActiveCriteria to results; empty criteria now score 0 instead of 1.0
src/benchmark/prompt-criteria.ts	New helper to resolve criteria per task/capability (resolveCriteriaForTask)
package.json	Updates test script to include new smoke tests and remove deleted one
README.md	Documents capabilityId and prompt-surface coverage/verdict semantics
CHANGELOG.md	Records fixes for prompt-surface scoring/verdict, model-id rewrite exemption, and module deletion

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

resolveCriteriaForTask(task, promptCaps) relies on task.capabilityId, but benchmark tasks are loaded via loadTasks() (TaskDefinition) which currently discards unknown fields when normalizing JSON. That means even if tasks.json contains capabilityId, it will be dropped and prompt runs will consistently hit the "missing capabilityId" error path and score 0.

Fix: plumb capabilityId through the benchmark task model and loader (e.g., add optional capabilityId?: string to TaskDefinition in src/benchmark/types.ts and preserve it in normalizeTaskDefinition / JSON parsing), so the runner receives the field for prompt surface tasks.

[Copilot]

The comment says parseGeneratedTasks "can validate and attach capabilityId", but the current implementation only attaches capabilityId when present and leaves validation (membership in knownCapabilityKeys) to grounding. Consider either updating the comment to match reality or adding an explicit check that rawCapabilityId is in knownCapabilityKeys so bad values fail fast during parsing.

Suggested change

	// can validate and attach capabilityId to each task.
	// can attach capabilityId metadata; membership validation is handled later.

[damienen]

Code review

Found 1 issue:

1. capabilityId is written to tasks.generated.json by freeze.ts but silently dropped by normalizeTaskDefinition in config.ts — the function's input type and return value only include {id, prompt, expected_actions, verify, expected_fetches}, and TaskDefinition has no capabilityId field. When the runner calls resolveCriteriaForTask(task, promptCaps), task.capabilityId is always undefined, so the function throws on every prompt-surface task. TypeScript does not catch this because TaskDefinition is structurally assignable to GeneratedTask (capabilityId is optional on GeneratedTask). Every prompt-surface benchmark run will report 100% task failures with "missing capabilityId — Regenerate tasks", even on freshly generated tasks.

   Fix: add capabilityId?: string to TaskDefinition and update normalizeTaskDefinition to read and pass it through.

skill-optimizer/src/benchmark/config.ts

Lines 67 to 115 in 3a5ae71

	function normalizeTaskDefinition(
	task: { id?: unknown; prompt?: unknown; expected_actions?: unknown; verify?: unknown; expected_fetches?: unknown },
	resolvedPath: string,
	index: number,
	): TaskDefinition {
	if (typeof task.id !== 'string' || task.id.trim() === '') {
	throw new Error(`Tasks file ${resolvedPath}: task at index ${index} must include a non-empty string id`);
	}
	if (!isSafeTaskId(task.id)) {
	throw new Error(`Tasks file ${resolvedPath}: task id "${task.id}" must match ${SAFE_TASK_ID.toString()} and cannot be . or ..`);
	}
	if (typeof task.prompt !== 'string' || task.prompt.trim() === '') {
	throw new Error(`Tasks file ${resolvedPath}: task ${task.id} must include a non-empty string prompt`);
	}
	const rawExpectedActions = Array.isArray(task.expected_actions) ? task.expected_actions : null;
	if (!rawExpectedActions) {
	throw new Error(`Tasks file ${resolvedPath}: task at index ${index} must include an expected_actions array`);
	}
	const expected_actions = rawExpectedActions.map((rawAction, actionIndex) => normalizeExpectedAction(rawAction, resolvedPath, index, actionIndex));
	const rawVerify = Array.isArray(task.verify) ? task.verify : undefined;
	if (rawVerify !== undefined) {
	for (let i = 0; i < rawVerify.length; i++) {
	if (!rawVerify[i] || typeof rawVerify[i] !== 'object') {
	throw new Error(`Tasks file ${resolvedPath}: task ${task.id} verify[${i}] must be an object`);
	}
	}
	}
	const rawFetches = Array.isArray(task.expected_fetches) ? task.expected_fetches : undefined;
	if (rawFetches !== undefined) {
	for (let i = 0; i < rawFetches.length; i++) {
	if (typeof rawFetches[i] !== 'string' || !(rawFetches[i] as string).trim()) {
	throw new Error(`Tasks file ${resolvedPath}: task ${task.id} expected_fetches[${i}] must be a non-empty string`);
	}
	}
	}
	return {
	id: task.id,
	prompt: task.prompt,
	expected_actions,
	verify: rawVerify as TaskDefinition['verify'] | undefined,
	expected_fetches: rawFetches as string[] | undefined,
	};
	}

skill-optimizer/src/benchmark/types.ts

Lines 141 to 149 in 3a5ae71

	export interface TaskDefinition {
	id: string;
	prompt: string;
	expected_actions: ExpectedAction[];
	verify?: TaskVerification[];
	expected_fetches?: string[];
	}

skill-optimizer/src/benchmark/prompt-criteria.ts

Lines 24 to 31 in 3a5ae71

	caps: readonly PromptCapabilityWithSection[],
	): ResolvedPromptCriteria {
	if (!task.capabilityId) {
	throw new Error(
	`Task ${task.id}: prompt-surface task is missing capabilityId. ` +
	`Regenerate tasks with \`skill-optimizer generate-tasks\`.`,
	);
	}

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}