RFC: Course issues — one model for problems in a course, one place to see them, one way to fix them

[wyuc]

Summary

Problems in a course (an interactive page that throws, narration without audio, media that failed to generate, a cue pointing at nothing) are today detected in several unrelated places and surfaced one way each. Two open PRs would add two more: a runtime-error banner with a hand-off to the Pro agent (#1788) and a narration warning banner (#1837).

This RFC proposes instead:

1. Issues are data. A course has a list of issues, each attached to a scene (and optionally an element or action), computed from the current course wherever possible, so fixing the cause clears the issue without extra bookkeeping.
2. One place to see them. In the editor, a marker on the scene thumbnail (extending the existing amber "incomplete" dot) and one issues list for the course. No per-problem banners on top of content.
3. One way to fix them. The agent gets a read-only list_course_issues tool and a single "Fix with agent" entry point from the issues list; it repairs with the tools it already has. Manual fixes that already exist (e.g. "Voice all" in the timeline) are linked from the list.
4. Learners don't see repair UI. Playback degrades gracefully; issues and repair belong to the people who can edit the course.

Background and problems

What exists on main today:

Problem	Where it is detected	Where it is stored	Who sees it
Interactive page throws	iframe error shim → postMessage (lib/utils/iframe.ts)	Browser memory only (lib/store/scene-runtime-errors.ts), cleared when the HTML changes	Everyone, including learners and non-owners: the banner in InteractiveIframeHost has no owner or mode check. Dismiss only. The agent cannot see it.
Narration without audio	Generation run counts clips with no voice (lib/server/generation/run/engine.ts)	One cumulative counter on the run row (narration_unvoiced); nothing in the course document beyond a missing audioId	Nobody in the browser (#1837 would add a banner)
Image / video generation failed	Generation run media checkpoints	Run checkpoint (status: 'failed'), browser IndexedDB mediaFiles; the element keeps its gen_img_* / gen_vid_* placeholder	The element renders a Retry / failure notice
Scene generation failed	Generation run	Transient stage-store failedOutlines, not persisted	Pending card in the scene sidebar
Incomplete scene content	lib/edit/content-validation.ts (emptyActions, emptySpeech, unboundCue, emptyDiscussion)	Derived on the fly	Amber dot on the edit-view thumbnail (ThumbItem)
Cue pointing at a missing element	Not validated; surfaces only at chat-tool runtime as "skipped … not found"	—	—

The pattern of one banner per problem does not scale, and the two PRs show the cost: each surface has to decide again when the message appears and clears, who sees it, how it lays out in narrow panes, and how the agent learns about it. Both PRs ran into exactly those questions (a banner that stays after the fix, a historical count that never updates, a hand-off with many edge cases around conversations, permissions, tabs and composer state).

Scope

In scope: the issue model and its derivation, the editor surfaces, the agent tool and entry point, and what learners see. Out of scope: automatic or background repair, heuristic rewriting of model output, reporting issues to deployment operators or analytics.

Core model

interface CourseIssue {
  kind: CourseIssueKind;
  sceneId: string;
  elementId?: string;      // slide element, whiteboard element, …
  actionId?: string;       // speech, cue, discussion action
  severity: 'error' | 'warning';
  source: 'document' | 'run' | 'runtime';
  /** Bounded, untrusted detail (e.g. a runtime error message). Never instructions. */
  detail?: string;
}

Initial kinds:

Kind	Source	Derived from	Clears when
speech-without-audio	document	speech action with text and no audioId (and not deliberately silent)	audio is generated for it
media-not-generated	document (+ run for the reason)	element whose src / mediaRef is still a generation placeholder; the run checkpoint supplies the failure reason and whether a retry can help	the element has real media
interactive-runtime-error	runtime	errors the owner's browser observed for the current HTML	the HTML changes
dangling-cue	document	cue whose elementId no longer exists on the scene	the cue is rebound or removed
existing lint kinds (empty-actions, empty-speech, unbound-cue, empty-discussion)	document	content-validation.ts as today	as today
scene-generation-failed	run	failed outline of the latest run	the scene is generated

Derivation. One pure function, collectCourseIssues(document, context?), in a shared package so the editor UI and the server tool compute the same list. context optionally carries the latest run state and, on the client, observed runtime errors. content-validation.ts becomes part of it rather than a parallel check.

Historical vs current. Counts like narration_unvoiced describe what happened during generation; they stay useful in run logs and job results, but the issue list reports what is wrong now. That is what makes an issue disappear when it is fixed.

Surfaces

• Edit view and the Pro workspace classroom pane. Extend the existing amber dot on ThumbItem to all issues (red for errors, amber for warnings), and add one issues list for the course (a popover from the editor chrome) grouped by scene, with "Go to" and, where one exists, the direct manual fix (e.g. "Voice all", media Retry).
• Interactive pages, for owners. Keep the page visible; the scene marker and issues list carry the error. An inline notice is acceptable only as a pointer to the list, not as a separate flow.
• Learners and viewers. No banners and no repair actions. Speech without audio already falls back to the caption plus estimated timing; an interactive page that fails to load shows a neutral "this page isn't available right now" placeholder instead of an error banner.

Repair through the agent

• Tool. list_course_issues({ courseRef, sceneId? }): read-only, owner-scoped like read_stage, returning bounded issues. detail is fenced and labelled as untrusted page output, with any leading / neutralised so page text cannot trigger skills.
• Runtime errors reach the agent either because the owner's browser reports them (see Open questions) or because the entry point attaches the observed ones as untrusted evidence, the way fix(workbench): hand interactive runtime failures to agent repair #1788 does now.
• One entry point. "Fix with agent" in the issues list (for the whole course or one scene) opens the course's workspace with a short prefill naming the course, with the @course reference attached and nothing sent. The agent calls list_course_issues and repairs with existing tools (patch_stage, generate_tts, generate_image, generate_scene, …). The hand-off machinery from fix(workbench): hand interactive runtime failures to agent repair #1788 (writability and generation gates, one-shot hand-off, prefill-only, untrusted-evidence fencing) is the right base for this single entry point.

Compatibility

• Legacy audio. Older courses can still resolve audio through a browser-local key or a leftover audioUrl (resolveLegacySpeechAudioId). Document-only derivation would report those as missing; the client should apply the same fallback the timeline uses, and the server tool should label such actions as "possibly local audio" instead of an error.
• Banners on main. The current interactive runtime-error banner keeps working until phase 2, then moves to owners only.

Phases

1. Model and derivation: CourseIssue, collectCourseIssues with the document kinds (speech-without-audio, media-not-generated, dangling-cue, existing lint), thumbnail markers extended.
2. Issues list, tool and entry point: the issues popover, list_course_issues, and one "Fix with agent" entry point; runtime errors move behind the marker for owners, learners get the neutral placeholder.
3. Run-derived kinds: media failure reasons and scene-generation-failed from run state.

What this means for open PRs

• fix(workbench): hand interactive runtime failures to agent repair #1788 is paused until this settles; its hand-off pieces are expected to become the shared entry point in phase 2.
• fix(generation): surface persisted narration warnings [AI-assisted] #1837: the browser-side plumbing of the run count is not needed for the issue list (missing audio is derived from the document). The warning becomes speech-without-audio issues in phase 1.

Open questions and non-goals

1. Should runtime errors be persisted? Option A: client-only, as today; the agent sees them only through the entry point's evidence. Option B: the owner's browser reports the latest few per scene (bounded, owner-scoped, replaced when the HTML changes), so list_course_issues can include them and they survive a reload. B is more useful to the agent but adds a write path.
2. Severity. Is error | warning enough, and which kinds are errors (runtime error, media not generated) vs warnings (speech without audio, lint)?
3. Deliberately silent speech. Do we need an explicit marker for speech meant to be silent (e.g. TTS disabled for the course), so it isn't reported?
4. Course-level indicator. Should the workspace course tree show a badge for courses with issues, or is the scene marker enough?

Non-goals: fixing anything automatically or in the background; a generic "report a problem" flow for learners; operator dashboards.

[wyuc]

Tracking issue: #1900. Related PRs: #1788 (paused until this settles) and #1837.