[RFC] Shared-course learning reports: lightweight registration and reusable observations

[cosarah]

Tracking issue: #1920

Summary

Add a teacher dashboard for a shared course, backed by reusable learning observations in open-source OpenMAIC. A student enters a name, the teacher confirms the registration, and the dashboard shows what that student has played, submitted, or started in each scene, together with foreground dwell time.

The first consumer is the teacher of one shared course. The observation contracts and aggregation rules should also support a future personal learning dashboard and hosted deployments, without making either a requirement for this release.

This is a design proposal, not an implementation commitment. It builds on the owner/identity work tracked in #1658 and the RuntimeStore foundation in #869. It complements the learning-event export discussion #1671: establish facts needed by this product feature, then allow optional export adapters to consume them. No xAPI exporter or consumer-specific integration is required here.

Problem or Motivation

After sharing a course, its creator cannot easily answer:

• Which registered students have listened to a scene's narration?
• Has a student actually operated an interactive activity?
• Has a student submitted the quiz, or started/completed the project?
• How much foreground time has the student spent in each scene, including return visits?

Opening a scene, finishing narration, and completing its activity are different facts. Treating them as one completion flag would hide useful information and imply more than the system can observe. Repeated visits also need explicit semantics: reviewing a scene should accumulate evidence without creating another student or duplicating a quiz submission.

Proposed Solution

1. Lightweight registration and identity

For a share with learning reporting enabled:

1. Show students what the teacher will be able to see and ask for a display name.
2. Bind the registration to the server-resolved learner identity and the share.
3. Let the share owner confirm the student's stated identity. Pending registrations remain separate from confirmed students and aggregate denominators.
4. Recognize returning learners through their identity, without repeating registration or losing earlier observations. Confirmation changes the registration state, not ownership of the observations.

A name is a label, not an authentication credential or deduplication key. Duplicate names are allowed. Changing a name does not create a new learner, and typing an existing name does not recover another learner's records.

Open-source deployments can use an independent anonymous browser identity; hosted deployments can resolve an existing account through an adapter. An anonymous learner who clears their identity cookie or switches devices cannot automatically be recognized as the same person. Automatic name-based merging is excluded. Cross-device continuity requires a stable identity supplied by the deployment; a recovery/merge workflow can be designed separately.

Deployments that resolve everyone to a single owner cannot distinguish students. The feature must make independent learner identity a prerequisite instead of silently showing misleading per-student reports. Teacher confirmation is lightweight roster bookkeeping, not proof of legal identity. Full account administration, strict rosters, organizations, and formal enrollment workflows are outside this proposal.

2. Two independent dimensions for every scene

All four scene types can contain narration. Report narration playback separately from scene-specific activity.

Common observations:

• Narration: not started / in progress / completed. A scene with no required narration is not applicable.
• Foreground scene dwell time: accumulated time while the scene is displayed and the document is visible. Count listening, reading, thinking, and operating; stop on scene changes, hidden documents, or session termination. This is observed foreground presence, not a measure of attention or mastery.

Scene type	Activity observation	Meaning
Slide	No additional activity metric	Narration and dwell time are sufficient.
Interactive	Has interacted: yes / no; unknown when unsupported	At least one real student operation on an activity control, such as a click, drag, or input. No interaction count.
Quiz	Submitted: yes / no; submission count; first/latest submission time	A successful persisted submission of an attempt. Opening, draft saving, failed requests, or network retries do not count.
PBL	Not started / in progress / completed; start/completion times	A persisted project launch and persisted project completion. Opening the introduction, playing narration, or viewing a report is not itself a lifecycle transition.

Interactive autoplay, AI demonstrations, mouse movement, and focus alone do not qualify as student interaction. Generic iframe instrumentation may not cover every activity: expose collection capability, and show unknown when the signal cannot be collected. A known interaction remains affirmative even if a later visit has incomplete instrumentation.

Narration completion means all required narration segments for that content version have actually played. A seek, maximum cursor position, or reaching an arbitrary percentage is insufficient. Playback coverage can accumulate across visits; replaying covered material does not increase completion. Skipping narration does not prevent a quiz submission or PBL completion from being recorded.

Existing quiz results can remain available as supporting detail. New grading analytics, mastery estimates, and inferred risk labels are not required.

3. Teacher dashboard and aggregation

The main view is a student × scene matrix, with independent narration/activity indicators and dwell time. For example:

Narration completed · Interacted · 3m 20s foreground time

Course summaries report separate measures: narration completion by scene, interactive participation, quiz submission, and PBL starts/completions. There is no single overall course completion percentage in this release: generic interactive activities do not have a universal completion criterion.

Per-scene rates use confirmed registrations in the selected share/content-version cohort as their denominator. Show pending registrations separately. Identify unknown collection states and exclude non-applicable narration from narration rates. Unknown is not a verified negative. Any time distribution must state its sample, such as students who visited the scene; it must not silently treat missing telemetry as zero. Counts refer to registered students, not an assumed class population.

The teacher can access only the observations associated with their share. A learner's earlier self-study, another teacher's share, and private conversations are not part of this report.

4. Returning to the same course

Aggregate by learner × share × immutable content version, then scene. Keep visits and activity attempts/runs distinct underneath that aggregate.

Action on a later visit	Expected result
Reopen the same share	Resume the existing learning context; keep a separate visit record.
Finish previously unplayed narration	Union actual playback coverage; completed remains completed.
Operate an interactive activity again	Has-interacted stays yes. No count is added.
Spend more foreground time on a scene	Add non-overlapping observed intervals.
Revisit a quiz draft or result	Resume it; create neither an attempt nor a submission merely by visiting.
Explicitly retry and submit a quiz	Create a new attempt and count its successful submission once.
Return to an unfinished PBL project	Resume the same project run.
Explicitly restart PBL after completion	Create a new run; preserve previous completion and show the current run separately, e.g. completed before / retry in progress.

Multiple tabs or devices must not inflate dwell time for the same learner/share/version/scene: union overlapping intervals rather than summing them. Transport retries must not duplicate any observation or submission. Visit counts may support diagnostics but are not a headline learning metric.

Continue-learning links must preserve share and version context. Distinct shares remain distinct even when they point to the same source course.

Content edits that change observable learning requirements create a new content version. Preserve the historical scene/assessment definition and denominators. Do not silently merge old coverage, submissions, or completion with incompatible new content. Pinning a share versus publishing a new version is an API/UX decision to settle during design; either must make the version explicit.

5. Decouple collection, facts, and reporting

Proposed flow:

Playback / scene visibility / interactive instrumentation
                                      \
                                       -> validated observations -> versioned projections -> authorized reports
                                      /
Persisted quiz / PBL runtime facts -- transactional delivery

The contracts should separate:

• Identity and access policy: resolve the learner, authorize the share context, and authorize the teacher report. Deployment adapters supply identity; browser payloads do not choose an arbitrary learner or teacher.
• Observation producers: playback, visibility, interactive controls, and persisted activity transitions emit semantic facts without depending on dashboard UI or database table layouts.
• Durable ingestion and storage: validate, deduplicate, and persist observations or references to authoritative facts.
• Projections: shared deterministic reducers compute coverage, dwell intervals, submissions, and project history. Projection versions permit rebuilding when aggregation rules change.
• Consumers: the teacher report first; a personal report or standard export can reuse the same facts with its own authorization and aggregation scope later.

A candidate envelope includes an observation ID, schema version, kind, server-bound learner/share/content-version/scene context, visit ID, optional attempt/run ID, occurrence and receipt times, source/provenance, and a minimal typed payload. Exact names and package boundaries remain open. Include collection capability/coverage so unsupported telemetry is distinguishable from an observed negative.

Prefer a small typed event registry with per-kind validation and reducers over arbitrary unvalidated payloads. A new scene activity should be able to add an observation and projection without changing the identity system or making existing producers depend on its UI.

6. Collection and storage approach

Start with the existing PostgreSQL/storage foundation. No additional queue service, LRS, or analytics database is required.

Browser observations: batch bounded visibility/playback intervals and interaction facts, retry with stable IDs, and flush opportunistically on lifecycle changes. Use periodic checkpoints so an abrupt exit does not depend on a final unload request. Specify checkpoint cadence, clock normalization, overlap rules, and maximum lost-tail bounds in implementation design. Reject impossible intervals; retain enough provenance to distinguish client observations from authoritative server facts. Foreground telemetry is useful evidence, not a tamper-proof attendance system.

Business facts: quiz submissions and PBL transitions remain authoritative in their existing runtime models. Capture their delivery through a transactionally coupled outbox or equivalent durable storage hook. Do not rely on a second browser request after a successful save. The observation layer stores the minimal statistical fact and source reference, not another copy of answer text, chat, or project content.

Idempotency: enforce uniqueness of observation IDs within the appropriate scope and reject conflicting reuse. Runtime sequence numbers are per session; existing record IDs and optimistic concurrency alone are not a complete ingestion idempotency contract. Replaying an outbox entry or rebuilding a projection must reproduce the same result.

Processing: use pending work with leases/retries rather than assuming client time, session update time, or a highest allocated sequence is a safe global cursor. Concurrent appends must not lose a dirty projection marker, and expired workers must not overwrite newer projections. Reports should expose freshness when updates are asynchronous.

Content definitions: store a minimal immutable definition of the scenes, narration requirements, and assessment identity needed to interpret each version. A freshness counter by itself is not a historical content snapshot. Create this definition through an authorized, coherent read.

Scope and lifecycle: bind runtime activity sessions to their learning context. All read/resume/mutation entry points must enforce that binding, including older routes without an explicit share parameter. Identity claims, ownership changes, and deletion must handle registrations, observations, outbox entries, and projections together. Rebuilding a projection must not resurrect deleted learner data.

These storage capabilities need explicit design and migration work; the proposal does not assume they all exist today.

7. Open-source and hosted reuse

Keep observation schemas, producer hooks, reducers, and persistence contracts in the shared OpenMAIC implementation. Keep account resolution and deployment-specific authorization/configuration behind narrow adapters. Open-source anonymous identity and hosted account identity should pass the same behavioral contract tests.

The initial UI belongs in open source as well as hosted deployments. Avoid maintaining two definitions of completed narration, a quiz submission, or scene dwell time. Adapter and schema compatibility should be versioned so downstream deployments can adopt the feature without rewriting instrumentation.

Acceptance Criteria

• A student can enter a name, a teacher can confirm the registration, and returning through the same identity/share preserves the registration and learning context. Same-name identities remain separate.
• Independent learner identity is required; a shared-owner deployment does not report separate students incorrectly.
• All four scene types show independent narration status and foreground dwell time, with not-applicable and unavailable states where appropriate.
• Actual narration coverage accumulates across visits; seeking to the end does not mark playback completed.
• Interactive activity reports only whether a real student interaction occurred, with unknown for unsupported observation.
• A quiz submission is counted once after persistence; reopening and request retries do not add submissions, while an explicitly submitted new attempt does.
• PBL starts/completions follow persisted transitions; resume keeps the run, restart creates a new run, and earlier completion remains visible.
• Overlapping foreground intervals, duplicate delivery, reordered delivery, worker retries, and interrupted visits have documented, tested aggregation behavior.
• The teacher matrix and summary rates use explicit share/version cohorts and confirmed registrations; pending and unknown states remain visible. No overall course completion percentage is shown.
• Access tests cover teacher ownership, another share, unbound/legacy runtime access paths, and student-to-student isolation. Registration does not expose private learning history or conversations.
• A content change preserves historical interpretation; an identity change or deletion propagates through stored facts and derived reports.
• Durable capture survives a successful business write followed by process interruption, and projection rebuilds reproduce the same metrics.
• The same observation fixtures and reducers work with anonymous and account-backed identity adapters, without requiring a personal dashboard or an export service.

Implementation Sequence

1. Settle semantic contracts, version/cohort rules, identity requirements, and representative repeated-visit fixtures.
2. Add scoped registration, content definitions, durable capture, storage lifecycle support, and deterministic projections.
3. Wire narration and foreground observations for every scene, interactive participation, and quiz/PBL persisted transitions.
4. Add the teacher matrix, aggregate summaries, collection/freshness indicators, and authorization tests.

Alternatives Considered

• Require a complete teacher/student account system: too much infrastructure for lightweight course sharing; retain independent identity and confirmation without embedding institutional management.
• Build this only in a hosted application: duplicates learning semantics and prevents open-source deployments from using the same capability.
• Calculate everything from current runtime state: insufficient for playback coverage, foreground intervals, retries, historical versions, and deleted/reset attempts.
• Send all facts from the browser: cannot reliably prove that a quiz/PBL write committed, and creates a failure window between the business write and observation delivery.
• Adopt xAPI or an LRS as the internal storage model now: optional interoperability can project shared facts later; it should not determine the initial dashboard implementation.
• Show a single completion percentage: combines incompatible notions of narration, participation, submission, and project completion.

Area

Storage / persistence; Playback / presentation; Quiz / Assessment; Interactive simulations; UI / UX.

Additional Context and Open Questions

Related: #23 (broader multi-user/admin request), #1658 (identity), #869 (RuntimeStore), #367 and Discussion #1671 (learning-event interoperability).

Questions to resolve before implementation:

1. Where should observation contracts and reducers live, and what is the smallest durable RuntimeStore extension needed?
2. Should each share pin one version, or allow explicitly published versions with separate registration cohorts?
3. What narration segment definition and checkpoint cadence give reliable coverage and a clear dwell-time accuracy bound?
4. Which interactive controls can the generic bridge observe reliably, and how should activity authors declare additional capability?
5. What retention default and learner-visible collection notice should ship in open source?

Personal dashboards, cross-course active learning time/learning days, strict roster management, automatic identity merging, mastery/risk inference, and standard export implementation are deferred.

[IlyaKhalafi]

For the smallest RuntimeStore extension, there is already a useful transaction boundary. appendRecord() accepts sessionTransition, and the PostgreSQL implementation commits the record and status change in one transaction. The interface offers no way to add other writes to that transaction, so an outbox row would need to be part of this operation rather than written after appendRecord() resolves.

One specific case deserves an explicit receipt contract. A completion commits, its response is lost, then the caller retries the same operation. The PostgreSQL implementation rejects an inactive session before inserting a record, and a stale expectedLastSeq is a conflict rather than a replay receipt. The HTTP contract explicitly excludes implicit retry safety.

For the proposed observation operation, I would persist an operation ID scoped to the authenticated learner and session, a payload digest and the original result in the same transaction as the business fact and outbox row. Check that receipt after authorization but before the active-status and expectedLastSeq gates, so an identical replay returns the original result even after the attempt has completed. Conflicting reuse should fail, while an intentional new attempt gets a new operation ID.

A small contract fixture would make this concrete across adapters. Commit completion, drop the response, retry against the completed attempt, then verify one fact and one outbox entry. Also retry with different content under the same ID and verify rejection. That tests the ambiguous-ack boundary beyond ordinary duplicate event delivery.