Khan Academy video recommendation system with knowledge-gain estimation

[jeremymanning]

Summary

Replace the current concept-based "Suggested Learning" modal (which links to Khan Academy search) with a video recommendation system that ranks actual Khan Academy videos by their estimated knowledge gain for each user. The system uses the existing GP knowledge estimator to predict how much a user would learn from each video, tracks actual learning via difference maps after watching, and improves recommendations over time.

Motivation

The current suggestion system (insights.js → showSuggestions()) identifies weak concepts and links to Khan Academy search pages. This has two limitations:

1. The search results may not match the concept well
2. There's no feedback loop — watching content doesn't improve future suggestions

A video recommendation system addresses both by (a) pre-mapping actual Khan Academy videos into the embedding space and (b) measuring observed knowledge gain after watching to refine future rankings.

Design Overview

Phase 0: Offline Data Pipeline (one-time pre-computation)

1. Scrape all Khan Academy YouTube videos (~9,000 videos via YouTube Data API v3)
2. Download transcripts via youtube-transcript-api (~85-95% coverage; mark unavailable videos)
3. Sliding window embeddings: Split each transcript into overlapping windows (512 words, stride 50 words), embed each with google/embeddinggemma-300m (768-dim) — same model as articles/questions
4. UMAP projection: Use the existing trained umap_reducer.pkl to transform() each window's embedding into the mapper's 2D [0,1] space
5. Export: Produce a static JSON file mapping each video to its metadata + array of 2D window coordinates

Phase 1: Initial Video Ranking (before any videos watched)

For each video v with N_v sliding windows at coordinates (x_w, y_w):

TLP(v) = (1/N_v) × Σ_w [ (1 - K(x_w, y_w)) × (1 - 0.5 × U(x_w, y_w)) ]

where K is the GP's predicted knowledge and U is uncertainty. This estimates "how much it would be theoretically possible to learn from this video, normalized for length."

• Present a clickable list of the top 10 videos (highest TLP = most learning potential)
• Clicking a video displays it inline via YouTube IFrame Player API
• Reaching the end of the video marks it as "watched"

Phase 2: Difference Maps (after watching first video + 5 questions)

After watching any suggested video and answering ≥5 new questions:

1. Compute difference map: D(x,y) = K_after(x,y) - K_before(x,y) over the 50×50 grid
2. Weight by relevance to the video's content (Matérn 3/2 RBF centered at each window coordinate)
3. Use the weighted difference map to estimate a transfer function — how much knowledge actually transfers from watching a video covering a given region

Phase 3: Expected Gain Scoring (using difference maps)

Once a difference map exists, rank new videos by expected gain:

ExpectedGain(v) = (1/N_v) × Σ_w [ (1 - K(x_w, y_w)) × TransferFactor(x_w, y_w) ]

where TransferFactor is derived from observed difference maps.

Phase 4: Recency-Weighted Running Average

After each subsequent video + 5 questions:

• Update a recency-weighted EMA of difference maps (α ≈ 0.9)
• Track relevance coverage (which regions of the space have been covered by watched videos)
• Factor in content coverage when estimating transfer at each coordinate

UI Requirements

• Video list modal: Ranked top 10, showing title, duration, estimated gain bar, watched indicator
• Inline video player: YouTube IFrame embed with native controls, supplemental speed buttons, themed container
• Gain indicator: "+N% estimated knowledge gain" with color-coded bar (green=high, yellow=moderate, gray=low)
• Domain scoping: Filter to 50×50 grid of active domain's bounding rectangle; "All (general)" uses full space
• Mobile: Bottom sheet list, full-screen player

New Files

• scripts/scrape_khan_videos.py — YouTube Data API scraper
• scripts/download_transcripts.py — Transcript downloader
• scripts/embed_video_windows.py — Sliding window embedding pipeline
• scripts/project_video_coords.py — UMAP transform + normalization
• data/videos/khan-academy-videos.json — Pre-computed video database
• src/learning/video-recommender.js — Knowledge gain estimation engine
• src/ui/video-player.js — Video list modal + inline YouTube player

Modified Files

• src/state/store.js — New state atoms ($watchedVideos, $differenceMap, etc.)
• src/state/persistence.js — Reset watched videos on full reset
• src/app.js — Wire video recommendation flow to suggest-btn

Technical Notes

• Khan Academy has ~9,000 YouTube videos; ~85-95% have transcripts
• Average video produces ~40 sliding windows → ~360K total windows to embed
• Embedding pipeline: ~1.7 hours on Apple Silicon MPS
• UMAP transform: ~10-15 min for 400K windows (uses existing reducer)
• Client-side video scoring: <1ms for 10K videos (well within 15ms budget)
• Video database JSON: ~4 MB (gzip ~1 MB), lazy-loaded on first suggest click
• All client-side computation; no server required after static asset build

Acceptance Criteria

• Pre-computed video database covers ≥7,500 Khan Academy videos with 2D window coordinates
• Top 10 video recommendations are ranked by estimated knowledge gain
• Clicking a video displays inline YouTube player with native + supplemental controls
• Video completion (reaching end) is detected and tracked persistently
• After watching a video + answering 5 questions, difference map is computed
• Subsequent recommendations use observed difference maps (not just theoretical gap)
• Recency-weighted running average improves with each video-watching cycle
• Each video in the list displays estimated gain indicator
• Domain dropdown scoping filters videos to active domain's region
• Watched videos persist across sessions (localStorage)
• Full reset clears watch history
• Mobile-responsive (bottom sheet list, full-screen player at ≤480px)
• Graceful fallback to current Khan Academy search links if video data unavailable

Labels

enhancement, P2