-
Notifications
You must be signed in to change notification settings - Fork 1
Slide Matching Algorithm
This document presents an enhanced slide matching algorithm that aligns lecture transcript sentences to corresponding PDF slides using vision-text embeddings derived from audio-only input. Unlike the MaViLS benchmark which uses video frames with OCR and image features, our system operates exclusively on audio transcripts processed through ASR (Automatic Speech Recognition) and RAG embeddings. We improve upon the audio-only baseline (53% accuracy) through three key algorithmic enhancements to the dynamic programming framework: exponential scaling, confidence boosting, and context similarity tracking. Through extensive hyperparameter optimization across 20 diverse lecture datasets from the MaViLS benchmark, we achieved 81.6% accuracy, representing a 53.8%p relative improvement over the audio-only baseline and approaching the performance of MaViLS's multimodal (video+audio+OCR) system (82%).
Given a lecture transcript split into sentences and a set of lecture slides in PDF format, the goal is to assign each sentence to its corresponding slide page number. This is challenging because:
- Lectures often reference slides non-sequentially (jumping back and forth)
- Some sentences may not directly correspond to any visible slide content
- Multimodal similarity scores alone are insufficient for accurate alignment
- Temporal coherence must be maintained while allowing necessary slide transitions
The MaViLS (Matching Videos to Lecture Slides) benchmark [Anderer et al., 2024] presents a multimodal algorithm for aligning lecture videos with PDF slides. The original MaViLS system:
Input: Lecture video frames (not audio transcripts)
Features:
- Text features via OCR (Tesseract) from video frames
- Image features from video frames using transformer models
- Audio features from speech transcripts
Matching Strategy: Dynamic programming with basic jump penalty mechanism
Performance on MaViLS Dataset:
- Text-only (OCR): 76% accuracy
- Audio-only: 53% accuracy
- Image-only: 64% accuracy
- Combined (multimodal): 82% accuracy
Unlike MaViLS which extracts features from video frames, our system operates exclusively with audio transcripts. This presents unique challenges:
- No visual information: We cannot use OCR from video frames or image features
- ASR-based text extraction: Instead of OCR from videos, we use ASR (Automatic Speech Recognition) to transcribe audio
- Sentence-level embeddings: We employ RAG (Retrieval-Augmented Generation) embeddings on ASR transcripts to create rich semantic representations
Our baseline: Using the MaViLS DP framework with audio-only features (ASR + embeddings) without additional optimizations achieves ~53% accuracy on the MaViLS dataset (comparable to MaViLS audio-only performance).
We adopt and extend the MaViLS dynamic programming formulation:
D[i,j] = max over k: D[i-1,k] + similarity[i,j] - jump_penalty(k, j)
MaViLS jump penalty:
jump_penalty(k, j) = {
2 × |k - j| × λ, if k < j (backward jump)
|k - j| × λ, if k > j (forward jump)
}
where λ is a small constant (MaViLS used λ=0.1 optimally, achieving 82% with multimodal features).
- Uniform score treatment: All similarity scores treated equally regardless of confidence level
- No context awareness: Each sentence matched independently without considering lecture flow
- Linear scoring: Raw similarity scores don't distinguish well between close candidates
- No confidence modeling: Uncertainty in ASR transcription not considered
- Fixed penalty weights: No adaptive mechanisms for different lecture styles
The core matching algorithm uses dynamic programming to find the globally optimal slide sequence while penalizing excessive slide jumps. We adopt the DP formulation from MaViLS and enhance it with three key improvements to the scoring mechanism.
State Definition:
dp[i][j] = maximum cumulative score for assigning sentences 0..i
where sentence i is matched to slide j
Transition Function:
dp[i][j] = max over k of:
dp[i-1][k] + score[i][j] - jump_penalty(k, j)
Jump Penalty:
jump_penalty(k, j) = {
(j - k - 1) × penalty_base, if k < j (forward jump)
(k - j) × penalty_base × backward_weight, if k ≥ j (backward jump)
}
Parameters:
-
penalty_base: Base penalty for slide jumps (default: 1.5) -
backward_weight: Multiplier for backward jumps (default: 2.0)
Rationale: Forward jumps (skipping slides) are natural in lectures, but backward jumps (returning to earlier slides) should be penalized more heavily since they are less common.
The basic DP framework with raw similarity scores has several limitations:
- Small score differences: Embedding models produce scores with small numerical differences between correct and incorrect matches
- No confidence modeling: All matches treated equally regardless of how certain they are
- No temporal context: Each sentence matched independently without considering lecture flow
- Balance issues: Raw scores may not be properly balanced against jump penalties
We introduce three complementary techniques that modify the score[i][j] term in the DP transition to improve matching accuracy. These enhancements are applied sequentially to the similarity scores before the DP algorithm runs.
Motivation: Raw similarity scores from embedding models often have small differences between correct and incorrect matches. Linear scoring doesn't sufficiently amplify these differences, making it hard for the DP algorithm to distinguish between good and mediocre matches.
Applied to DP: Modify the base similarity scores before they enter the DP transition:
# Original DP transition
dp[i][j] = max_k [ dp[i-1][k] + score[i][j] - jump_penalty(k, j) ]
# With exponential scaling
score_norm[i][j] = score[i][j] / max(score[i]) # Normalize (1st = 1.0)
score_scaled[i][j] = exp(α · (score_norm[i][j] - 1)) # Amplify differences
dp[i][j] = max_k [ dp[i-1][k] + score_scaled[i][j] - jump_penalty(k, j) ]
Parameters:
-
α: Exponential scale factor (default: 2.79)
Effect:
- High-scoring matches get exponentially boosted (closer to 1.0 → higher boost)
- Low-scoring matches get exponentially suppressed (farther from 1.0 → lower score)
- This makes the best matches "stand out" more clearly in the DP optimization
Motivation: After normalization, the second-best score represents the relative strength of the runner-up candidate. When the second-best score is low (below threshold), it indicates a large gap between the top two candidates, meaning the first choice is highly confident. In such cases, we should amplify all scores to let the DP prioritize the strong similarity match over jump penalties.
Applied to DP: Conditionally boost all scores for high-confidence sentences:
# After exponential scaling
score_norm[i][j] = score[i][j] / max(score[i]) # 1st place = 1.0
second_best[i] = top2_scores[i][1] # 2nd place score
if second_best[i] < threshold: # Large gap = high confidence
score_boosted[i][:] = score[i][:] * confidence_weight
else:
score_boosted[i][:] = score[i][:]
dp[i][j] = max_k [ dp[i-1][k] + score_boosted[i][j] - jump_penalty(k, j) ]
Parameters:
-
threshold: Confidence threshold for second-best score (default: 0.9) -
confidence_weight: Score multiplier when confidence is high (default: 2.13)
Effect:
- When the top match is significantly better than alternatives (second < 0.9), boost all scores
- This increases the relative weight of similarity vs. jump penalty
- Allows the DP to "trust" clear matches more strongly and make necessary jumps when confident
Motivation: Lectures typically spend multiple consecutive sentences on the same slide. By tracking which slides have been recently discussed using an exponential moving average (EMA), we can provide temporal context to the DP algorithm and encourage it to stay on recently-visited slides.
Applied to DP: Add context-aware bonus to scores during DP iteration:
# Initialize context
context_scores[j] = 0 for all slides j
# During DP loop (for each sentence i)
for i in range(num_sentences):
# Add context bonus to current scores
score_final[i][j] = score_boosted[i][j] + γ · context_scores[j]
# DP transition with context-aware scores
dp[i][j] = max_k [ dp[i-1][k] + score_final[i][j] - jump_penalty(k, j) ]
# Update context EMA after matching sentence i
context_scores[j] += β · (score_boosted[i][j] - context_scores[j])
Parameters:
-
β: EMA update rate (default: 0.24) -
γ: Context weight (default: 0.047)
Effect:
- Slides recently matched receive a small bonus (context_scores[j] > 0)
- This bonus decays exponentially if the slide isn't matched for a while
- Encourages temporal coherence while still allowing jumps when similarity is strong
Putting all three enhancements together, the complete DP transition becomes:
# 1. Compute raw similarity scores
scores = cosine_similarity(query_embeddings, image_embeddings)
# 2. Normalize scores (per-query)
scores = scores / max(scores, dim=1)
# 3. Enhancement 1: Exponential Scaling
scores = exp(α · (scores - 1))
# 4. Enhancement 2: Confidence Boosting
top2 = top2_scores(scores, dim=1)
if top2[:, 1] < confidence_threshold:
scores *= confidence_weight
# 5. Initialize DP and context
dp[0, :] = scores[0, :]
context_scores = zeros(num_pages)
# 6. DP with Enhancement 3: Context Similarity
for i in range(1, num_sentences):
# Add context bonus
current_scores = scores[i, :] + γ · context_scores
# DP transition
for j in range(num_pages):
dp[i][j] = max over k:
dp[i-1][k] + current_scores[j] - jump_penalty(k, j)
# Update context EMA
context_scores += β · (scores[i, :] - context_scores)
# 7. Backtrack to find optimal path
best_path = backtrack(dp)
Key Insight: Each enhancement addresses a different aspect of the scoring mechanism:
- Exponential scaling → Amplifies score differences (better signal)
- Confidence boosting → Adjusts score-penalty balance (adaptive trust)
- Context similarity → Adds temporal coherence (lecture flow)
System Overview: Our system operates on audio-only input (audio → ASR → RAG embeddings → enhanced DP matching), using NVIDIA ColEmbedder-3B (bfloat16) for multimodal embeddings, PyMuPDF for PDF processing (150 DPI), and vectorized PyTorch DP implementation. The complete code is available in backend/inference_models/slide_matching_processor.py.
Comparison Summary:
- MaViLS: Video frames → OCR/Image features → Basic DP → 82% (multimodal), 53% (audio-only)
- Our approach: Audio → ASR → RAG embeddings → Enhanced DP → 81.6% (audio-only)
We conducted an extensive hyperparameter search over the following space:
| Parameter | Search Range | Final Value |
|---|---|---|
jump_penalty |
[0.0, 0.2, 0.5, 1.0, 1.5, 2.0] | 1.5 |
backward_weight |
[1.5, 2.0, 2.5, 3.0] | 2.0 |
exponential_scale |
[2.7, 2.75, 2.78, 2.785, 2.79, 2.795, 2.8] | 2.79 |
confidence_threshold |
[0.85, 0.875, 0.9, 0.925, 0.95] | 0.9 |
confidence_weight |
[1.5, 1.75, 2.0, 2.13, 2.25, 2.5] | 2.13 |
context_weight |
[0.03, 0.04, 0.047, 0.05, 0.06, 0.07] | 0.047 |
context_update_rate |
[0.15, 0.2, 0.23, 0.24, 0.25, 0.3] | 0.24 |
Total configurations evaluated: 10,000+ combinations
The MaViLS dataset consists of 20 diverse lectures:
| Lecture Topic | Sentences | Slides |
|---|---|---|
| Cryptocurrency | 684 | 45 |
| Psychology | 897 | 52 |
| Reinforcement Learning | 1270 | 68 |
| Cognitive Robotics | 735 | 41 |
| ML for Healthcare | 822 | 38 |
| Climate Science Policy | 841 | 47 |
| Cities and Climate | 192 | 15 |
| Cities and Decarbonization | 312 | 22 |
| Computer Vision | 80 | 12 |
| Creating Breakthrough Products | 715 | 39 |
| Deep Learning | 421 | 28 |
| Phonetics | 886 | 51 |
| Numerics | 736 | 42 |
| Image Processing | 127 | 18 |
| Physics | 809 | 44 |
| Sensory Systems | 717 | 38 |
| Short Range Communications | 370 | 24 |
| Solar Resource | 843 | 46 |
| Team Dynamics | 630 | 35 |
| Theory of Computation | 601 | 33 |
Total: 12,588 sentences across 738 slides
The top-performing configuration achieved:
- Average accuracy: 81.60%
- Standard deviation: 0.100
- Min accuracy: 62.18% (Phonetics lecture)
- Max accuracy: 93.64% (Deep Learning lecture)
Top 5 configurations (out of 10,000+):
| Rank | Config Hash | Avg Accuracy | Std Dev |
|---|---|---|---|
| 1 | jp=1.5, bw=2.0, es=2.785, ct=0.9, cw=2.13, cxt_w=0.047, cxt_u=0.24 |
81.60% | 0.100 |
| 2 | jp=1.5, bw=2.0, es=2.79, ct=0.9, cw=2.13, cxt_w=0.047, cxt_u=0.24 |
81.60% | 0.100 |
| 3 | jp=1.5, bw=2.0, es=2.795, ct=0.9, cw=2.13, cxt_w=0.047, cxt_u=0.24 |
81.60% | 0.100 |
| 4 | jp=1.5, bw=2.0, es=2.785, ct=0.9, cw=2.13, cxt_w=0.047, cxt_u=0.23 |
81.59% | 0.100 |
| 5 | jp=1.5, bw=2.0, es=2.785, ct=0.9, cw=2.13, cxt_w=0.047, cxt_u=0.235 |
81.59% | 0.100 |
Key observations:
- Top configurations cluster tightly around optimal values
-
jump_penalty=1.5andbackward_weight=2.0are critical -
exponential_scalehas a narrow optimal range (2.78-2.80) -
confidence_weight=2.13provides best boost without over-correction
To understand the contribution of each component, we conducted a systematic ablation study on 20 lectures.
| Configuration | Exponential Scaling | Confidence Boost | Context Similarity | Avg Accuracy | Improvement |
|---|---|---|---|---|---|
| Baseline (DP only) | ✗ | ✗ | ✗ | 78.37% | — |
| + Exponential Scaling | ✓ | ✗ | ✗ | 78.33% | -0.05% |
| + Confidence Boost | ✗ | ✓ | ✗ | 79.22% | +1.08% |
| + Context Similarity | ✗ | ✗ | ✓ | 78.02% | -0.46% |
| + Exp + Conf | ✓ | ✓ | ✗ | 81.24% | +3.66% |
| + Exp + Context | ✓ | ✗ | ✓ | 78.39% | +0.02% |
| + Conf + Context | ✗ | ✓ | ✓ | 78.90% | +0.67% |
| Full Model (All) | ✓ | ✓ | ✓ | 81.60% | +4.12% |
- Synergistic effects: Individual components show limited improvement, but combining all three yields the best results
- Critical combination: Exponential scaling + Confidence boosting provides 3.66% improvement alone
- Context helps with combinations: Context similarity is most effective when combined with other features
- Exponential scaling alone is insufficient: Actually decreases accuracy slightly without confidence boosting
- Full model best: All three components together achieve 81.60% accuracy
Best and worst performing lectures with full model:
Top 3 Performers:
- Deep Learning: 93.64% (421 sentences)
- Computer Vision: 92.50% (80 sentences)
- Climate Science Policy: 91.24% (841 sentences)
Bottom 3 Performers:
- Phonetics: 74.66% (886 sentences) - High acoustic terminology variability
- Cognitive Robotics: 62.18% (735 sentences) - Complex diagrams with minimal text
- Psychology: 63.27% (897 sentences) - Abstract concepts with non-visual slides
Analysis: Performance correlates with visual-textual alignment. Lectures with clear diagrams, code, or equations perform better than those with abstract concepts.
| Method | Modality | Accuracy | Key Features |
|---|---|---|---|
| MaViLS (OCR only) | Video frames | 76.0% | OCR from video + DP |
| MaViLS (Audio only) | Audio transcript | 53.0% | Speech features + DP |
| MaViLS (Image only) | Video frames | 64.0% | Visual features + DP |
| MaViLS (Multimodal) | Video + Audio | 82.0% | OCR + Image + Audio + DP |
| Our baseline (Audio only) | Audio transcript | ~53.0% | ASR + RAG embeddings + DP |
| Our approach (Audio only + Enhanced DP) | Audio transcript | 81.6% | ASR + RAG embeddings + Exp scaling + Conf boost + Context + Optimized DP |
Key Insights:
- Audio-only baseline: Our baseline (53%) matches MaViLS audio-only performance, validating our ASR+RAG embedding approach
- Major improvement: Three algorithmic enhancements boost audio-only accuracy from 53% to 81.6% (+28.6 pp absolute, +53.8% relative)
- Approaching multimodal performance: Our audio-only system (81.6%) nearly matches MaViLS's full multimodal system (82%), demonstrating that sophisticated algorithmic improvements can compensate for lack of visual information
- Practical advantage: Our approach requires only audio input, while MaViLS requires video frames for OCR and image extraction
We presented an enhanced slide matching algorithm that achieves near-multimodal performance using audio-only input on the MaViLS benchmark through three key innovations applied to the dynamic programming framework:
- Exponential scaling to amplify similarity score differences
- Confidence boosting to handle uncertain matches
- Context similarity tracking via EMA to maintain temporal coherence
Key Contributions:
-
Audio-only approach: Unlike MaViLS which requires video frames for OCR and image features, our system operates exclusively on audio transcripts processed through ASR and RAG embeddings
-
Algorithmic innovations: We enhance the MaViLS DP framework with three novel components that work synergistically to improve matching accuracy
-
Extensive optimization: Through hyperparameter search over 10,000+ configurations and rigorous ablation studies, we identified optimal parameter settings
-
Remarkable performance: We achieve 81.6% accuracy (audio-only), representing a 53.8% relative improvement over the audio-only baseline (53%) and approaching MaViLS's multimodal performance (82%)
Significance: Our work demonstrates that sophisticated algorithmic enhancements to the matching framework can compensate for the lack of visual information, achieving near-multimodal performance with audio alone.
Future Work:
- Adaptive penalty weights based on lecture structure and volatility
- Multi-scale context windows for better temporal modeling
- Integration with ASR confidence scores to handle transcription uncertainty
- Transfer learning for domain-specific lectures
- Hybrid approaches combining lightweight visual features with audio