Review In Context Imitation

In-Depth Survey — In-Context Imitation & Demo-Following for Robot Policies

Question: how does a policy watch a demonstration and reproduce it at test time — ideally without per-task fine-tuning — by holding that demo (and its own history) in memory/context? Framing: in-context imitation is fundamentally a memory-conditioning problem. The demo is external context the policy attends to; long-horizon history is self-context. The controlled benchmark for both is RoboMME (its Imitation suite = replicate a demonstrated motion strategy, i.e. procedural memory). Companion: VLA Memory · RoboTTT.

---

1. The idea — demo as context, not as new weights

Classic imitation learning trains on demonstrations. In-context imitation instead conditions on a demo held in the model's context/memory and produces actions that copy its strategy — the way in-context learning works for LLMs. Two properties define the family:

• Test-time skill acquisition — a new task is specified by showing one (or a few) demos, no gradient step per task.
• Memory is the mechanism — the demo must be encoded and retrieved during rollout, so every method is really a choice of how to store and attend to context.

This is exactly the faculty RoboMME's Imitation suite isolates (MoveCube / InsertPeg / PatternLock / RouteStick — pick-place vs push vs hook, linear vs circular), scored as procedural memory.

---

2. Taxonomy — how the demo is conditioned

A. Cross-attention on the demo (video/demo-conditioned policies)

Encode the demo video and cross-attend from the current robot state to it.

• Vid2Robot (2403.12943) — prompt-video encoder + robot-state encoder joined by cross-attention transformers; imitates a human video without per-task fine-tuning, +20% over BC-Z, with cross-object motion transfer. The canonical demo-conditioned policy.
• VLBiMan — vision-language-anchored one-shot demonstration → generalizable bimanual manipulation.
• See Once, Then Act (2512.07582) — VLA that learns a task from a single video demonstration at test time.

B. Recurrent / query memory (history-aware VLAs)

Compress rollout (and demo) history into a recurrent latent or a set of learnable queries.

• HAMLET — "switch your VLA into a history-aware policy."
• MemoryVLA — perceptual-cognitive memory in a VLA.
• RememVLA (2026) — memory via dual-level recurrent queries.
• ContextVLA (2510.04246) — amortized multi-frame context.

C. Fast-weight / Test-Time-Training (compress the demo into weights)

Turn context into a parametric recurrent state updated by gradient descent at inference.

• RoboTTT (NVIDIA GEAR) — 8K-timestep context via TTT fast weights in GR00T N1.7's DiT; enables one-shot in-context imitation from a human video (Circuit 6/10 vs a baseline's 0/10) at constant latency. The strongest recent unification of long memory + demo-following.

D. Retrieval-augmented (fetch the relevant memory/demo on demand)

Store many experiences/keyframes and retrieve the ones relevant to the current step.

• Memory Retrieval in Visuomotor Policies (long-horizon control) · MemER (scale memory via experience retrieval) · MAP-VLA (memory-augmented prompting) · KEMO (2606.23589, event-driven keyframe memory) · Long-Context IL (focus on key history frames).

E. Token-sequence ICL (next-token prediction over demo + rollout)

Treat [demo tokens; rollout tokens] as one sequence and predict the next action token — LLM-style ICL.

• In-Context Robot Transformer / ICRT (ICRA 2025) — in-context imitation via next-token prediction over sensorimotor tokens (the cluster-E anchor). Keypoint Action Tokens (KAT) and Instant Policy are related graph/token-ICL variants.
• Behavior Prompting Policy (2606.30457) — demonstrations as prompts for manipulation; finds task diversity (not quantity) drives the prompting ability.

F. Play-video in-context (learn from unstructured human play)

• MimicDroid (ICRA 2026) — in-context learning for humanoid manipulation from human play videos — learns to ICL from unlabeled video, no teleop; ~2× real success.

---

3. The benchmark that ties it together — RoboMME

RoboMME turns "which memory helps which task?" into a controlled sweep: 14 memory-augmented policies on one π0.5 backbone (3 memory representations × 3 integration mechanisms). Its Imitation (procedural) suite is the in-context-imitation testbed; its temporal/spatial/object suites test self-history memory. Headline: no single memory design wins everywhere (best non-oracle = FrameSamp+Modulator, 44.5% avg; humans 90.5%) — so the §2 mechanisms are complementary, not competing. Design-space detail: VLA Memory.

---

4. Comparison

Approach	Demo / context form	Conditioning mechanism	No per-task FT?	Memory representation
Vid2Robot	human prompt video	cross-attention (A)	✅	encoded demo tokens
VLBiMan	one-shot demo	vision-language anchor (A)	✅	anchored demo
RoboTTT	human video (one-shot)	fast-weight TTT (C)	✅	parametric (fast weights)
HAMLET / MemoryVLA	own history	recurrent / cognitive memory (B)	n/a (history)	latent / token buffer
MemER / MAP-VLA	stored experiences	retrieval (D)	✅ (retrieve)	external memory store
ICRT / Behavior Prompting	demo token sequence	next-token / cross-attn+diffusion ICL (E)	✅	in-context tokens
MimicDroid	human play video	in-context (F)	✅	in-context

(✅ = specifies a new task by showing a demo, no gradient step per task.)

---

5. Design axes — when to use what

• Clean expert video demo, want motion transfer? → cross-attention (Vid2Robot, VLBiMan).
• Very long demo/rollout, must stay real-time? → fast-weight/TTT (RoboTTT) — memory is parametric, latency constant.
• Many past experiences, need the relevant one? → retrieval (MemER, MAP-VLA, KEMO).
• Long-horizon own-history dependence (counting, re-finding)? → recurrent/keyframe memory (HAMLET, Long-Context-IL) — and consult RoboMME for which representation fits which memory type.
• Unstructured/noisy demos (human play)? → play-video ICL (MimicDroid).

Unifying view: in-context imitation and history-memory are the same operation — attend to non-current context — differing only in whose context (a demonstrator's vs the robot's own) and where it's stored (attention tokens · recurrent latent · fast weights · external store).

---

6. Open challenges

1. One demo ≠ robust policy — sensitivity to demo viewpoint, object identity, and speed; generalization beyond the shown instance is uneven.
2. Context length vs latency — attention over long demos is quadratic; fast-weight/retrieval are the escape hatches but under-benchmarked at scale (RoboTTT §6).
3. Which memory for which task is unsolved — RoboMME shows no universal winner; humans still lead 90.5% vs ~44%.
4. Human→robot demo gap — video demos carry the same embodiment/retargeting gap as egocentric-video pretraining.
5. Evaluation — few benchmarks isolate in-context skill acquisition from ordinary multi-task competence (RoboMME's Imitation suite is a start).

---

7. Links

• Demo-conditioned / one-shot: RoboTTT · VLBiMan · Vid2Robot (2403.12943) · See-Once-Then-Act (2512.07582) · Behavior Prompting (2606.30457) · MimicDroid (2026)
• IROS 2026 🆕: ICLR: In-Context Imitation with Visual Reasoning (E×visual-reasoning — image-space intent traces, USC) · RoboSSM (scalable in-context imitation via state-space models — linear-cost long context) — context: IROS 2026 survey §5.2
• Memory-augmented VLAs: HAMLET · MemoryVLA · MemER · MAP-VLA · Memory Retrieval · Long-Context IL · KEMO (2606.23589)
• Benchmark & synthesis: RoboMME · VLA Memory · VLA Architectures

← Back to Reviews · Home