Skip to content

Add JEPA / V-JEPA 2 / LeWorldModel HF adapter#1

Merged
jascal merged 3 commits into
mainfrom
add-jepa-vjepa-leworldmodel-adapter
May 30, 2026
Merged

Add JEPA / V-JEPA 2 / LeWorldModel HF adapter#1
jascal merged 3 commits into
mainfrom
add-jepa-vjepa-leworldmodel-adapter

Conversation

@jascal

@jascal jascal commented May 30, 2026

Copy link
Copy Markdown
Owner

What

Adds full support for JEPA-family joint-embedding world models so users can hf download / hf convert them like any other family. Primary focus: V-JEPA 2 (facebook/vjepa2-*) and LeWorldModel (quentinll/lewm-*); also covers I-JEPA (facebook/ijepa_*).

n-orca hf convert facebook/vjepa2-vitl-fpc64-256 --out vjepa2.n.orca.md --mermaid vjepa2.mmd
n-orca verify vjepa2.n.orca.md          # VALID  params=325,967,488  depth=222
n-orca hf convert quentinll/lewm-pusht --out lewm-pusht.n.orca.md

New JepaAdapter (n_orca/hf/adapters/jepa.py)

Normalizes two unrelated config schemas into one encoder → predictor DAG:

  • Flat HF transformers configs — V-JEPA 2 (vjepa2 / VJEPA2Model) and I-JEPA (ijepa / IJepaModel), with pred_* predictor fields.
  • Nested Hydra configs — LeWorldModel (quentinll/lewm-*), which ship no model_type and are matched structurally (_target_ + nested encoder/predictor dicts).

The graph parses the ViT encoder, the joint-embedding predictor (distinct width/depth), a latent bridge between them, optional additive action conditioning, and an optional projector head. It emits two outputs — encoder_latents (the representation) and predicted_latents (the forecast) — that share the encoder's embedding space, so a single output_shape invariant enforces latent-dim consistency.

Mask tokens, the EMA stop-gradient target, SIGReg, and rotary/sincos positions are captured as verification rules + hyperparameters (mirroring how the LLaMA adapter treats RoPE/RMSNorm), keeping output round-trippable and compilable.

Supporting changes

  • OpsTubeletEmbed (Conv3d, video) and PatchEmbed (Conv2d, image) patch embedders with shape inference + param counting. Verifier param count matches PyTorch exactly (11,943,104 for LeWM).
  • Blockspre_norm_decoder_block gains a prefix= arg so encoder/predictor stacks don't collide (default reproduces prior names byte-for-byte).
  • CLIn-orca hf download --include-processor also fetches preprocessor_config.json / video_preprocessor_config.json (V-JEPA ships the latter).
  • Examplesexamples/hf-generated/vjepa2.{n.orca.md,mmd} (V-JEPA 2 ViT-L, 326M params) and lewm-pusht.{n.orca.md,mmd} (action-conditioned). Verified byte-identical to live-Hub hf convert output.
  • README — JEPA row in the family table; TubeletEmbed/PatchEmbed in the op table; download/convert examples.

Testing

140 passing (was 118). +22 tests: new ops, dispatch (incl. structural LeWM detection with no model_type), clean verify, parser round-trip, Mermaid branch rendering, and torch forward passes for video V-JEPA 2 (5D (B,C,T,H,W) clip), encoder-only I-JEPA, and action-conditioned LeWM. The exact target commands were run against the live Hub.

🤖 Generated with Claude Code

jascal and others added 3 commits May 29, 2026 23:17
Adds full support for joint-embedding predictive world models so users can
download and convert them like any other HF family.

New `JepaAdapter` (n_orca/hf/adapters/jepa.py) normalizes two unrelated config
schemas into one encoder -> predictor DAG:
- Flat HF transformers configs: V-JEPA 2 (vjepa2 / VJEPA2Model) and I-JEPA
  (ijepa / IJepaModel), with pred_* predictor fields.
- Nested Hydra configs: LeWorldModel (quentinll/lewm-*), which ship no
  model_type and are matched structurally.

The graph parses the ViT encoder and the joint-embedding predictor (distinct
widths/depths), a latent bridge between them, optional additive action
conditioning, and an optional projector head. It emits two latent outputs
(encoder_latents, predicted_latents) that share the encoder's embedding space,
so a single output_shape invariant enforces latent-dim consistency. Mask
tokens, the EMA stop-gradient target, SIGReg, and rotary/sincos positions are
captured as verification rules + hyperparameters (mirroring how the LLaMA
adapter treats RoPE/RMSNorm), keeping the output round-trippable and
compilable.

Op library: add TubeletEmbed (Conv3d, video) and PatchEmbed (Conv2d, image)
patch embedders with shape inference + param counting; the verifier's param
count matches PyTorch exactly. Add a `prefix=` arg to pre_norm_decoder_block so
encoder and predictor stacks don't collide (default reproduces prior names
byte-for-byte).

CLI: `n-orca hf download --include-processor` also fetches
preprocessor / video_preprocessor configs (V-JEPA ships the latter).

Examples: vjepa2.n.orca.md (V-JEPA 2 ViT-L, 326M params) and lewm-pusht
.n.orca.md (action-conditioned LeWorldModel), each with a matching .mmd —
verified byte-identical to live-Hub `hf convert` output.

Tests: +22 covering the new ops, dispatch (incl. structural LeWM detection),
clean verify, parser round-trip, Mermaid, and torch forward passes for video
V-JEPA 2, encoder-only I-JEPA, and action-conditioned LeWM. Full suite: 140
passing (was 118).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Expand the LeWM Hydra-detection docstring/comments so the structural
  fallback path (encoder+predictor dict pair, no model_type) is obvious to
  future maintainers.
- README: "SIGReg (latent regularizer)" for clarity in the family table.

No behavior change; 140 tests still pass and example files are byte-stable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New backward-compatible feature: JepaAdapter + TubeletEmbed/PatchEmbed ops +
`hf download --include-processor`. Minor version bump under semver.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jascal jascal merged commit 1ccdb84 into main May 30, 2026
3 checks passed
@jascal jascal deleted the add-jepa-vjepa-leworldmodel-adapter branch May 30, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant