Skip to content

v0.2.0 — JEPA / V-JEPA 2 / LeWorldModel world models

Latest
Latest

Choose a tag to compare

View all tags
@jascal jascal released this 30 May 03:32
v0.2.0
1ccdb84
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

Adds first-class support for JEPA-family joint-embedding world models (#1).

New JepaAdapter

Convert V-JEPA 2, I-JEPA, and LeWorldModel checkpoints to verified .n.orca.md:

n-orca hf convert facebook/vjepa2-vitl-fpc64-256 --out vjepa2.n.orca.md --mermaid vjepa2.mmd
n-orca hf convert quentinll/lewm-pusht --out lewm-pusht.n.orca.md
  • Normalizes the flat HF transformers config (V-JEPA 2 / I-JEPA, pred_* fields) and the nested Hydra config (LeWorldModel, matched structurally — no model_type) into one encoder → predictor DAG.
  • Dual latent outputs (encoder_latents, predicted_latents) share the encoder's embedding space; an output_shape invariant enforces latent-dim consistency.
  • Optional additive action conditioning and projector head. Mask tokens, the EMA stop-gradient target, SIGReg, and rotary/sincos positions are captured as verification rules.

New ops

  • TubeletEmbed (Conv3d, video) and PatchEmbed (Conv2d, image) patch embedders — shape inference + param counting that matches PyTorch exactly.

CLI

  • n-orca hf download --include-processor also fetches preprocessor_config.json / video_preprocessor_config.json (V-JEPA ships the latter).

Examples

  • examples/hf-generated/vjepa2.n.orca.md (V-JEPA 2 ViT-L, 326M params) and lewm-pusht.n.orca.md (action-conditioned), each with a matching .mmd.

Tests

  • 140 passing (+22): new ops, dispatch incl. structural LeWM detection, verify, parser round-trip, Mermaid, and torch forward passes.

Full diff: v0.1.0...v0.2.0

Assets 4
Loading