-
Notifications
You must be signed in to change notification settings - Fork 0
JEPA learning
The JEPA (Joint-Embedding Predictive Architecture) primarily uses Regularized Self-Supervised Learning rather than standard contrastive learning or generative reconstruction.
While many AI models rely on comparing "positive" and "negative" examples (contrastive) or rebuilding pixels (generative), JEPA focuses on predicting missing information in a latent (abstract) space.
Yann LeCun, the architect of JEPA, argues that contrastive learning scales poorly because it requires a massive number of "negative" examples (different images) to prevent the model from collapsing. Instead, JEPA uses regularization to ensure the model doesn't "cheat" by giving the same output for every input (a problem called representation collapse).
Key components of this method include:
-
Predictive Objective: Instead of predicting pixels (like a Generative model), it predicts the embedding of a missing part of the data. For example, in I-JEPA, the model looks at a few parts of an image and tries to predict the mathematical representation of the hidden parts.
-
No Negative Samples: Unlike contrastive methods (like SimCLR or CLIP), JEPA doesn't need to look at "wrong" images to learn what's "right."
-
VICReg (Variance-Invariance-Covariance Regularization): Many JEPA implementations use this specific technique to keep representations informative. It forces the embeddings to have high variance (so they don't all look the same) and low covariance (so the model doesn't waste dimensions repeating the same info).
| Method | Target | Primary Loss Type |
|---|---|---|
| Generative (MAE) | Pixels / Tokens | Reconstruction Loss ( |
| Contrastive (CLIP) | Similar vs. Dissimilar | InfoNCE / Ranking Loss |
| JEPA | Latent Embeddings | Prediction Error + Regularization |
-
Context Encoder: Processes the visible part of the data.
-
Target Encoder: Processes the full data to provide the "ground truth" embedding (often updated via Exponential Moving Average (EMA) to keep training stable).
-
Predictor: A small network that takes the context's output and tries to guess the target's output.
By predicting in "thought space" (embeddings) rather than "pixel space," JEPA ignores irrelevant details—like the exact position of every leaf on a tree—and focuses on high-level semantic structures. This makes it significantly more computationally efficient than generative models.