Skip to content

fusion-embedding-1-2b-preview

Choose a tag to compare

View all tags
@abtonmoy abtonmoy released this 07 Jul 01:32
v0.1-preview
5147d6e

fusion-embedding-1-2b-preview

First public preview of Fusion Embedding 1: a single embedding space covering text, images, video, and audio, built by extending the frozen Qwen3-VL-Embedding-2B base with a trained ~16M-parameter audio connector. The base model is unmodified; its text/image/video retrieval performance carries over unchanged.

Model weights and model card: https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview

Highlights

  • Emergent cross-modal alignment: audio→image retrieval at R@10 0.368 over 696 VGGSound candidates (chance 0.014) with no audio-visual pairs in training.
  • On the same protocol, outperforms ImageBind-Huge on audio↔text (0.555/0.592 vs 0.404/0.348) and text↔image (0.331/0.319 vs 0.243/0.282); ImageBind leads on its directly supervised audio↔image pair.
  • AudioCaps test (5-reference protocol): A→T R@10 0.626; Clotho v2.1 zero-shot: A→T R@10 0.252.
  • Matryoshka embeddings (2048 → 64), packaged inference API (FusionEmbedder), ~60MB distribution.

In this repository at this tag

  • Training and evaluation pipeline: contrastive connector training with a full-corpus frozen-text negative bank, native chat-template protocol, published-protocol evaluation (AudioCaps multi-reference, Clotho zero-shot, VGGSound cross-modal with an ImageBind baseline).
  • Data pipeline: resumable fused ingestion (HF datasets and WebDataset tars), eval-leakage blacklisting, precomputed frame and text-embedding caches.
  • Reliability: per-shard/per-tar ingestion resume, training resume checkpoints with config fingerprinting, automatic retry, divergence and OOM handling.

License

Code: repository license. Preview weights: CC-BY-NC-4.0 (research; see the model card for training-data provenance). Updated checkpoints trained on a substantially larger corpus will be released under the same model family.

Assets 2
Loading