fusion-embedding-2-2b-preview v0.3

Generation 2, third release: two revisions from the v0.3 training line.

v0.3 ships two checkpoints under one repository. The default (v0.3-preview) is retrieval-tuned; a keyword-tuned revision (v0.3-kw-preview) is available for spoken-command use. Non-audio outputs remain bit-for-bit the base model's, verified on the shipped artifacts before push.

Highlights (v0.3-preview, the default)

• AudioCaps audio-to-text R@10 0.743 -> 0.785 over v0.2 (R@1 0.310 -> 0.333); text-to-audio R@10 0.782.
• MAEB(audio-only) 19-task mean 0.454; leads the board on IEMOCAP speaker-gender classification (0.938).
• Emotion recognition regressed: RAVDESS zero-shot 0.35 -> 0.21. The v0.3 mix improved spoken-content understanding at the cost of vocal-prosody sensitivity; evaluate before adopting if speech-emotion is central.

v0.3-kw-preview (keyword-tuned)

• SpeechCommands zero-shot 0.133 -> 0.929. AudioCaps a2t R@10 0.749, lower than the default's 0.785; use the default for general audio-text retrieval and this revision only when spoken-command matching is the priority.

Weights and model card: https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview (revision v0.3-preview is the default; v0.3-kw-preview for the keyword variant; v0.1-preview and v0.2-preview remain pinned for reproducibility). Technical report: https://arxiv.org/abs/2607.18666

fusion-embedding-2-2b-preview v0.2

Generation 2, second release: fine-tuned on the expanded AudioCaps 2.0 training pool.

v0.2 fine-tunes the v0.1 checkpoint's trained components on the AudioCaps 2.0-expanded in-domain pool (86,394 pairs, 1,600 steps; train split verified disjoint from every evaluation protocol we report). Non-audio outputs remain bit-for-bit the base model's.

Highlights

• AudioCaps audio-to-text R@10 0.743 -> 0.759 (R@1 0.302 -> 0.310), text-to-audio 0.775 -> 0.785.
• Clotho zero-shot t2a 0.482 -> 0.485; VGGSound within noise of v0.1 under paired protocols; emergent audio-to-image up ~1.5 points.
• Public MAEB leaderboard results updated to this revision (mean 0.361 vs 0.354 across the nine-task sound-event tier).

Weights and model card: https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview (revision tag v0.2-preview; v0.1-preview remains pinned for reproducibility). All gains verified against v0.1 under identical protocols.

fusion-embedding-2-2b-preview v0.1

First release of generation 2: modality-gated deep adapters inside the frozen backbone.

fusion-embedding-2 keeps generation 1's frozen stack (byte-frozen Qwen3-VL-Embedding-2B base, frozen Qwen2.5-Omni audio tower, the 16.4M connector) and adds a bottleneck adapter to each of the 28 frozen decoder layers, hard-gated to audio inputs: 44.2M adapter parameters that never execute on text, image, or video, whose outputs remain bit-for-bit those of the released base, verified after every training run.

Highlights

• AudioCaps audio-to-text R@10 0.743, text-to-audio 0.775; Clotho zero-shot t2a 0.482; VGGSound audio-text 0.665/0.681 — the family's best text-to-audio results on every benchmark at release.
• Emergent audio-to-image retrieval preserved and strengthened at the pretraining stage (R@10 0.443, zero paired audio-visual data).
• Trained in hours on a single GPU; adapters warm-start from the generation-1 recipe.

Weights and model card: https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview (revision tag v0.1-preview). Loadable in mteb via mteb.get_model("EximiusLabs/fusion-embedding-2-2b-preview").

fusion-embedding-1-2b-preview v0.3

fusion-embedding-1-2b-preview v0.3

Third preview checkpoint. v0.3 keeps the v0.2 contrastive stage (484K pairs) and adds the training stage every top-scoring audio-text model uses but connector-only models had not: a short in-domain fine-tune on the AudioCaps train split (400 steps at reduced learning rate, warm-started from the v0.2 checkpoint, both towers still frozen, base still byte-identical). Architecture and inference API are unchanged; existing integrations work as-is.

Model weights and model card: https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview (tags v0.1-preview / v0.2-preview / v0.3-preview; from_pretrained(..., revision=...) pins a version)

v0.2 → v0.3

Benchmark	v0.2	v0.3
AudioCaps test A→T R@1	0.279	0.332
AudioCaps test A→T R@10	0.717	0.741
AudioCaps test T→A R@10	0.736	0.746
Clotho zero-shot A→T / T→A R@10	0.448 / 0.449	0.433 / 0.460
VGGSound audio↔text R@10	0.588 / 0.631	0.625 / 0.645
VGGSound audio↔image R@10 (emergent)	0.418 / 0.440	0.407 / 0.428

The in-domain stage trades 1.5 points of zero-shot Clotho A→T and ~1 point of emergent audio→image for the AudioCaps and audio↔text gains; Clotho T→A improves. v0.2 remains available via its tag for use cases that prioritize the emergent audio→image direction.

All protocols unchanged (AudioCaps 5-reference min-rank over 883 clips; Clotho v2.1 excluded from all training; VGGSound-AV 696 pairs with zero audio-visual training pairs; text↔image is the frozen base's, byte-identical across versions). Reference points: fine-tuned CLAP-family models reach AudioCaps A→T R@10 0.906–0.928 with both encoders trained end-to-end.

Training changes in this repository at this tag

• init_from_ckpt: second-stage fine-tuning entry that warm-starts the connector, whitening, and temperature from a finished checkpoint (fresh optimizer/schedule; resume-state isolated per stage)
• Relevance-aware loss flags (soft_label_beta, fn_mask_threshold) validated in a matched A/B on a 592K-pair corpus (+2.6 A→T R@10 over plain InfoNCE at equal steps)
• Ingestion safety: finalized shard directories refuse accidental re-ingestion; shard-alignment verifier and tail-ingestion tooling

License

Preview weights: CC-BY-NC-4.0 (research; see the model card for training-data provenance).

fusion-embedding-1-2b-preview v0.2

fusion-embedding-1-2b-preview v0.2

Second preview checkpoint, trained on a 484K-pair corpus (3.7× v0.1): the full AudioCaps train split, FSD50K, WavCaps/AudioSet_SL, and a 318K-clip LAION-FreeSound subset, with 10-second training windows. Architecture and inference API are unchanged from v0.1; existing integrations work as-is.

Model weights and model card: https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview (v0.1 remains available as a pinned revision)

v0.1 → v0.2

Benchmark	v0.1	v0.2
AudioCaps test A→T R@1 / R@10	0.216 / 0.626	0.279 / 0.717
AudioCaps test T→A R@10	0.680	0.736
Clotho v2.1 zero-shot A→T / T→A R@10	0.252 / 0.329	0.448 / 0.449
VGGSound audio↔image R@10 (emergent, centered readout)	0.368 / 0.388	0.418 / 0.440
VGGSound audio↔text R@10	0.555 / 0.592	0.588 / 0.631

All evaluations use the same protocols as v0.1 (AudioCaps 5-reference min-rank; Clotho excluded from training; VGGSound-AV 696 pairs with no audio-visual training pairs). Reference points: fully fine-tuned CLAP-family models reach AudioCaps A→T R@10 0.906–0.928; zero-shot WavCaps-CLAP reaches Clotho A→T R@10 0.576.

Training changes in this repository at this tag

• Corpus ingestion for AudioCaps train and LAION-FreeSound WebDataset tars (resumable, blacklisted against all evaluation sets)
• 10-second random-crop training windows for long-form audio
• Full-corpus negative bank scaled to 484K entries

License

Preview weights: CC-BY-NC-4.0 (research; see the model card for training-data provenance).

fusion-embedding-1-2b-preview

fusion-embedding-1-2b-preview

First public preview of Fusion Embedding 1: a single embedding space covering text, images, video, and audio, built by extending the frozen Qwen3-VL-Embedding-2B base with a trained ~16M-parameter audio connector. The base model is unmodified; its text/image/video retrieval performance carries over unchanged.

Model weights and model card: https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview

Highlights

• Emergent cross-modal alignment: audio→image retrieval at R@10 0.368 over 696 VGGSound candidates (chance 0.014) with no audio-visual pairs in training.
• On the same protocol, outperforms ImageBind-Huge on audio↔text (0.555/0.592 vs 0.404/0.348) and text↔image (0.331/0.319 vs 0.243/0.282); ImageBind leads on its directly supervised audio↔image pair.
• AudioCaps test (5-reference protocol): A→T R@10 0.626; Clotho v2.1 zero-shot: A→T R@10 0.252.
• Matryoshka embeddings (2048 → 64), packaged inference API (FusionEmbedder), ~60MB distribution.

In this repository at this tag

• Training and evaluation pipeline: contrastive connector training with a full-corpus frozen-text negative bank, native chat-template protocol, published-protocol evaluation (AudioCaps multi-reference, Clotho zero-shot, VGGSound cross-modal with an ImageBind baseline).
• Data pipeline: resumable fused ingestion (HF datasets and WebDataset tars), eval-leakage blacklisting, precomputed frame and text-embedding caches.
• Reliability: per-shard/per-tar ingestion resume, training resume checkpoints with config fingerprinting, automatic retry, divergence and OOM handling.

License

Code: repository license. Preview weights: CC-BY-NC-4.0 (research; see the model card for training-data provenance). Updated checkpoints trained on a substantially larger corpus will be released under the same model family.