Skip to content

fusion-embedding-1-2b-preview v0.3

Choose a tag to compare

View all tags
@abtonmoy abtonmoy released this 09 Jul 01:13
v0.3-preview
d971f03

fusion-embedding-1-2b-preview v0.3

Third preview checkpoint. v0.3 keeps the v0.2 contrastive stage (484K pairs) and adds the training stage every top-scoring audio-text model uses but connector-only models had not: a short in-domain fine-tune on the AudioCaps train split (400 steps at reduced learning rate, warm-started from the v0.2 checkpoint, both towers still frozen, base still byte-identical). Architecture and inference API are unchanged; existing integrations work as-is.

Model weights and model card: https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview (tags v0.1-preview / v0.2-preview / v0.3-preview; from_pretrained(..., revision=...) pins a version)

v0.2 → v0.3

Benchmark v0.2 v0.3
AudioCaps test A→T R@1 0.279 0.332
AudioCaps test A→T R@10 0.717 0.741
AudioCaps test T→A R@10 0.736 0.746
Clotho zero-shot A→T / T→A R@10 0.448 / 0.449 0.433 / 0.460
VGGSound audio↔text R@10 0.588 / 0.631 0.625 / 0.645
VGGSound audio↔image R@10 (emergent) 0.418 / 0.440 0.407 / 0.428

The in-domain stage trades 1.5 points of zero-shot Clotho A→T and ~1 point of emergent audio→image for the AudioCaps and audio↔text gains; Clotho T→A improves. v0.2 remains available via its tag for use cases that prioritize the emergent audio→image direction.

All protocols unchanged (AudioCaps 5-reference min-rank over 883 clips; Clotho v2.1 excluded from all training; VGGSound-AV 696 pairs with zero audio-visual training pairs; text↔image is the frozen base's, byte-identical across versions). Reference points: fine-tuned CLAP-family models reach AudioCaps A→T R@10 0.906–0.928 with both encoders trained end-to-end.

Training changes in this repository at this tag

  • init_from_ckpt: second-stage fine-tuning entry that warm-starts the connector, whitening, and temperature from a finished checkpoint (fresh optimizer/schedule; resume-state isolated per stage)
  • Relevance-aware loss flags (soft_label_beta, fn_mask_threshold) validated in a matched A/B on a 592K-pair corpus (+2.6 A→T R@10 over plain InfoNCE at equal steps)
  • Ingestion safety: finalized shard directories refuse accidental re-ingestion; shard-alignment verifier and tail-ingestion tooling

License

Preview weights: CC-BY-NC-4.0 (research; see the model card for training-data provenance).

Assets 2
Loading