SigLIP impl

[rwightman]

Re #618

[lucasb-eyer]

Can't comment on the distributed part of the code as I don't know that part of PyTorch, but the rest (loss details, bias/temp/inits) LGTM.

[rwightman]

@lucasb-eyer thanks for taking a look, yeah the dist part is where a lot of the risk is, but seems to be behaving on local cc12m runs comparing single to 4x GPU.

[lucasb-eyer]

FYI: in our code, Basil implemented a small unit-test checking both formulations for "almost equalness" of chunked vs non-chunked, this gave us good reassurance in the implementation (+looking at profiler for memory use).

[rwightman]

I've tested

• convnext_tiny on cc12m old InfoNCE run vs new SigLIP (36.13 vs 36.46 zero-shot in1k) (4 GPU)
• initial convergence w/ siglip + grad accum enabled
• initial convergence of custom text and original clip models w/o siglip (original CLIP InfoNCE loss)
• initial convergence with bidirection exchange and unidirectional
• validating several existing models

Will merge shortly to prevent this getting stale