CLIP Training Example Bug - Overfitting

[humanely]

Describe the bug

After reading numerous posts on overfitting, I am convinced that the example run_clip.py has bugs. it doesn't converge. Earlier I posted it for discussion here

Reproduction

Execute the run_clip.py and watch for conv/divg.

Logs

No response

System Info

Ubuntu 22
transfomer 4.39
diffuser 0.27

Who can help?

No response

[bghira]

do you have training logs handy? wandb project would be nice to see