Why stair-like loss curve?

[ChenDelong1999]

As in here as well as my own implementation, stair-like loss curves are observed. Any possible reason for this?

[rwightman]

I can't speak to the training run for the graph as I didn't do them, @mitchellnw would have a better idea... but looks like it could be a shuffling issue (as in not properly shuffling)

[mitchellnw]

My guess is also a shuffling issue with webdataset when these were run

[rom1504]

If the data is not preshuffled you need both shards shuffling and local shuffling

[ChenDelong1999]

Perhaps it is not a shuffling issue with webdataset, since I train the model on CC3M (csv dataset), and observed the following curves.

which looks very similar to this curve in open_clip/docs/clip_conceptual_captions.md:

Loss increases within each epoch, then decrease after each epoch...

[rom1504]

@ChenDelong1999 did you preshuffle the dataset (sort randomly the dataset) ?

[rwightman]

CsvDataset should be shuffled every epoch, pre-shuffle isn' really relevant. Might be worth checking that

open_clip/src/training/train.py

Line 62 in d9ee4aa

sampler.set_epoch(epoch)

set_epoch is def being called in distributed case...

[mitchellnw]

Looking at this again I wonder if it is caused by the scale param, which also exhibits stair-like behaviour.

I would expect that stair-like scale => stair-like loss.

But I have no guesses for why scale has stair-like behaviour.

To test this hypothesis I would use a 10x smaller learning rate on the scale parameter and see if this resolves the issue.

[rwightman]

@mitchellnw I've noticed that the scale param has interesting relationship with the LR/loss, I wonder if it's almost behaving in a slightly oscillatory control systems fashion. The scale is strongly impacted by the LR as well, if the LR is high enough the scale will not converge to 100 until it lowers

[mitchellnw]

Interesting. I wonder how accuracy/loss would be impacted if this learnable param was replaced by a scheduled param---something like 100 - k*cosine_decay(iteration).

[viyjy]

here

Hi, thanks for your answer, but the scale param does not exhibit stair-like behaviour during the training process, isn't it? On the other hand, scale is learnable, it shouldn't be stair-like, right?

[JiachengCheng96]

Hi @mitchellnw @rom1504, I observed a similar stair-like loss curve when training a ViT-B-32 model on the LAION400M. Do you think this is normal? Or could you share one loss curve of any of your models on LAION400M? Any information would be greatly appreciated.

[mitchellnw]

are you using --dataset-resampled? I think our working hypothesis is that this is a shuffling issue..

[JiachengCheng96]

@mitchellnw Thanks for the reply! No, I didn't use --dataset-resampled. Do you think such stair-like loss should be expected when training without --dataset-resampled? I am trying to reproduce the VIT-B-32 result on LAION400M using the exact same setting provided in #191 (which didn't use --dataset-resampled), but gets accuracy a bit lower than your 62.9%. Hence, I am wondering if there is something wrong with my stair-like loss.