Save interval not working

[naimavahab]

I am running Mosaicbert pretraining code from /examples/benchmarks/bert using the docker container. Everything works well, other than saving number of checkpoints.
My conf is as below :

save_interval: 1ep
save_num_checkpoints_to_keep: -1 
save_folder: ./{run_name}/ckpt    
save_overwrite: True

Even tried with giving batch interval and without specifying save interval as well(which should by default save every epoch). Whatever interval I specify, only 1 checkpoint is getting saved. I was wondering if any one else facing the same issue.

[mvpatel2000]

How many epochs does your run go for? Also, what is the save_filename? Is it unique per timestamp (to verify it isn't being overwritten each time)

[naimavahab]

Only 1 checkpoint is getting saved..in the format of 'ep3-ba1458-rank0.pt'. If I specify save interval 1ep only 1st epoch gets saved even if I have 30 epochs. And if save interval is 3ep, the 3rd epoch gets saved, rest are ignored

[mvpatel2000]

Hm... do you get any errors or traces? Mind sharing a minimal repro please?

[naimavahab]

There are no errors and everything working fine including plotting the loss etc.
I am using this git branch for mosaicbert pretraining. https://github.com/Skylion007/mosaicml-examples/tree/skylion007/add-fa2-to-bert/examples

[eracah]

Mind sharing what version of composer you are using?

[naimavahab]

0.17.2

[eracah]

That's a very old (>6 months old) version of composer. is there a reason you need to use that old of a version?

[naimavahab]

I have been using this particular docker image https://github.com/Skylion007/mosaicml-examples/tree/skylion007/add-fa2-to-bert/examples to use flashattention, triton etc. These set up require sepcific versions, if I update to latest composer, triton fails.
But now I managed by updating to a slightly higher version like 0.19. And it works fine. But I wonder why composer failed only at checkpoint part for the previous 0.17.2 version

[mvpatel2000]

Going to close as it seems to work.

I'm not super sure what the bug was, but it's certainly possible there was an issue in older versions. Definitely recommend upgrading to latest :)