Loading checkpoints may cause the progress bar to display incorrectly

[aihao2000]

Describe the bug

The part of the training script about loading checkpoints. I noticed that when step<resume_step, progress_bar.update(1) happened. Is this correct? In my understanding this causes the progress bar to display incorrectly. The progress bar marks the number of gradient descents: (global_step, max_train_step). It's just skipping samples and not doing gradient descent

Reproduction

don't need

Logs

No response

System Info

• diffusers version: 0.21.0.dev0
• Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.35
• Python version: 3.11.4
• PyTorch version (GPU?): 2.0.1 (True)
• Huggingface_hub version: 0.16.4
• Transformers version: 4.32.0.dev0
• Accelerate version: 0.22.0.dev0
• xFormers version: 0.0.20
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@williamberman, @sayakpaul, @yiyixuxu

[sayakpaul]

Could you maybe provide some visual description and more information for us to understand this better? I don't understand what's wrong in the existing implementation.

[aihao2000]

@sayakpaul Assume that max train_step is 10000, resume from checkpoint-1000, batch size is 1, and the total batch is 10000. I still need to do 9000 steps of gradient descent. The progressbar will display (0/9000) when training starts. resume_step avoids repeating the data at the beginning of training. The first 1000 batches are trained, but data_loader doesn't know it, so it has the operation of skipping data, right? It will execute "progress_bar.update(1)" a thousand times. At this time progress_bar will change to (1000/9000). I don’t know what the progress bar will become if it executes the real gradient descent nine thousand times, (10000/9000)? It doesn’t feel right to me personally. I modified the code and added a resume_progress_bar, so there is no original picture for the time being.

[sayakpaul]

Sounds like a legitimate concern. WDYT @patil-suraj @williamberman?

[aihao2000]

@sayakpaul By the way,I have another doubt, why does the script call model.train() at the beginning of each epoch? Is it possible that the state of the model is changed during verification? Then why not call it at the end of validation.
And
When I was training under 100 batchsize, I replaced with_transform with map to speed up the process. It had a certain effect, but the training speed was sometimes fast and sometimes slow, and it would get stuck every few steps. Is there any optimization method? Each sample has three pictures.
Thanks

[williamberman]

@sayakpaul Assume that max train_step is 10000, resume from checkpoint-1000, batch size is 1, and the total batch is 10000. I still need to do 9000 steps of gradient descent. The progressbar will display (0/9000) when training starts.

Yes, this is something small in configuring tqdm, would be nice to get a fix in but not super urgent.

resume_step avoids repeating the data at the beginning of training. The first 1000 batches are trained, but data_loader doesn't know it, so it has the operation of skipping data, right?

I don't make assumptions like this about training data ordering. I don't think this is a big deal

It will execute "progress_bar.update(1)" a thousand times. At this time progress_bar will change to (1000/9000).

Yes this is unfortunately already programmed into a good number of the training scripts, and actually takes quite a bit of time. When I commit new scripts I do proper step resuming without manually skipping or what not. Would be nice if someone has bandwidth to go in and fix the other training scripts.

[bghira]

i asked about this last week but was said this is how it's been for a while and that it's all fine, #5005

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.