Trainer doesn't finish at last step.

[roperi]

Version (branch): data_backend

Once trainer finishes going through all the total steps, it continues with even more steps.

8/27/2023 13:50:25 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:25 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:25 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:25 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:25 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:25 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:25 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:25 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
Steps: : 624it [2:35:21, 23.78s/it, lr=1.5e-6, step_loss=0.19]  08/27/2023 13:50:28 - DEBUG - __main__ - Starting into epoch: 98
08/27/2023 13:50:28 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:28 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:28 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:28 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:28 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:28 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:28 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:28 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:29 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:29 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:29 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:29 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:29 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:29 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:29 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:29 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
Steps: : 624it [2:35:25, 23.78s/it, lr=1.5e-6, step_loss=0.115]08/27/2023 13:50:32 - DEBUG - __main__ - Starting into epoch: 99
08/27/2023 13:50:32 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:32 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:32 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:32 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:32 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:32 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:32 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:32 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:33 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:33 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:33 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:33 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:33 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:33 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)
08/27/2023 13:50:33 - DEBUG - MultiaspectImage - Image size before rotation: (768, 768)
08/27/2023 13:50:33 - DEBUG - MultiaspectImage - Image size after rotation: (768, 768)

In fact, trainer continues beyond number of NUM_EPOCHS and even does validation samples every VALIDATION_STEPS. I set my MAX_NUM_STEPS to a incredible high number. I wonder if it's is restarting all over with MAX_NUM_STEPS.

[bghira]

the bug seemed to be that it just didn't consider that argument's value when ending training. i've never noticed that as i rely on setting the number of training steps instead.

now you can set the value for training steps to zero, and it will use epochs instead.

[roperi]

now you can set the value for training steps to zero, and it will use epochs instead.

very good

[roperi]

Still won't stop.
version: data_backend (latest)

Steps: 100%|██████████| 200/200 [1:00:22<00:00, 16.91s/it, lr=1.5e-6, step_loss=0.102]{'attention_type'} was not found in config. Values will be initialized to default values.
Configuration saved in simpletuner-results/checkpoint-200/unet_ema/config.json
Model weights saved in simpletuner-results/checkpoint-200/unet_ema/diffusion_pytorch_model.safetensors
Configuration saved in simpletuner-results/checkpoint-200/unet/config.json
Model weights saved in simpletuner-results/checkpoint-200/unet/diffusion_pytorch_model.safetensors
08/28/2023 14:45:38 - INFO - __main__ - Saved state to simpletuner-results/checkpoint-200
Steps: 100%|██████████| 200/200 [1:01:36<00:00, 16.91s/it, lr=1.5e-6, step_loss=0.081]08/28/2023 14:45:54 - INFO - __main__ - Running validation... 
 Generating 1 images.
{'add_watermarker'} was not found in config. Values will be initialized to default values.
                                                                     Loaded scheduler as DDIMScheduler from `scheduler` subfolder of something/something.
Loading pipeline components...: 100%|██████████| 5/5 [00:00<00:00, 2200.58it/s]
08/28/2023 14:46:00 - INFO - __main__ - Generating validation image: close up photo of someone
08/28/2023 14:46:09 - DEBUG - __main__ - Starting into epoch: 1
Steps: 100%|██████████| 200/200 [1:01:57<00:00, 16.91s/it, lr=1.5e-6, step_loss=0.124]08/28/2023 14:46:15 - INFO - __main__ - Running validation... 
 Generating 1 images.
{'add_watermarker'} was not found in config. Values will be initialized to default values.
                                                                     Loaded scheduler as DDIMScheduler from `scheduler` subfolder of something/something.
Loading pipeline components...: 100%|██████████| 5/5 [00:00<00:00, 1571.14it/s]
08/28/2023 14:46:22 - INFO - __main__ - Generating validation image: close up photo of someone
08/28/2023 14:46:31 - DEBUG - __main__ - Starting into epoch: 2
Steps: 100%|██████████| 200/200 [1:02:18<00:00, 16.91s/it, lr=1.5e-6, step_loss=0.108]08/28/2023 14:46:37 - INFO - __main__ - Running validation... 
 Generating 1 images.
{'add_watermarker'} was not found in config. Values will be initialized to default values.
                                                                     Loaded scheduler as DDIMScheduler from `scheduler` subfolder of something/something.
Loading pipeline components...: 100%|██████████| 5/5 [00:00<00:00, 2181.81it/s]
08/28/2023 14:46:43 - INFO - __main__ - Generating validation image: close up photo of someone
08/28/2023 14:46:52 - DEBUG - __main__ - Starting into epoch: 3
Steps: : 201it [1:02:40, 53.24s/it, lr=1.5e-6, step_loss=0.11] 08/28/2023 14:46:58 - DEBUG - __main__ - Starting into epoch: 4
Steps: : 201it [1:02:46, 53.24s/it, lr=1.5e-6, step_loss=0.0871]08/28/2023 14:47:04 - DEBUG - __main__ - Starting into epoch: 5
Steps: : 201it [1:02:52, 53.24s/it, lr=1.5e-6, step_loss=0.0717]08/28/2023 14:47:10 - DEBUG - __main__ - Starting into epoch: 6
Steps: : 201it [1:02:57, 53.24s/it, lr=1.5e-6, step_loss=0.164] 08/28/2023 14:47:16 - DEBUG - __main__ - Starting into epoch: 7
Steps: : 202it [1:03:03, 44.26s/it, lr=1.5e-6, step_loss=0.0542]08/28/2023 14:47:22 - DEBUG - __main__ - Starting into epoch: 8
Steps: : 202it [1:03:09, 44.26s/it, lr=1.5e-6, step_loss=0.0899]08/28/2023 14:47:27 - DEBUG - __main__ - Starting into epoch: 9

[bghira]

there should be more info at the start of the log on how it calculated the epochs. can you show that?

[bghira]

i would also suggest now using the release branch, as it contains the most recently-accepted and tested changes.

[roperi]

there should be more info at the start of the log on how it calculated the epochs. can you show that?

08/28/2023 13:44:16 - INFO - __main__ - After all of the heave-ho messing around, we have settled on 100 epochs and 2 steps per epoch.                                                                                           
08/28/2023 13:44:16 - DEBUG - git.cmd - Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/workspace/SimpleTuner, universal_newlines=False, shell=None, istream=None)                                                           
wandb: Currently logged in as: someuser. Use `wandb login --relogin` to force relogin                                                                                                                                          
08/28/2023 13:44:17 - DEBUG - git.cmd - Popen(['git', 'cat-file', '--batch-check'], cwd=/workspace/SimpleTuner, universal_newlines=False, shell=None, istream=<valid stream>)                                                    
wandb: Tracking run with wandb version 0.15.8                                                                                                                                                                                    
wandb: Run data is saved locally in /workspace/SimpleTuner/wandb/run-20230828_134417-zp1i9vfq                                                                                                                                    
wandb: Run `wandb offline` to turn off syncing.                                                                                                                                                                                  
wandb: Syncing run misty-shy-1                                                                                                                                                                                                   
wandb: ⭐️ View project at https://wandb.ai/someuser/myrun                                                                                                                                                     
wandb: 🚀 View run at https://wandb.ai/someuser/myrun/runs/zp1i9vfq                                                                                                                                           
08/28/2023 13:44:18 - INFO - __main__ - ***** Running training *****                                                                                                                                                             
08/28/2023 13:44:18 - INFO - __main__ -   Num examples = 38                                                                                                                                                                      
08/28/2023 13:44:18 - INFO - __main__ -   Num Epochs = 100                                                                                                                                                                       
08/28/2023 13:44:18 - INFO - __main__ -   Instantaneous batch size per device = 6                                                                                                                                                
08/28/2023 13:44:18 - INFO - __main__ -   Total train batch size (w. parallel, distributed & accumulation) = 24                                                                                                                  
08/28/2023 13:44:18 - INFO - __main__ -   Gradient Accumulation steps = 4                                                                                                                                                        
08/28/2023 13:44:18 - INFO - __main__ -   Total optimization steps = 200                                                                                                                                                         

[bghira]

it does say num epochs 100. how are you setting it?

[roperi]

it does say num epochs 100. how are you setting it?

In sdxl-env.sh

# Max number of steps OR epochs can be used. But we default to Epochs.
export MAX_NUM_STEPS=0
# Will likely overtrain, but that's fine.
export NUM_EPOCHS=100

[bghira]

and it already did the 100 epochs? we're just not detecting that?

[roperi]

and it already did the 100 epochs? we're just not detecting that?

I'm not sure if i understand. After completing the 100 epochs (or 200 steps) the trainer continued generating samples, running steps, starting with epoch 1, then 2, 3, etc.