ValueError: num_samples should be a positive integer value, but got num_samples=0

[popovidis]

Traceback (most recent call last):
  File "/workspace/kohya_ss/sd-scripts/train_db.py", line 529, in <module>
    train(args)
  File "/workspace/kohya_ss/sd-scripts/train_db.py", line 190, in train
    train_dataloader = torch.utils.data.DataLoader(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 350, in __init__
    sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/sampler.py", line 143, in __init__
    raise ValueError(f"num_samples should be a positive integer value, but got num_samples={self.num_samples}")
ValueError: num_samples should be a positive integer value, but got num_samples=0
Traceback (most recent call last):
  File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1017, in launch_command
    simple_launcher(args)
  File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 637, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python3', '/workspace/kohya_ss/sd-scripts/train_db.py', '--config_file', '/workspace/kohya_ss/outputs/model/config_dreambooth-20240721-213429.toml']' returned non-zero exit status 1.

Not sure what I am doing wrong, worked before...

This happens for both Lora and Model training.

[popovidis]

12:43:13-616120 INFO     Start training Dreambooth...                           
12:43:13-622362 INFO     Validating lr scheduler arguments...                   
12:43:13-625454 INFO     Validating optimizer arguments...                      
12:43:13-630413 INFO     Validating /workspace/kohya_ss/outputs/log existence   
                         and writability... SUCCESS                             
12:43:13-633533 INFO     Validating /workspace/kohya_ss/outputs/model existence 
                         and writability... SUCCESS                             
12:43:13-635718 INFO     Validating runwayml/stable-diffusion-v1-5 existence... 
                         SKIPPING: huggingface.co model                         
12:43:13-639497 INFO     Validating /workspace/kohya_ss/outputs/img existence...
                         SUCCESS                                                
12:43:13-644170 INFO     Folder 40_nnjl aesthetic style: 40 repeats found       
12:43:13-647465 INFO     Folder 40_nnjl aesthetic style: 0 images found         
12:43:13-649391 INFO     Folder 40_nnjl aesthetic style: 0 * 40 = 0 steps       
12:43:13-651204 INFO     Regulatization factor: 1                               
12:43:13-654225 INFO     Total steps: 0                                         
12:43:13-656585 INFO     Train batch size: 1                                    
12:43:13-658860 INFO     Gradient accumulation steps: 1                         
12:43:13-661399 INFO     Epoch: 10                                              
12:43:13-664664 INFO     Max train steps: 1600                                  
12:43:13-666684 INFO     lr_warmup_steps = 0                                    
12:43:13-683022 WARNING  Here is the trainer command as a reference. It will not
                         be executed:                                           
                                                                                
12:43:13-686054 INFO     /workspace/kohya_ss/venv/bin/accelerate launch         
                         --dynamo_backend no --dynamo_mode default              
                         --mixed_precision fp16 --num_processes 1 --num_machines
                         1 --num_cpu_threads_per_process 2                      
                         /workspace/kohya_ss/sd-scripts/train_db.py             
                         --config_file                                          
                         /workspace/kohya_ss/outputs/model/config_dreambooth-202
                         40722-124313.toml                                      

12:43:13-689859 INFO     Showing toml config file:                              
                         /workspace/kohya_ss/outputs/model/config_dreambooth-202
                         40722-124313.toml                                      

12:43:13-696744 INFO     bucket_no_upscale = true                               
                         bucket_reso_steps = 64                                 
                         cache_latents = true                                   
                         cache_latents_to_disk = true                           
                         caption_extension = ".txt"                             
                         clip_skip = 1                                          
                         dynamo_backend = "no"                                  
                         enable_bucket = true                                   
                         epoch = 10                                             
                         gradient_accumulation_steps = 1                        
                         gradient_checkpointing = true                          
                         huber_c = 0.1                                          
                         huber_schedule = "snr"                                 
                         learning_rate = 0.0003                                 
                         learning_rate_te = 1e-5                                
                         logging_dir = "/workspace/kohya_ss/outputs/log"        
                         loss_type = "l2"                                       
                         lr_scheduler = "constant"                              
                         lr_scheduler_args = []                                 
                         lr_scheduler_num_cycles = 1                            
                         lr_scheduler_power = 1                                 
                         max_bucket_reso = 2048                                 
                         max_data_loader_n_workers = 0                          
                         max_timestep = 1000                                    
                         max_token_length = 75                                  
                         max_train_steps = 1600                                 
                         min_bucket_reso = 256                                  
                         mixed_precision = "fp16"                               
                         multires_noise_discount = 0.3                          
                         noise_offset_type = "Original"                         
                         optimizer_args = [ "scale_parameter=False",            
                         "relative_step=False", "warmup_init=False",]           
                         optimizer_type = "Adafactor"                           
                         output_dir = "/workspace/kohya_ss/outputs/model"       
                         output_name = "nnjl-sd15-new-model"                    
                         pretrained_model_name_or_path =                        
                         "runwayml/stable-diffusion-v1-5"                       
                         prior_loss_weight = 1                                  
                         resolution = "1024,1024"                               
                         sample_prompts =                                       
                         "/workspace/kohya_ss/outputs/model/prompt.txt"         
                         sample_sampler = "euler_a"                             
                         save_every_n_epochs = 1                                
                         save_model_as = "safetensors"                          
                         save_precision = "fp16"                                
                         train_batch_size = 1                                   
                         train_data_dir = "/workspace/kohya_ss/outputs/img"     
                         xformers = true                                        
                                                                                
12:43:13-706015 INFO     end of toml config file:                               
                         /workspace/kohya_ss/outputs/model/config_dreambooth-202
                         40722-124313.toml

That is the training command

[TynClause]

@popovidis Based on your error message, ValueError: num_samples should be a positive integer value, but got num_samples=0 means the data is not loaded, so the num_samples is 0.

You can also check in your logs during Dreambooth training that 12:43:13-647465 INFO Folder 40_nnjl aesthetic style: 0 images found. Please check the dataset format again.

[popovidis]

@popovidis Based on your error message, ValueError: num_samples should be a positive integer value, but got num_samples=0 means the data is not loaded, so the num_samples is 0.

You can also check in your logs during Dreambooth training that 12:43:13-647465 INFO Folder 40_nnjl aesthetic style: 0 images found. Please check the dataset format again.

It was not empty. Trying over and over with small value tweaks somehow solved that issue...