Lycoris extreme slow training after update to latest version

[killerciao]

After updating to the latest release Lycoris/LoCon (the only one i tested) training in kohya_ss is super slow. With my 4090 with the same setting loaded, in the previous version was running at about 1it/s for SDXL training, now 4-10s/it.
Nothing has changed in the training data.
Uninstalling 2.2post3 and installing 2.1.0post2 fixes the problem

[killerciao]

V2.2post3

V2.1.0post2

[KohakuBlueleaf]

please provide full configuration.

[killerciao]

These are the setting used in both Versions

{
"LoRA_type": "LyCORIS/LoCon",
"LyCORIS_preset": "full",
"adaptive_noise_scale": 0,
"additional_parameters": "--max_grad_norm=0",
"block_alphas": "",
"block_dims": "",
"block_lr_zero_threshold": "",
"bucket_no_upscale": true,
"bucket_reso_steps": 64,
"bypass_mode": false,
"cache_latents": true,
"cache_latents_to_disk": true,
"caption_dropout_every_n_epochs": 0.0,
"caption_dropout_rate": 0,
"caption_extension": ".txt",
"clip_skip": "1",
"color_aug": false,
"constrain": 0.0,
"conv_alpha": 16,
"conv_block_alphas": "",
"conv_block_dims": "",
"conv_dim": 32,
"dataset_config": "",
"debiased_estimation_loss": false,
"decompose_both": false,
"dim_from_weights": false,
"dora_wd": false,
"down_lr_weight": "",
"enable_bucket": true,
"epoch": 6,
"factor": -1,
"flip_aug": false,
"fp8_base": false,
"full_bf16": false,
"full_fp16": false,
"gpu_ids": "",
"gradient_accumulation_steps": 1,
"gradient_checkpointing": false,
"keep_tokens": "0",
"learning_rate": 1.0,
"log_tracker_config": "",
"log_tracker_name": "",
"logging_dir": "H:/t00nstyle/log",
"lora_network_weights": "",
"lr_scheduler": "cosine",
"lr_scheduler_args": "",
"lr_scheduler_num_cycles": "",
"lr_scheduler_power": "",
"lr_warmup": 0,
"max_bucket_reso": 2048,
"max_data_loader_n_workers": "0",
"max_grad_norm": 1,
"max_resolution": "1024,1024",
"max_timestep": 1000,
"max_token_length": "225",
"max_train_epochs": "",
"max_train_steps": "",
"mem_eff_attn": false,
"mid_lr_weight": "",
"min_bucket_reso": 256,
"min_snr_gamma": 0,
"min_timestep": 0,
"mixed_precision": "bf16",
"model_list": "custom",
"module_dropout": 0,
"multi_gpu": false,
"multires_noise_discount": 0.3,
"multires_noise_iterations": 6,
"network_alpha": 16,
"network_dim": 32,
"network_dropout": 0,
"noise_offset": 0.0357,
"noise_offset_type": "Multires",
"num_cpu_threads_per_process": 2,
"num_machines": 1,
"num_processes": 1,
"optimizer": "Prodigy",
"optimizer_args": "decouple=True weight_decay=0.5 betas=0.9,0.99 use_bias_correction=False",
"output_dir": "H:/t00nstyle/model",
"output_name": "t00nstylev1PonySDXL",
"persistent_data_loader_workers": false,
"pretrained_model_name_or_path": "F:/IA FILES/Models/Stablediffusion/ponyDiffusionV6XL_v6StartWithThisOne.safetensors",
"prior_loss_weight": 1.0,
"random_crop": false,
"rank_dropout": 0,
"rank_dropout_scale": false,
"reg_data_dir": "",
"rescaled": false,
"resume": "",
"sample_every_n_epochs": 0,
"sample_every_n_steps": 250,
"sample_prompts": "score_9, score_8_up, score_7_up,flashing tits, nipples, looking at viewer, tongue out, wink, in pool, bikini, t00nstyle --n low quality, worst quality, bad anatomy,bad composition, poor, low effort --w 1024 --h 1024 --d 1 --l 7 --s 28",
"sample_sampler": "euler_a",
"save_every_n_epochs": 1,
"save_every_n_steps": 0,
"save_last_n_steps": 0,
"save_last_n_steps_state": 0,
"save_model_as": "safetensors",
"save_precision": "bf16",
"save_state": false,
"scale_v_pred_loss_like_noise_pred": false,
"scale_weight_norms": 1,
"sdxl": true,
"sdxl_cache_text_encoder_outputs": false,
"sdxl_no_half_vae": true,
"seed": "12345",
"shuffle_caption": true,
"stop_text_encoder_training_pct": 0,
"text_encoder_lr": 1.0,
"train_batch_size": 1,
"train_data_dir": "H:/t00nstyle/img",
"train_norm": false,
"train_on_input": false,
"training_comment": "",
"unet_lr": 1.0,
"unit": 1,
"up_lr_weight": "",
"use_cp": true,
"use_scalar": false,
"use_tucker": false,
"use_wandb": false,
"v2": false,
"v_parameterization": false,
"v_pred_like_loss": 0,
"vae": "",
"vae_batch_size": 0,
"wandb_api_key": "",
"wandb_run_name": "",
"weighted_captions": false,
"xformers": "xformers"
}

[KohakuBlueleaf]

What's your hardware

[killerciao]

What's your hardware

Win 11, Nvidia 4090, 64gb ddr5 ram, i9 13900k, running from nvme 4th gen

[KohakuBlueleaf]

I do some test and cannot reproduce this error...
But I'm using plain kohya-ss/sd-scripts
Will try to use the GUI to test.

I'm using almost same hardware (64GB -> 128GB ram) as yours so it should be ok

[whythisusername]

@KohakuBlueleaf
Since someone already reported this issue, can confirm something changed specifically after 2.2.0.dev7 update (tested prev and after versions) that drops the speed from ~1.5 it/s to ~1.5s/it with 4090 training using this config, its for easy-scripts

[KohakuBlueleaf]

@whythisusername Can you try drouput=0 (all kind of dropout)?
Maybe something with dropout goes wrong

[whythisusername]

@KohakuBlueleaf
Yeah, here is some speed measurements between two versions, dropout significantly affects the speed with the latest, still a little slower than old version without it though

• 2.2.0.dev7 with dropout: 1321/2500 [17:22<15:30, 1.27it/s, avr_loss=0.107]

• 2.2.0.dev7 no dropout: 587/2500 [07:32<24:34, 1.30it/s, avr_loss=0.105]

• 2.3.0.dev6 with dropout: 533/2500 [10:59<40:33, 1.24s/it, avr_loss=0.115]

• 2.3.0.dev6 no dropout: 721/2500 [10:31<25:57, 1.14it/s, avr_loss=0.0954]

[KohakuBlueleaf]

@whythisusername does 2.3.0.dev10 still so slow?

[whythisusername]

@KohakuBlueleaf
yes, haven't significantly changed

• 2.3.0.dev10 with dropout: 1021/2500 [20:14<29:18, 1.19s/it, avr_loss=0.102]

• 2.3.0.dev10 no dropout: 1174/2500 [17:35<19:52, 1.11it/s, avr_loss=0.0961]

[KohakuBlueleaf]

@whythisusername @killerciao Can you try 3.0.0.dev4?
I totally reconstruct the whole library structure.
Also avoid some redundant operation. I wonder if it will be better now

[killerciao]

@whythisusername @killerciao Can you try 3.0.0.dev4? I totally reconstruct the whole library structure. Also avoid some redundant operation. I wonder if it will be better now

Tested with latest kohya_ss gui:

• 4s/it 3.0.0dev4
• 1.25s/it 2.1.0post2

🤷‍♂️

[KohakuBlueleaf]

@whythisusername @killerciao Can you try 3.0.0.dev4? I totally reconstruct the whole library structure. Also avoid some redundant operation. I wonder if it will be better now

Tested with latest kohya_ss gui:

• 4s/it 3.0.0dev4
• 1.25s/it 2.1.0post2

🤷‍♂️

umm ok

I think it is due to some dtype things

[KohakuBlueleaf]

@whythisusername @killerciao Can you try 3.0.0.dev4? I totally reconstruct the whole library structure. Also avoid some redundant operation. I wonder if it will be better now

Tested with latest kohya_ss gui:

• 4s/it 3.0.0dev4
• 1.25s/it 2.1.0post2

🤷‍♂️

can you try other algorithm?
LoKr in my env have same speed across different version

[killerciao]

LoKr:
1.3s/it 3.0.0dev4
Crash/ 2.1.0post2
Traceback (most recent call last): File "H:\kohya_ss\sd-scripts\sdxl_train_network.py", line 185, in <module> trainer.train(args) File "H:\kohya_ss\sd-scripts\train_network.py", line 864, in train noise_pred = self.call_unet( File "H:\kohya_ss\sd-scripts\sdxl_train_network.py", line 164, in call_unet noise_pred = unet(noisy_latents, timesteps, text_embedding, vector_embedding) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\accelerate\utils\operations.py", line 680, in forward return model_forward(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\accelerate\utils\operations.py", line 668, in __call__ return convert_to_fp32(self.model_forward(*args, **kwargs)) File "H:\kohya_ss\venv\lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast return func(*args, **kwargs) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 1104, in forward h = call_module(module, h, emb, context) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 1095, in call_module x = layer(x, context) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 750, in forward hidden_states = block(hidden_states, context=encoder_hidden_states, timestep=timestep) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 673, in forward output = self.forward_body(hidden_states, context, timestep) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 655, in forward_body hidden_states = self.ff(self.norm3(hidden_states)) + hidden_states File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 599, in forward hidden_states = module(hidden_states) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 577, in forward hidden_states, gate = self.proj(hidden_states).chunk(2, dim=-1) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\lycoris\modules\lokr.py", line 342, in forward self.org_module[0].weight.data.to(x.device, dtype=self.lokr_w1.dtype) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1695, in __getattr__ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'") AttributeError: 'LokrModule' object has no attribute 'lokr_w1'. Did you mean: 'lokr_w1_a'? steps: 0%| | 0/3588 [00:00<?, ?it/s] Traceback (most recent call last): File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "H:\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in <module> File "H:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main args.func(args) File "H:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command simple_launcher(args) File "H:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['H:\\kohya_ss\\venv\\Scripts\\python.exe', 'H:/kohya_ss/sd-scripts/sdxl_train_network.py', '--config_file', 'H:/alariko/model/config_lora-20240525-123304.toml', '--max_grad_norm=0']' returned non-zero exit status 1.

[KohakuBlueleaf]

LoKr: 1.3s/it 3.0.0dev4 Crash/ 2.1.0post2 Traceback (most recent call last): File "H:\kohya_ss\sd-scripts\sdxl_train_network.py", line 185, in <module> trainer.train(args) File "H:\kohya_ss\sd-scripts\train_network.py", line 864, in train noise_pred = self.call_unet( File "H:\kohya_ss\sd-scripts\sdxl_train_network.py", line 164, in call_unet noise_pred = unet(noisy_latents, timesteps, text_embedding, vector_embedding) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\accelerate\utils\operations.py", line 680, in forward return model_forward(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\accelerate\utils\operations.py", line 668, in __call__ return convert_to_fp32(self.model_forward(*args, **kwargs)) File "H:\kohya_ss\venv\lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast return func(*args, **kwargs) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 1104, in forward h = call_module(module, h, emb, context) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 1095, in call_module x = layer(x, context) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 750, in forward hidden_states = block(hidden_states, context=encoder_hidden_states, timestep=timestep) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 673, in forward output = self.forward_body(hidden_states, context, timestep) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 655, in forward_body hidden_states = self.ff(self.norm3(hidden_states)) + hidden_states File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 599, in forward hidden_states = module(hidden_states) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\sd-scripts\library\sdxl_original_unet.py", line 577, in forward hidden_states, gate = self.proj(hidden_states).chunk(2, dim=-1) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "H:\kohya_ss\venv\lib\site-packages\lycoris\modules\lokr.py", line 342, in forward self.org_module[0].weight.data.to(x.device, dtype=self.lokr_w1.dtype) File "H:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1695, in __getattr__ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'") AttributeError: 'LokrModule' object has no attribute 'lokr_w1'. Did you mean: 'lokr_w1_a'? steps: 0%| | 0/3588 [00:00<?, ?it/s] Traceback (most recent call last): File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "H:\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in <module> File "H:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main args.func(args) File "H:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command simple_launcher(args) File "H:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['H:\\kohya_ss\\venv\\Scripts\\python.exe', 'H:/kohya_ss/sd-scripts/sdxl_train_network.py', '--config_file', 'H:/alariko/model/config_lora-20240525-123304.toml', '--max_grad_norm=0']' returned non-zero exit status 1.

Thx, looks like locon have some bugs
lokr and loha may be fine

[whythisusername]

@KohakuBlueleaf
It's way more better now with my config, but still a little slower than it was with 2.2.0.dev7

• 3.0.0.dev4 with dropout: 1224/2500 [17:51<18:36, 1.14it/s, avr_loss=0.0945]
• 3.0.0.dev4 no dropout: 1246/2500 [17:51<17:58, 1.16it/s, avr_loss=0.0945]
• 2.2.0.dev7 with dropout: 1964/2500 [25:01<06:49, 1.31it/s, avr_loss=0.0993]

[KohakuBlueleaf]

@KohakuBlueleaf It's way more better now with my config, but still a little slower than it was with 2.2.0.dev7

• 3.0.0.dev4 with dropout: 1224/2500 [17:51<18:36, 1.14it/s, avr_loss=0.0945]
• 3.0.0.dev4 no dropout: 1246/2500 [17:51<17:58, 1.16it/s, avr_loss=0.0945]
• 2.2.0.dev7 with dropout: 1964/2500 [25:01<06:49, 1.31it/s, avr_loss=0.0993]

Can you try to enable "bypass_mode"?
--network_args "bypass_mode=True"

[whythisusername]

@KohakuBlueleaf
Almost on par with old performance now

• 1983/2500 [27:16<07:06, 1.21it/s, avr_loss=0.124]

[KohakuBlueleaf]

@KohakuBlueleaf Almost on par with old performance now

• 1983/2500 [27:16<07:06, 1.21it/s, avr_loss=0.124]

Ok I think the problem is solved
The reconstruction mode for LoCon is not fast in bp.
Just enable bypass_mode if you think the speed is slower than expectation (note: LoHa and LoKr with bypass mode will be slower than default)