It's working. Training LORA of the latest version of kohya_ss on AMD GPU,Ubuntu 22.04.2 LTS ,test on RX6800 ,sd1.5&sdxl

[tornado73]

ROCm 5.6.0 и 5.7.1 ,
Dependencies fixed

Change on
requirements.txt

Details

accelerate==0.23.0
albumentations==1.3.0
aiofiles==23.2.1
altair==4.2.2
dadaptation==3.1
diffusers[torch]==0.18.2
easygui==0.98.3
einops==0.6.0
fairscale==0.4.13
ftfy==6.1.1
gradio==3.36.1
huggingface-hub==0.15.1
keras==2.12.0

invisible-watermark==0.2.0
lion-pytorch==0.0.6
lycoris_lora==1.8.3
gradio==3.36.1; sys_platform == 'darwin'
gradio==3.36.1; sys_platform != 'darwin'
huggingface-hub==0.15.1; sys_platform == 'darwin'
huggingface-hub==0.15.1; sys_platform != 'darwin'

open-clip-torch==2.20.0
opencv-python==4.7.0.68
prodigyopt==1.0
pytorch-lightning==1.9.0
tensorflow-rocm==2.12.0.560
tensorboard==2.12.0 ; sys_platform != 'darwin'
tensorboard==2.12.0 ; sys_platform == 'darwin'
tensorflow==2.12.0; sys_platform != 'darwin'
rich==13.4.1
safetensors==0.3.1
timm==0.6.12
tk==0.1.0
toml==0.10.2
transformers==4.30.2
voluptuous==0.13.1
wandb==0.15.0

-e . # no_verify leave this to specify not checking this a verification stage

---

install `
tested on ubuntu Ubuntu 22.04.2 LTS + rx6800

git clone https://github.com/bmaltais/kohya_ss.git
cd kohya_ss
python -m venv venv
source venv/bin/activate
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm5.6
pip install --use-pep517 --upgrade -r requirements.txt
accelerate config

• This machine
• No distributed training
• no
• no
• all
• fp16

sudo apt install python3-tk
export HSA_OVERRIDE_GFX_VERSION=10.3.0
source venv/bin/activate
python kohya_gui.py "$@"

---

Change on gui.sh

#!/usr/bin/env bash
export HSA_OVERRIDE_GFX_VERSION=10.3.0
source venv/bin/activate
python kohya_gui.py "$@"

---

Customization files from previous versions are not suitable ,
this is my working sample, change it for yourself

LoRa training on SD1.5

.json

Details

"LoRA_type": "Standard",
"adaptive_noise_scale": 0,
"additional_parameters": "",
"block_alphas": "",
"block_dims": "",
"block_lr_zero_threshold": "",
"bucket_no_upscale": true,
"bucket_reso_steps": 64,
"cache_latents": true,
"cache_latents_to_disk": false,
"caption_dropout_every_n_epochs": 0.0,
"caption_dropout_rate": 0,
"caption_extension": ".txt",
"clip_skip": 2,
"color_aug": false,
"conv_alpha": 1,
"conv_block_alphas": "",
"conv_block_dims": "",
"conv_dim": 1,
"decompose_both": false,
"dim_from_weights": false,
"down_lr_weight": "",
"enable_bucket": false,
"epoch": 1,
"factor": -1,
"flip_aug": false,
"full_bf16": false,
"full_fp16": false,
"gradient_accumulation_steps": "1",
"gradient_checkpointing": false,
"keep_tokens": "0",
"learning_rate": 0.0001,
"logging_dir": "/home/tor/kohya_ss/LORA/log",
"lora_network_weights": "",
"lr_scheduler": "constant",
"lr_scheduler_args": "",
"lr_scheduler_num_cycles": "",
"lr_scheduler_power": "",
"lr_warmup": 0,
"max_bucket_reso": 2048,
"max_data_loader_n_workers": "1",
"max_resolution": "768,768",
"max_timestep": 1000,
"max_token_length": "75",
"max_train_epochs": "",
"max_train_steps": "",
"mem_eff_attn": true,
"mid_lr_weight": "",
"min_bucket_reso": 256,
"min_snr_gamma": 0,
"min_timestep": 0,
"mixed_precision": "fp16",
"model_list": "custom",
"module_dropout": 0,
"multires_noise_discount": 0,
"multires_noise_iterations": 0,
"network_alpha": 128,
"network_dim": 256,
"network_dropout": 0,
"no_token_padding": false,
"noise_offset": 0,
"noise_offset_type": "Original",
"num_cpu_threads_per_process": 2,
"optimizer": "AdamW",
"optimizer_args": "",
"output_dir": "/home/tor/kohya_ss/LORA/model",
"output_name": "dzetaA4_80_",
"persistent_data_loader_workers": false,
"pretrained_model_name_or_path": "/home/tor/kohya_ss/model/absolutereality_v181.safetensors",
"prior_loss_weight": 1.0,
"random_crop": false,
"rank_dropout": 0,
"reg_data_dir": "",
"resume": "",
"sample_every_n_epochs": 0,
"sample_every_n_steps": 0,
"sample_prompts": "",
"sample_sampler": "euler_a",
"save_every_n_epochs": 1,
"save_every_n_steps": 0,
"save_last_n_steps": 0,
"save_last_n_steps_state": 0,
"save_model_as": "safetensors",
"save_precision": "fp16",
"save_state": false,
"scale_v_pred_loss_like_noise_pred": false,
"scale_weight_norms": 0,
"sdxl": false,
"sdxl_cache_text_encoder_outputs": false,
"sdxl_no_half_vae": true,
"seed": "",
"shuffle_caption": false,
"stop_text_encoder_training": 0,
"text_encoder_lr": 0.0004,
"train_batch_size": 1,
"train_data_dir": "/home/tor/kohya_ss/LORA/img",
"train_on_input": true,
"training_comment": "",
"unet_lr": 0.0001,
"unit": 1,
"up_lr_weight": "",
"use_cp": false,
"use_wandb": false,
"v2": false,
"v_parameterization": false,
"v_pred_like_loss": 0,
"vae_batch_size": 0,
"wandb_api_key": "",
"weighted_captions": false,
"xformers": "none"

---

LoRa training on sdxl
.json

Details

"LoRA_type": "Standard",
"adaptive_noise_scale": 0,
"additional_parameters": "",
"block_alphas": "",
"block_dims": "",
"block_lr_zero_threshold": "",
"bucket_no_upscale": false,
"bucket_reso_steps": 64,
"cache_latents": true,
"cache_latents_to_disk": true,
"caption_dropout_every_n_epochs": 0.0,
"caption_dropout_rate": 0.05,
"caption_extension": ".txt",
"clip_skip": "1",
"color_aug": false,
"conv_alpha": 1,
"conv_block_alphas": "",
"conv_block_dims": "",
"conv_dim": 1,
"decompose_both": false,
"dim_from_weights": false,
"down_lr_weight": "",
"enable_bucket": false,
"epoch": 50,
"factor": -1,
"flip_aug": false,
"full_bf16": false,
"full_fp16": false,
"gradient_accumulation_steps": "1",
"gradient_checkpointing": true,
"keep_tokens": "0",
"learning_rate": 3e-05,
"logging_dir": "/home/tor/kohya_ss/LORA/log",
"lora_network_weights": "",
"lr_scheduler": "constant",
"lr_scheduler_args": "",
"lr_scheduler_num_cycles": "",
"lr_scheduler_power": "",
"lr_warmup": 0,
"max_bucket_reso": 2048,
"max_data_loader_n_workers": "0",
"max_resolution": "1024,1024",
"max_timestep": 1000,
"max_token_length": "75",
"max_train_epochs": "50",
"max_train_steps": "",
"mem_eff_attn": true,
"mid_lr_weight": "",
"min_bucket_reso": 256,
"min_snr_gamma": 5,
"min_timestep": 0,
"mixed_precision": "fp16",
"model_list": "custom",
"module_dropout": 0,
"multires_noise_discount": 0,
"multires_noise_iterations": 0,
"network_alpha": 32,
"network_dim": 32,
"network_dropout": 0,
"no_token_padding": false,
"noise_offset": 0,
"noise_offset_type": "Original",
"num_cpu_threads_per_process": 2,
"optimizer": "AdamW",
"optimizer_args": "",
"output_dir": "/home/tor/kohya_ss/LORA/model",
"output_name": "dzetaA4xl",
"persistent_data_loader_workers": false,
"pretrained_model_name_or_path": "/home/tor/kohya_ss/model/sdXL_v10VAEFix.safetensors",
"prior_loss_weight": 1.0,
"random_crop": false,
"rank_dropout": 0,
"reg_data_dir": "",
"resume": "",
"sample_every_n_epochs": 0,
"sample_every_n_steps": 0,
"sample_prompts": "",
"sample_sampler": "euler_a",
"save_every_n_epochs": 1,
"save_every_n_steps": 0,
"save_last_n_steps": 0,
"save_last_n_steps_state": 0,
"save_model_as": "safetensors",
"save_precision": "fp16",
"save_state": false,
"scale_v_pred_loss_like_noise_pred": false,
"scale_weight_norms": 0,
"sdxl": true,
"sdxl_cache_text_encoder_outputs": false,
"sdxl_no_half_vae": true,
"seed": "",
"shuffle_caption": false,
"stop_text_encoder_training": 0,
"text_encoder_lr": 3e-05,
"train_batch_size": 3,
"train_data_dir": "/home/tor/kohya_ss/LORA/img",
"train_on_input": true,
"training_comment": "3 repeats. More info: https://civitai.com/articles/1771",
"unet_lr": 3e-05,
"unit": 1,
"up_lr_weight": "",
"use_cp": false,
"use_wandb": false,
"v2": false,
"v_parameterization": false,
"v_pred_like_loss": 0,
"vae_batch_size": 0,
"wandb_api_key": "",
"weighted_captions": false,
"xformers": "none"

Good luck

[tornado73]

sd1.5

sdxl

[shssoichiro]

Thanks, this has been driving me nuts however as I get a segfault when trying to run Lora training. I'm on a 7900 XTX, I went through all setup as you and many others have shown, yet getting this segfault:

EDIT: I got past the original segfault by changing to export HSA_OVERRIDE_GFX_VERSION=11.0.0, this part seems very important for 7000-series. However now I'm just getting a segfault further down in the process 🤦 Below is the new segfault location. What's interesting is a1111's UI works out of the box with no special pytorch or variable exports (for generating images, haven't tried training with it), but kohya requires all these extra steps and is still segfaulting.

create LoRA network. base dim (rank): 128, alpha: 64.0
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder 1:
create LoRA for Text Encoder 2:
create LoRA for Text Encoder: 264 modules.
create LoRA for U-Net: 722 modules.
enable LoRA for text encoder
enable LoRA for U-Net
prepare optimizer, data loader etc.
use AdamW optimizer | {}
running training / 学習開始  num train images * repeats / 学習画像の数×繰り返し回数: 408
  num reg images / 正則化画像の数: 408
  num batches per epoch / 1epochのバッチ数: 816
  num epochs / epoch数: 5
  batch size per device / バッチサイズ: 1
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 4080
steps:   0%|                                                                                        | 0/4080 [00:00<?, ?it/s]
epoch 1/5
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/soichiro/build/kohya_ss/venv/bin/accelerate:8 in <module>                                  │
│                                                                                                  │
│   5 from accelerate.commands.accelerate_cli import main                                          │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│                                                                                                  │
│ /home/soichiro/build/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_c │
│ li.py:45 in main                                                                                 │
│                                                                                                  │
│   42 │   │   exit(1)                                                                             │
│   43 │                                                                                           │
│   44 │   # Run                                                                                   │
│ ❱ 45 │   args.func(args)                                                                         │
│   46                                                                                             │
│   47                                                                                             │
│   48 if __name__ == "__main__":                                                                  │
│                                                                                                  │
│ /home/soichiro/build/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py:91 │
│ 8 in launch_command                                                                              │
│                                                                                                  │
│   915 │   elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA   │
│   916 │   │   sagemaker_launcher(defaults, args)                                                 │
│   917 │   else:                                                                                  │
│ ❱ 918 │   │   simple_launcher(args)                                                              │
│   919                                                                                            │
│   920                                                                                            │
│   921 def main():                                                                                │
│                                                                                                  │
│ /home/soichiro/build/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py:58 │
│ 0 in simple_launcher                                                                             │
│                                                                                                  │
│   577 │   process.wait()                                                                         │
│   578 │   if process.returncode != 0:                                                            │
│   579 │   │   if not args.quiet:                                                                 │
│ ❱ 580 │   │   │   raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)    │
│   581 │   │   else:                                                                              │
│   582 │   │   │   sys.exit(1)                                                                    │
│   583                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['/home/soichiro/build/kohya_ss/venv/bin/python3.10', './sdxl_train_network.py', 
'--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', 
'--pretrained_model_name_or_path=/home/soichiro/build/stable-diffusion-webui/models/Stable-diffusion/SD_XL/RealitiesEdgeXL_20
.safetensors', '--train_data_dir=/home/soichiro/Pictures/training/asdf/img/', 
'--reg_data_dir=/home/soichiro/Pictures/training/asdf/reg/', '--resolution=1024,1024', 
'--output_dir=/home/soichiro/Pictures/training/asdf/model/', 
'--logging_dir=/home/soichiro/Pictures/training/asdf/log/', '--network_alpha=64', '--save_model_as=safetensors', 
'--network_module=networks.lora', '--text_encoder_lr=5e-05', '--unet_lr=0.0001', '--network_dim=128', 
'--output_name=TESTING_v1', '--lr_scheduler_num_cycles=5', '--no_half_vae', '--learning_rate=0.0001', 
'--lr_scheduler=cosine_with_restarts', '--lr_warmup_steps=408', '--train_batch_size=1', '--max_train_steps=4080', 
'--save_every_n_epochs=1', '--mixed_precision=fp16', '--save_precision=fp16', '--caption_extension=.txt', 
'--optimizer_type=AdamW', '--max_data_loader_n_workers=0', '--caption_dropout_rate=0.1', '--bucket_reso_steps=64', 
'--shuffle_caption', '--gradient_checkpointing', '--sdpa', '--bucket_no_upscale', '--noise_offset=0.1', 
'--sample_sampler=euler_a', '--sample_prompts=/home/soichiro/Pictures/training/asdf/model/sample/prompt.txt', 
'--sample_every_n_steps=25']' died with <Signals.SIGSEGV: 11>.

[tornado73]

I attached the contents of the fil requirements.txt
,you have CUDA installed during installation
do step by step as I wrote
DO NOT run immediately gui.sh

Models, unfortunately, curves are obtained -(((
I didn't have much time to check, but three SD 1.5 models trained on the latest version give an absolutely inadequate result when generated
+
Learning takes place three times slower

Unfortunately, the latest version that trains normally, in our case for 1.5 models, is 21.7.10

[shssoichiro]

Thanks, I went back and re-did the full process and it is working now, only having to change HSA_OVERRIDE_GFX_VERSION to 11.0.0 for 7000-series. Although for SDXL it is using a very large amount of VRAM even with cache latents disabled. All 24 GB to the point of locking up my machine. Other users seem to have reported as little as 12 GB works with SDXL but that is with nvidia+xformers sadly.

[tornado73]

strange, but I had 16 GB to work without crashes

[royallife88]

Hi,
@tornado73 did everything as you write the instructions above. But training still running on CPU that have a huge time. So what is the problem here that it's not using GPU (RX 5500 XT).
Thanks.

[2blackbar]

I think it installs torch cpu version, installed needs to be checked very carefuly by the dev of this repo, he has all this installed since the start so he doesnt test if its working when installing from 0

[ngovandang]

Linux only?

[tornado73]

Только линукс?
yes ,only Linux

использует графический процессор (RX 5500 XT)?
AMD Radeon RX 5500 XT based on RDNA 1.0 architecture

https://rocm.docs.amd.com/en/latest/release/gpu_os_support.html

unfortunately not supported-(
teach on google colab,

[FrostyForest]

6700xt，Ubuntu，ROCm5.3 successfully train a sdxl lora too.

[Charmandrigo]

has anyone tested rocm 5.7?

[tornado73]

Yes it works
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm5.7

[danielaixer]

When I try to train a Dreambooth LoRA I get ModuleNotFoundError: No module named 'xformers'

I'm on Ubuntu 22.04, 7900XTX and I can generate images on Stable Diffusion using the GPU.

Install commands:

sudo apt-get install -y python3-tk
rm -rf kohya_ss
git clone https://github.com/bmaltais/kohya_ss.git 
cd kohya_ss
python -m venv venv
source venv/bin/activate
# This is the same I installed for SD
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.6
pip install --use-pep517 --upgrade -r requirements.txt
accelerate config

requirements.txt contains exactly what was shown here: #1484 (comment)

Run commands:

export HSA_OVERRIDE_GFX_VERSION=11.0.0
source venv/bin/activate
python kohya_gui.py --server_port 7862 --listen 0.0.0.0

I'm pretty sure xFormers for kohya_ss is mandatory, but xFormers doesn't support AMD GPUs... facebookresearch/xformers#807 (comment)

Edit: I got rid of the xFormers error by setting "Parameters > Basic > CrossAttention" to "none", but it still doesn't work and it looks like it's not detecting or using the GPU, as I get messages like:

2023-12-20 19:18:42.944527: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.

And also:
ModuleNotFoundError: No module named 'bitsandbytes'

Which I fixed with:

source venv/bin/activate
pip install bitsandbytes

And now I get
AttributeError: 'NoneType' object has no attribute 'split'

But even if I fixed that, it's still not using the GPU

Edit 2: Using the optimizer AdamW (which doesn't require bitsandbytes) instead of AdamW8bit gives this other error:
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'

Edit 3: the above issue is fixed with "Mixed precision" set to "no" on the UI parameters. I can train LoRAs now.