[Error] 'device cuda is invalid'

[zakinp]

This error occur when using the lastest version of these scripts. The messages are:

Traceback (most recent call last):
File "/root/autodl-tmp/sd-scripts/train_network.py", line 719, in
train(args)
File "/root/autodl-tmp/sd-scripts/train_network.py", line 134, in train
text_encoder, vae, unet, _ = train_util.load_target_model(args, weight_dtype, accelerator.device)
File "/root/autodl-tmp/sd-scripts/library/train_util.py", line 2649, in load_target_model
text_encoder, vae, unet = model_util.load_models_from_stable_diffusion_checkpoint(args.v2, name_or_path, device)
File "/root/autodl-tmp/sd-scripts/library/model_util.py", line 869, in load_models_from_stable_diffusion_checkpoint
_, state_dict = load_checkpoint_with_text_encoder_conversion(ckpt_path, device)
File "/root/autodl-tmp/sd-scripts/library/model_util.py", line 844, in load_checkpoint_with_text_encoder_conversion
state_dict = load_file(ckpt_path, device)
File "/root/miniconda3/envs/kohya/lib/python3.10/site-packages/safetensors/torch.py", line 98, in load_file
with safe_open(filename, framework="pt", device=device) as f:
Exception: device cuda is invalid
Traceback (most recent call last):
File "/root/miniconda3/envs/kohya/bin/accelerate", line 8, in
sys.exit(main())
File "/root/miniconda3/envs/kohya/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
args.func(args)
File "/root/miniconda3/envs/kohya/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1104, in launch_command
simple_launcher(args)
File "/root/miniconda3/envs/kohya/lib/python3.10/site-packages/accelerate/commands/launch.py", line 567, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/root/miniconda3/envs/kohya/bin/python', '/root/autodl-tmp/sd-scripts/train_network.py', '--pretrained_model_name_or_path=/root/autodl-tmp/anything-v45-pruned.safetensors', '--output_dir=/root/autodl-tmp/chichi-v1fix-lora-32dim-10ep-novae-any45base', '--output_name=chichi-v1fix-lora-32dim-10ep-novae-any45base', '--save_every_n_epochs=1', '--train_batch_size=4', '--max_token_length=150', '--learning_rate=1', '--unet_lr=1', '--text_encoder_lr=0.5', '--optimizer_type=DAdaptation', '--max_train_epochs=10', '--gradient_accumulation_steps=1', '--mixed_precision=bf16', '--save_precision=fp16', '--xformers', '--clip_skip=2', '--logging_dir=logs', '--lr_scheduler=cosine_with_restarts', '--train_data_dir=/root/autodl-tmp/loradata/image_dir', '--shuffle_caption', '--caption_extension=.txt', '--keep_tokens=1', '--resolution=768,768', '--enable_bucket', '--noise_offset=0.05', '--min_snr_gamma=5', '--cache_latents', '--vae_batch_size=4', '--persistent_data_loader_workers', '--max_data_loader_n_workers=4', '--reg_data_dir=/root/autodl-tmp/loradata/reg_dir', '--save_model_as=safetensors', '--network_module=networks.lora', '--network_dim=32', '--network_alpha=16']' returned non-zero exit status 1.

[Coloured-glaze]

I also ran into this problem. (Exception: device cuda:0 is invalid )

[Coloured-glaze]

I also ran into this problem. (Exception: device cuda:0 is invalid )

the latest commit seems to have fixed this issue and I have been able to run the script normally.

[kohya-ss]

PR #357 fixed this issue :)