Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Exception training model: [WinError 5] Access is denied #1371

Closed
1 task done
klubkelli opened this issue Oct 9, 2023 · 10 comments
Closed
1 task done

[Bug]: Exception training model: [WinError 5] Access is denied #1371

klubkelli opened this issue Oct 9, 2023 · 10 comments

Comments

@klubkelli
Copy link

Is there an existing issue for this?

  • I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

When attempting to train a new model (or Lora) it fails with the error Exception training model: '[WinError 5] Access is denied: '

I'm on a windows machine with a Nvidia graphics card.

Steps to reproduce the problem

  1. Go to .... Dreambooth tab
  2. Press .... attempt to create a model or lora (any kind any steps)
  3. .That's it. There is no step three. IT fails.

Commit and libraries

Initializing Dreambooth
Dreambooth revision: 1a1d162
[!] xformers NOT installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.23.0 installed.
[+] diffusers version 0.21.4 installed.
[+] transformers version 4.32.1 installed.
[+] bitsandbytes version 0.41.1 installed.

Launching Web UI with arguments:
No module 'xformers'. Proceeding without it.
[2023-10-09 00:24:16,878][DEBUG][git.cmd] - Popen(['git', 'version'], cwd=D:\stable-diffusion-webui, universal_newlines=False, shell=None, istream=None)
[2023-10-09 00:24:16,939][DEBUG][git.cmd] - Popen(['git', 'version'], cwd=D:\stable-diffusion-webui, universal_newlines=False, shell=None, istream=None)
[2023-10-09 00:24:17,458][DEBUG][filelock] - Attempting to acquire lock 2169171614000 on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:17,459][DEBUG][filelock] - Lock 2169171614000 acquired on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:17,459][DEBUG][filelock] - Attempting to release lock 2169171614000 on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:17,460][DEBUG][filelock] - Lock 2169171614000 released on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:18,332][DEBUG][api.py] - SD-Webui API layer loaded
Loading weights [4199bcdd14] from D:\stable-diffusion-webui\models\Stable-diffusion\revAnimated_v122.safetensors
[2023-10-09 00:24:18,846][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.reference] - entering reference: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.html_block] - entering html_block: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.heading] - entering heading: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.lheading] - entering lheading: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,850][DEBUG][markdown_it.rules_block.paragraph] - entering paragraph: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,926][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,927][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.reference] - entering reference: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.html_block] - entering html_block: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.heading] - entering heading: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.lheading] - entering lheading: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.paragraph] - entering paragraph: StateBlock(line=0,level=0,tokens=0), 0, 1, False
False

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
Creating model from config: D:\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
DiffusionWrapper has 859.52 M params.
[2023-10-09 00:24:19,373][DEBUG][git.cmd] - Popen(['git', 'remote', 'get-url', '--all', 'origin'], cwd=D:\stable-diffusion-webui\extensions\sd_dreambooth_extension, universal_newlines=False, shell=None, istream=None)
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
[2023-10-09 00:24:19,645][DEBUG][api.py] - Loading Dreambooth API Endpoints.
Startup time: 5.4s (import torch: 1.1s, import gradio: 0.5s, import ldm: 0.3s, other imports: 1.3s, load scripts: 1.2s, create ui: 0.7s, gradio launch: 0.2s).
preload_extensions_git_metadata for 8 extensions took 0.56s
Applying attention optimization: Doggettx... done.
Textual inversion embeddings loaded(0):
Model loaded in 3.4s (load weights from disk: 0.4s, create model: 0.8s, apply weights to model: 0.8s, apply half(): 0.7s, move model to device: 0.7s).
[2023-10-09 00:25:40,801][DEBUG][git.cmd] - Popen(['git', 'fetch', '-v', '--dry-run', '--', 'origin'], cwd=D:\stable-diffusion-webui\extensions\sd_dreambooth_extension, universal_newlines=True, shell=None, istream=None)
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:05<00:00, 3.92it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:03<00:00, 5.80it/s]
[2023-10-09 00:27:46,901][DEBUG][dreambooth.dataclasses.db_config] - Saving to D:\stable-diffusion-webui\models\dreambooth\GemmaHiles
Custom model name is gemmahiles1
[2023-10-09 00:27:48,022][DEBUG][dreambooth.dataclasses.db_config] - Saving to D:\stable-diffusion-webui\models\dreambooth\GemmaHiles
Initializing dreambooth training...
[2023-10-09 00:27:48,108][DEBUG][dreambooth.train_dreambooth] - Adding 'get_velocity' method to DEISMultistepScheduler...
[2023-10-09 00:27:48,108][DEBUG][dreambooth.train_dreambooth] - Adding 'get_velocity' method to UniPCMultistepScheduler...
[2023-10-09 00:27:48,258][DEBUG][dreambooth.train_dreambooth] - Pretrained path: D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\working
[2023-10-09 00:27:48,455][INFO][dreambooth.utils.gen_utils] - Nothing to generate. | 12/16 [00:00<00:00, 113.51it/s]

===================================BUG REPORT=================================== | 0/1 [00:00<?, ?it/s]

The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
[2023-10-09 00:29:27,405][WARNING][dreambooth.optimization] - Exception importing 8bit AdamW:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

Traceback (most recent call last):
File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\optimization.py", line 594, in get_optimizer
from bitsandbytes.optim import AdamW8bit
File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes_init_.py", line 6, in
from . import cuda_setup, utils, research
File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research_init_.py", line 1, in
from . import nn
File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research\nn_init_.py", line 1, in
from .modules import LinearFP8Mixed, LinearFP8Global
File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in
from bitsandbytes.optim import GlobalOptimManager
File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\optim_init_.py", line 6, in
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\cextension.py", line 20, in
raise RuntimeError('''
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

    CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

WARNING: Using default optimizer (AdamW from Torch)
[2023-10-09 00:29:27,553][DEBUG][dreambooth.utils.gen_utils] - Found 0 reg images. | 0/1 [00:00<?, ?it/s]
[2023-10-09 00:29:27,554][DEBUG][dreambooth.utils.gen_utils] - Preparing dataset...
Init dataset!
Preparing Dataset (With Caching)
Loading latent cache...h Caching): 0%| | 0/1 [00:00<?, ?it/s]
Bucket 0 (512, 512, 0) - Instance Images: 16 | Class Images: 0 | Max Examples/batch: 16
Saving cache!ed latents...: 100%|███████████████████████████████████████████████████████████████| 16/16 [00:00<?, ?it/s]
[2023-10-09 00:29:27,579][ERROR][dreambooth.dataset.db_dataset] - Error saving cache!
Traceback (most recent call last):
File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\dataset\db_dataset.py", line 165, in save_cache_file
os.remove(cache_file)
PermissionError: [WinError 5] Access is denied: 'D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\cache\cache_512.safetensors'
Total Buckets 1 - Instance Images: 16 | Class Images: 0 | Max Examples/batch: 16

Total images / batch: 16, total examples: 16█████████████████████████████████████████| 16/16 [00:00<00:00, 1069.58it/s]
[2023-10-09 00:29:27,594][DEBUG][dreambooth.utils.gen_utils] - Total dataset length (steps): 16
Initializing bucket counter!
[2023-10-09 00:29:27,795][DEBUG][dreambooth.train_dreambooth] - ***** Running training *****
[2023-10-09 00:29:27,795][DEBUG][dreambooth.train_dreambooth] - Num batches each epoch = 4
[2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] - Num Epochs = 2
[2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] - Batch Size Per Device = 4
[2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] - Gradient Accumulation steps = 1
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] - Total train batch size (w. parallel, distributed & accumulation) = 4
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] - Text Encoder Epochs: 2
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] - Total optimization steps = 32
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] - Total training steps = 32
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] - Resuming from checkpoint: False
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - First resume epoch: 0
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - First resume step: 0
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - Lora: True, Optimizer: 8bit AdamW, Prec: fp16
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - Gradient Checkpointing: True
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - EMA: True
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] - UNET: True
[2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] - Freeze CLIP Normalization Layers: False
[2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] - LR (Lora): 0.0001
[2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] - Tenc LR (Lora): 5e-05
[2023-10-09 00:29:27,800][DEBUG][dreambooth.train_dreambooth] - V2: False
Steps: 0%| | 0/32 [00:00<?, ?it/s]D:\stable-diffusion-webui\venv\lib\site-packages\diffusers\models\attention_processor.py:1468: FutureWarning: LoRAAttnProcessor2_0 is deprecated and will be removed in version 0.26.0. Make sure use AttnProcessor2_0 instead by settingLoRA layers to self.{to_q,to_k,to_v,to_out[0]}.lora_layer respectively. This will be done automatically when using LoraLoaderMixin.load_lora_weights
deprecate(
Steps: 100%|█████████████████████████████████████████| 32/32 [00:14<00:00, 3.07it/s, loss=0.0995, lr=6.6e-6, vram=5.8][2023-10-09 00:29:42,182][DEBUG][dreambooth.train_dreambooth] -
Save completed/canceled.
[2023-10-09 00:29:42,182][DEBUG][dreambooth.train_dreambooth] - Completed, enabling saves.
D:\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead.
warnings.warn(
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:08<00:00, 1.26s/it]
[2023-10-09 00:29:51,704][DEBUG][dreambooth.utils.model_utils] - Enabled AttnProcessor2_0 for UNet2DConditionModel
[2023-10-09 00:29:51,706][DEBUG][dreambooth.utils.model_utils] - Enabled AttnProcessor2_0 for AutoencoderKL, 1.75s/it]
[2023-10-09 00:29:51,706][DEBUG][dreambooth.train_dreambooth] - Save image is set.
[2023-10-09 00:29:51,707][DEBUG][dreambooth.train_dreambooth] - Loras dir: D:\stable-diffusion-webui\models\Lora
[2023-10-09 00:29:51,707][DEBUG][dreambooth.train_dreambooth] - Saving lora to D:\stable-diffusion-webui\models\Lora\gemmahiles1_32.safetensors
[2023-10-09 00:29:51,839][DEBUG][dreambooth.train_dreambooth] - Saving images...
[2023-10-09 00:29:51,891][DEBUG][dreambooth.train_dreambooth] - Loading image pipeline from D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\working...
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:02<00:00, 2.38it/s]
[2023-10-09 00:29:55,623][DEBUG][dreambooth.train_dreambooth] - Generating 0 samples...
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:02<00:00, 2.32it/s]Model name: GemmaHiles 0%| | 0/2 [00:00<?, ?it/s]
D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\helpers\log_parser.py:324: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.
all_df_loss = all_df_loss.fillna(method="ffill")
Saving D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\logging\loss_plot_0.png
Saving D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\logging\ram_plot_0.png
Cleanup log parse.
Traceback (most recent call last):████████████████████████ | 1/2 [00:00<00:00, 3.00it/s]
File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 422, in run_predict
output = await app.get_blocks().process_api(
File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1326, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1251, in postprocess_data
prediction_value = postprocess_update_dict(
File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 434, in postprocess_update_dict
prediction_value["value"] = block.postprocess(prediction_value["value"])
File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\components.py", line 4461, in postprocess
file = self.pil_to_temp_file(img, dir=self.DEFAULT_TEMP_DIR)
File "D:\stable-diffusion-webui\modules\ui_tempdir.py", line 55, in save_pil_to_file
file_obj = tempfile.NamedTemporaryFile(delete=False, suffix=".png", dir=dir)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 559, in NamedTemporaryFile
file = _io.open(dir, mode, buffering=buffering,
File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 556, in opener
fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 256, in _mkstemp_inner
fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\Admin\AppData\Local\Temp\gradio\tmp0o0la6h4.png'
Traceback (most recent call last):
File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\ui_functions.py", line 730, in start_training
result = main(class_gen_method=class_gen_method)
File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1809, in main
return inner_loop()
File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\memory.py", line 126, in decorator
return function(batch_size, grad_size, prof, *args, **kwargs)
File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1766, in inner_loop
check_save(True)
File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 976, in check_save
save_weights(
File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1380, in save_weights
convert_diffusers_to_kohya_lora(lora_save_file, meta, args.lora_weight)
File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\diff_lora_to_sd_lora.py", line 120, in convert_diffusers_to_kohya_lora
os.remove(path)
PermissionError: [WinError 5] Access is denied: 'D:\stable-diffusion-webui\models\Lora\gemmahiles1_32.safetensors'
Steps: 100%|█████████████████████████████████████████| 32/32 [00:28<00:00, 1.10it/s, loss=0.0995, lr=6.6e-6, vram=5.8]
Duration: 00:02:09
Generating Samples: 100%|████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00, 1.55it/s]
[2023-10-09 00:29:57,795][DEBUG][dreambooth.utils.model_utils] - Restored system models.
Duration: 00:02:10

Command Line Arguments

git pull
@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=

call webui.bat

Console logs

D:\stable-diffusion-webui>git pull
error: Your local changes to the following files would be overwritten by merge:
        requirements.txt
        requirements_versions.txt
Please commit your changes or stash them before you merge.
Aborting
Updating f865d3e1..5ef669de
venv "D:\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.4.1
Commit hash: f865d3e11647dfd6c7b2cdf90dde24680e58acd8
Installing requirements
No module 'xformers'. Proceeding without it.
If submitting an issue on github, please provide the full startup log for debugging purposes.

Initializing Dreambooth
Dreambooth revision: 1a1d1621086a4725fda1200256f319c845dc7a8a
[!] xformers NOT installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.23.0 installed.
[+] diffusers version 0.21.4 installed.
[+] transformers version 4.32.1 installed.
[+] bitsandbytes version 0.41.1 installed.

Launching Web UI with arguments:
No module 'xformers'. Proceeding without it.
[2023-10-09 00:24:16,878][DEBUG][git.cmd] - Popen(['git', 'version'], cwd=D:\stable-diffusion-webui, universal_newlines=False, shell=None, istream=None)
[2023-10-09 00:24:16,939][DEBUG][git.cmd] - Popen(['git', 'version'], cwd=D:\stable-diffusion-webui, universal_newlines=False, shell=None, istream=None)
[2023-10-09 00:24:17,458][DEBUG][filelock] - Attempting to acquire lock 2169171614000 on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:17,459][DEBUG][filelock] - Lock 2169171614000 acquired on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:17,459][DEBUG][filelock] - Attempting to release lock 2169171614000 on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:17,460][DEBUG][filelock] - Lock 2169171614000 released on D:\stable-diffusion-webui\cache.json.lock
[2023-10-09 00:24:18,332][DEBUG][api.py] - SD-Webui API layer loaded
Loading weights [4199bcdd14] from D:\stable-diffusion-webui\models\Stable-diffusion\revAnimated_v122.safetensors
[2023-10-09 00:24:18,846][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,847][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,848][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.reference] - entering reference: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.html_block] - entering html_block: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.heading] - entering heading: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,849][DEBUG][markdown_it.rules_block.lheading] - entering lheading: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,850][DEBUG][markdown_it.rules_block.paragraph] - entering paragraph: StateBlock(line=0,level=2,tokens=2), 0, 1, False
[2023-10-09 00:24:18,926][DEBUG][markdown_it.rules_block.code] - entering code: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,927][DEBUG][markdown_it.rules_block.fence] - entering fence: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.blockquote] - entering blockquote: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.hr] - entering hr: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,928][DEBUG][markdown_it.rules_block.list] - entering list: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.reference] - entering reference: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.html_block] - entering html_block: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.heading] - entering heading: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.lheading] - entering lheading: StateBlock(line=0,level=0,tokens=0), 0, 1, False
[2023-10-09 00:24:18,929][DEBUG][markdown_it.rules_block.paragraph] - entering paragraph: StateBlock(line=0,level=0,tokens=0), 0, 1, False
False

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
Creating model from config: D:\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
DiffusionWrapper has 859.52 M params.
[2023-10-09 00:24:19,373][DEBUG][git.cmd] - Popen(['git', 'remote', 'get-url', '--all', 'origin'], cwd=D:\stable-diffusion-webui\extensions\sd_dreambooth_extension, universal_newlines=False, shell=None, istream=None)
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
[2023-10-09 00:24:19,645][DEBUG][api.py] - Loading Dreambooth API Endpoints.
Startup time: 5.4s (import torch: 1.1s, import gradio: 0.5s, import ldm: 0.3s, other imports: 1.3s, load scripts: 1.2s, create ui: 0.7s, gradio launch: 0.2s).
preload_extensions_git_metadata for 8 extensions took 0.56s
Applying attention optimization: Doggettx... done.
Textual inversion embeddings loaded(0):
Model loaded in 3.4s (load weights from disk: 0.4s, create model: 0.8s, apply weights to model: 0.8s, apply half(): 0.7s, move model to device: 0.7s).
[2023-10-09 00:25:40,801][DEBUG][git.cmd] - Popen(['git', 'fetch', '-v', '--dry-run', '--', 'origin'], cwd=D:\stable-diffusion-webui\extensions\sd_dreambooth_extension, universal_newlines=True, shell=None, istream=None)
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:05<00:00,  3.92it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:03<00:00,  5.80it/s]
[2023-10-09 00:27:46,901][DEBUG][dreambooth.dataclasses.db_config] - Saving to D:\stable-diffusion-webui\models\dreambooth\GemmaHiles
Custom model name is gemmahiles1
[2023-10-09 00:27:48,022][DEBUG][dreambooth.dataclasses.db_config] - Saving to D:\stable-diffusion-webui\models\dreambooth\GemmaHiles
Initializing dreambooth training...
[2023-10-09 00:27:48,108][DEBUG][dreambooth.train_dreambooth] - Adding 'get_velocity' method to DEISMultistepScheduler...
[2023-10-09 00:27:48,108][DEBUG][dreambooth.train_dreambooth] - Adding 'get_velocity' method to UniPCMultistepScheduler...
                  [2023-10-09 00:27:48,258][DEBUG][dreambooth.train_dreambooth] - Pretrained path: D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\working
                                                                                                                       [2023-10-09 00:27:48,455][INFO][dreambooth.utils.gen_utils] - Nothing to generate.     | 12/16 [00:00<00:00, 113.51it/s]

===================================BUG REPORT===================================                 | 0/1 [00:00<?, ?it/s]
================================================================================
The following directories listed in your path were found to be non-existent: {WindowsPath('tmp/restart')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
[2023-10-09 00:29:27,405][WARNING][dreambooth.optimization] - Exception importing 8bit AdamW:
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
Traceback (most recent call last):
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\optimization.py", line 594, in get_optimizer
    from bitsandbytes.optim import AdamW8bit
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research\__init__.py", line 1, in <module>
    from . import nn
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research\nn\__init__.py", line 1, in <module>
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in <module>
    from bitsandbytes.optim import GlobalOptimManager
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\optim\__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "D:\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\cextension.py", line 20, in <module>
    raise RuntimeError('''
RuntimeError:
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
WARNING: Using default optimizer (AdamW from Torch)
                                                                                                                       [2023-10-09 00:29:27,553][DEBUG][dreambooth.utils.gen_utils] - Found 0 reg images.                | 0/1 [00:00<?, ?it/s]
[2023-10-09 00:29:27,554][DEBUG][dreambooth.utils.gen_utils] - Preparing dataset...
Init dataset!
Preparing Dataset (With Caching)
                                                                                                                       Loading latent cache...h Caching):   0%|                                                          | 0/1 [00:00<?, ?it/s]
Bucket 0 (512, 512, 0) - Instance Images: 16 | Class Images: 0 | Max Examples/batch: 16
                                                                                                                       Saving cache!ed latents...: 100%|███████████████████████████████████████████████████████████████| 16/16 [00:00<?, ?it/s]
[2023-10-09 00:29:27,579][ERROR][dreambooth.dataset.db_dataset] - Error saving cache!
Traceback (most recent call last):
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\dataset\db_dataset.py", line 165, in save_cache_file
    os.remove(cache_file)
PermissionError: [WinError 5] Access is denied: 'D:\\stable-diffusion-webui\\models\\dreambooth\\GemmaHiles\\cache\\cache_512.safetensors'
Total Buckets 1 - Instance Images: 16 | Class Images: 0 | Max Examples/batch: 16

Total images / batch: 16, total examples: 16█████████████████████████████████████████| 16/16 [00:00<00:00, 1069.58it/s]
                                                                                                                       [2023-10-09 00:29:27,594][DEBUG][dreambooth.utils.gen_utils] - Total dataset length (steps): 16
                  Initializing bucket counter!
[2023-10-09 00:29:27,795][DEBUG][dreambooth.train_dreambooth] -   ***** Running training *****
[2023-10-09 00:29:27,795][DEBUG][dreambooth.train_dreambooth] -   Num batches each epoch = 4
[2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] -   Num Epochs = 2
[2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] -   Batch Size Per Device = 4
[2023-10-09 00:29:27,796][DEBUG][dreambooth.train_dreambooth] -   Gradient Accumulation steps = 1
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] -   Total train batch size (w. parallel, distributed & accumulation) = 4
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] -   Text Encoder Epochs: 2
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] -   Total optimization steps = 32
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] -   Total training steps = 32
[2023-10-09 00:29:27,797][DEBUG][dreambooth.train_dreambooth] -   Resuming from checkpoint: False
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   First resume epoch: 0
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   First resume step: 0
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   Lora: True, Optimizer: 8bit AdamW, Prec: fp16
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   Gradient Checkpointing: True
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   EMA: True
[2023-10-09 00:29:27,798][DEBUG][dreambooth.train_dreambooth] -   UNET: True
[2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] -   Freeze CLIP Normalization Layers: False
[2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] -   LR (Lora): 0.0001
[2023-10-09 00:29:27,799][DEBUG][dreambooth.train_dreambooth] -   Tenc LR (Lora): 5e-05
[2023-10-09 00:29:27,800][DEBUG][dreambooth.train_dreambooth] -   V2: False
Steps:   0%|                                                                                    | 0/32 [00:00<?, ?it/s]D:\stable-diffusion-webui\venv\lib\site-packages\diffusers\models\attention_processor.py:1468: FutureWarning: `LoRAAttnProcessor2_0` is deprecated and will be removed in version 0.26.0. Make sure use AttnProcessor2_0 instead by settingLoRA layers to `self.{to_q,to_k,to_v,to_out[0]}.lora_layer` respectively. This will be done automatically when using `LoraLoaderMixin.load_lora_weights`
  deprecate(
Steps: 100%|█████████████████████████████████████████| 32/32 [00:14<00:00,  3.07it/s, loss=0.0995, lr=6.6e-6, vram=5.8][2023-10-09 00:29:42,182][DEBUG][dreambooth.train_dreambooth] -
Save completed/canceled.
[2023-10-09 00:29:42,182][DEBUG][dreambooth.train_dreambooth] - Completed, enabling saves.
                                                                                                                       D:\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead.
  warnings.warn(
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:08<00:00,  1.26s/it]
[2023-10-09 00:29:51,704][DEBUG][dreambooth.utils.model_utils] - Enabled AttnProcessor2_0 for UNet2DConditionModel
[2023-10-09 00:29:51,706][DEBUG][dreambooth.utils.model_utils] - Enabled AttnProcessor2_0 for AutoencoderKL,  1.75s/it]
[2023-10-09 00:29:51,706][DEBUG][dreambooth.train_dreambooth] - Save image is set.
[2023-10-09 00:29:51,707][DEBUG][dreambooth.train_dreambooth] - Loras dir: D:\stable-diffusion-webui\models\Lora
                                                                                                                       [2023-10-09 00:29:51,707][DEBUG][dreambooth.train_dreambooth] - Saving lora to D:\stable-diffusion-webui\models\Lora\gemmahiles1_32.safetensors
[2023-10-09 00:29:51,839][DEBUG][dreambooth.train_dreambooth] - Saving images...
[2023-10-09 00:29:51,891][DEBUG][dreambooth.train_dreambooth] - Loading image pipeline from D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\working...
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:02<00:00,  2.38it/s]
[2023-10-09 00:29:55,623][DEBUG][dreambooth.train_dreambooth] - Generating 0 samples...
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 7/7 [00:02<00:00,  2.32it/s]Model name: GemmaHiles 0%|                                                                        | 0/2 [00:00<?, ?it/s]
D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\helpers\log_parser.py:324: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.
  all_df_loss = all_df_loss.fillna(method="ffill")
Saving D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\logging\loss_plot_0.png
Saving D:\stable-diffusion-webui\models\dreambooth\GemmaHiles\logging\ram_plot_0.png
Cleanup log parse.
                                                                                                                       Traceback (most recent call last):████████████████████████                                | 1/2 [00:00<00:00,  3.00it/s]
  File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 422, in run_predict
    output = await app.get_blocks().process_api(
  File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1326, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1251, in postprocess_data
    prediction_value = postprocess_update_dict(
  File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 434, in postprocess_update_dict
    prediction_value["value"] = block.postprocess(prediction_value["value"])
  File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\components.py", line 4461, in postprocess
    file = self.pil_to_temp_file(img, dir=self.DEFAULT_TEMP_DIR)
  File "D:\stable-diffusion-webui\modules\ui_tempdir.py", line 55, in save_pil_to_file
    file_obj = tempfile.NamedTemporaryFile(delete=False, suffix=".png", dir=dir)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 559, in NamedTemporaryFile
    file = _io.open(dir, mode, buffering=buffering,
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 556, in opener
    fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 256, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Admin\\AppData\\Local\\Temp\\gradio\\tmp0o0la6h4.png'
Traceback (most recent call last):
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\ui_functions.py", line 730, in start_training
    result = main(class_gen_method=class_gen_method)
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1809, in main
    return inner_loop()
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\memory.py", line 126, in decorator
    return function(batch_size, grad_size, prof, *args, **kwargs)
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1766, in inner_loop
    check_save(True)
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 976, in check_save
    save_weights(
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1380, in save_weights
    convert_diffusers_to_kohya_lora(lora_save_file, meta, args.lora_weight)
  File "D:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\diff_lora_to_sd_lora.py", line 120, in convert_diffusers_to_kohya_lora
    os.remove(path)
PermissionError: [WinError 5] Access is denied: 'D:\\stable-diffusion-webui\\models\\Lora\\gemmahiles1_32.safetensors'
Steps: 100%|█████████████████████████████████████████| 32/32 [00:28<00:00,  1.10it/s, loss=0.0995, lr=6.6e-6, vram=5.8]
Duration: 00:02:09
Generating Samples: 100%|████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.55it/s]
[2023-10-09 00:29:57,795][DEBUG][dreambooth.utils.model_utils] - Restored system models.
Duration: 00:02:10

Additional information

Version v1.4.1
Python 3.10.6
Torch 2.0.1+Cu118
xformers: N/A
Gradio: 3.32.0
Checkpoint: 4199bcdd14
@klubkelli
Copy link
Author

So no update?

@klubkelli klubkelli mentioned this issue Oct 11, 2023
Closed
1 task
@klubkelli
Copy link
Author

As an update (For what it's worth) I updated my Nvidia graphics drivers to the last available as a new one came out today 537.58 and still did not fix my issue. Same error.

@klubkelli
Copy link
Author

This issue seems to also be related to the problems that others are having such as #1369 and #1372.

@Rocksar
Copy link

Rocksar commented Oct 20, 2023

Hey just here to say that i have the exact same problem and still no solution (more than 5hour of digging solutions)

@SilverRider76
Copy link

Same here, I've also try a clean installation. Nothing is working

@noahjgreer
Copy link

Also still having this problem. It's a bit of a pain as it limits me from saving checkpoints during generation. If I let it run through all the steps, it will give the error at the end but it will also still output the trained lora model to the models/Lora folder. Hopefully a fix for this issue will come out soon.

@swumagic
Copy link

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang）
1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui
2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

@swumagic
Copy link

OR you are Linux distribution (Ubuntu, MacOS, etc.)system ,AND CUDA Version: 11.X.

Bitsandbytes can support ubuntu.（yuhuang）
1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui
2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/TimDettmers/bitsandbytes/releases/download/0.41.0/bitsandbytes-0.41.0-py3-none-any.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

1 similar comment
@swumagic
Copy link

OR you are Linux distribution (Ubuntu, MacOS, etc.)system ,AND CUDA Version: 11.X.

Bitsandbytes can support ubuntu.（yuhuang）
1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui
2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/TimDettmers/bitsandbytes/releases/download/0.41.0/bitsandbytes-0.41.0-py3-none-any.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

@d8ahazard
Copy link
Owner

Closed, duplicate of #1360 , #1369 , also fixed with latest release.

@d8ahazard d8ahazard mentioned this issue Nov 16, 2023
Closed
@thecakeisal1e thecakeisal1e mentioned this issue Dec 26, 2023
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@d8ahazard @Rocksar @noahjgreer @SilverRider76 @swumagic @klubkelli