AdamW8 bit and Lion8bit not working - CUDA_SETUP: WARNING! libcudart.so not found in any environmental path.

[FurkanGozukara]

I am on windows

It is looking for libcudart.so however i think this only exists in linux

this is what we have in windows

i have cuda installed and there is also no such file

the problem is your code only looking for linux not for window cuda. it is cudart.lib in windows

this is cuda folder screenshot

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
================================================================================
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
F:\kohya_ss\venv\lib\site-packages\bitsandbytes\cuda_setup\paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
  warn(
WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
CUDA SETUP: Loading binary F:\kohya_ss\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ F:\kohya_ss\train_network.py:873 in <module>                                                     │
│                                                                                                  │
│   870 │   args = parser.parse_args()                                                             │
│   871 │   args = train_util.read_config_from_file(args, parser)                                  │
│   872 │                                                                                          │
│ ❱ 873 │   train(args)                                                                            │
│   874                                                                                            │
│                                                                                                  │
│ F:\kohya_ss\train_network.py:262 in train                                                        │
│                                                                                                  │
│   259 │   │   )                                                                                  │
│   260 │   │   trainable_params = network.prepare_optimizer_params(args.text_encoder_lr, args.u   │
│   261 │                                                                                          │
│ ❱ 262 │   optimizer_name, optimizer_args, optimizer = train_util.get_optimizer(args, trainable   │
│   263 │                                                                                          │
│   264 │   # dataloaderを準備する                                                                 │
│   265 │   # DataLoaderのプロセス数：0はメインプロセスになる                                      │
│                                                                                                  │
│ F:\kohya_ss\library\train_util.py:2699 in get_optimizer                                          │
│                                                                                                  │
│   2696 │                                                                                         │
│   2697 │   if optimizer_type == "AdamW8bit".lower():                                             │
│   2698 │   │   try:                                                                              │
│ ❱ 2699 │   │   │   import bitsandbytes as bnb                                                    │
│   2700 │   │   except ImportError:                                                               │
│   2701 │   │   │   raise ImportError("No bitsand bytes / bitsandbytesがインストールされていない  │
│   2702 │   │   print(f"use 8-bit AdamW optimizer | {optimizer_kwargs}")                          │
│                                                                                                  │
│ F:\kohya_ss\venv\lib\site-packages\bitsandbytes\__init__.py:6 in <module>                        │
│                                                                                                  │
│    3 # This source code is licensed under the MIT license found in the                           │
│    4 # LICENSE file in the root directory of this source tree.                                   │
│    5                                                                                             │
│ ❱  6 from .autograd._functions import (                                                          │
│    7 │   MatmulLtState,                                                                          │
│    8 │   bmm_cublas,                                                                             │
│    9 │   matmul,                                                                                 │
│                                                                                                  │
│ F:\kohya_ss\venv\lib\site-packages\bitsandbytes\autograd\_functions.py:5 in <module>             │
│                                                                                                  │
│     2 import warnings                                                                            │
│     3                                                                                            │
│     4 import torch                                                                               │
│ ❱   5 import bitsandbytes.functional as F                                                        │
│     6                                                                                            │
│     7 from dataclasses import dataclass                                                          │
│     8 from functools import reduce  # Required in Python 3                                       │
│                                                                                                  │
│ F:\kohya_ss\venv\lib\site-packages\bitsandbytes\functional.py:13 in <module>                     │
│                                                                                                  │
│     10 from typing import Tuple                                                                  │
│     11 from torch import Tensor                                                                  │
│     12                                                                                           │
│ ❱   13 from .cextension import COMPILED_WITH_CUDA, lib                                           │
│     14 from functools import reduce  # Required in Python 3                                      │
│     15                                                                                           │
│     16 # math.prod not compatible with python < 3.8                                              │
│                                                                                                  │
│ F:\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py:41 in <module>                     │
│                                                                                                  │
│   38 │   │   return cls._instance                                                                │
│   39                                                                                             │
│   40                                                                                             │
│ ❱ 41 lib = CUDALibrary_Singleton.get_instance().lib                                              │
│   42 try:                                                                                        │
│   43 │   lib.cadam32bit_g32                                                                      │
│   44 │   lib.get_context.restype = ct.c_void_p                                                   │
│                                                                                                  │
│ F:\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py:37 in get_instance                 │
│                                                                                                  │
│   34 │   def get_instance(cls):                                                                  │
│   35 │   │   if cls._instance is None:                                                           │
│   36 │   │   │   cls._instance = cls.__new__(cls)                                                │
│ ❱ 37 │   │   │   cls._instance.initialize()                                                      │
│   38 │   │   return cls._instance                                                                │
│   39                                                                                             │
│   40                                                                                             │
│                                                                                                  │
│ F:\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py:31 in initialize                   │
│                                                                                                  │
│   28 │   │   │   self.lib = ct.cdll.LoadLibrary(binary_path)                                     │
│   29 │   │   else:                                                                               │
│   30 │   │   │   print(f"CUDA SETUP: Loading binary {binary_path}...")                           │
│ ❱ 31 │   │   │   self.lib = ct.cdll.LoadLibrary(binary_path)                                     │
│   32 │                                                                                           │
│   33 │   @classmethod                                                                            │
│   34 │   def get_instance(cls):                                                                  │
│                                                                                                  │
│ C:\Python3108\lib\ctypes\__init__.py:452 in LoadLibrary                                          │
│                                                                                                  │
│   449 │   │   return getattr(self, name)                                                         │
│   450 │                                                                                          │
│   451 │   def LoadLibrary(self, name):                                                           │
│ ❱ 452 │   │   return self._dlltype(name)                                                         │
│   453 │                                                                                          │
│   454 │   __class_getitem__ = classmethod(_types.GenericAlias)                                   │
│   455                                                                                            │
│                                                                                                  │
│ C:\Python3108\lib\ctypes\__init__.py:364 in __init__                                             │
│                                                                                                  │
│   361 │   │   │   else:                                                                          │
│   362 │   │   │   │   import nt                                                                  │
│   363 │   │   │   │   mode = nt._LOAD_LIBRARY_SEARCH_DEFAULT_DIRS                                │
│ ❱ 364 │   │   │   │   if '/' in name or '\\' in name:                                            │
│   365 │   │   │   │   │   self._name = nt._getfullpathname(self._name)                           │
│   366 │   │   │   │   │   mode |= nt._LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR                           │
│   367                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: argument of type 'WindowsPath' is not iterable
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ C:\Python3108\lib\runpy.py:196 in _run_module_as_main                                            │
│                                                                                                  │
│   193 │   main_globals = sys.modules["__main__"].__dict__                                        │
│   194 │   if alter_argv:                                                                         │
│   195 │   │   sys.argv[0] = mod_spec.origin                                                      │
│ ❱ 196 │   return _run_code(code, main_globals, None,                                             │
│   197 │   │   │   │   │    "__main__", mod_spec)                                                 │
│   198                                                                                            │
│   199 def run_module(mod_name, init_globals=None,                                                │
│                                                                                                  │
│ C:\Python3108\lib\runpy.py:86 in _run_code                                                       │
│                                                                                                  │
│    83 │   │   │   │   │      __loader__ = loader,                                                │
│    84 │   │   │   │   │      __package__ = pkg_name,                                             │
│    85 │   │   │   │   │      __spec__ = mod_spec)                                                │
│ ❱  86 │   exec(code, run_globals)                                                                │
│    87 │   return run_globals                                                                     │
│    88                                                                                            │
│    89 def _run_module_code(code, init_globals=None,                                              │
│                                                                                                  │
│ in <module>:7                                                                                    │
│                                                                                                  │
│   4 from accelerate.commands.accelerate_cli import main                                          │
│   5 if __name__ == '__main__':                                                                   │
│   6 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 7 │   sys.exit(main())                                                                         │
│   8                                                                                              │
│                                                                                                  │
│ F:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py:45 in main              │
│                                                                                                  │
│   42 │   │   exit(1)                                                                             │
│   43 │                                                                                           │
│   44 │   # Run                                                                                   │
│ ❱ 45 │   args.func(args)                                                                         │
│   46                                                                                             │
│   47                                                                                             │
│   48 if __name__ == "__main__":                                                                  │
│                                                                                                  │
│ F:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py:918 in launch_command           │
│                                                                                                  │
│   915 │   elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA   │
│   916 │   │   sagemaker_launcher(defaults, args)                                                 │
│   917 │   else:                                                                                  │
│ ❱ 918 │   │   simple_launcher(args)                                                              │
│   919                                                                                            │
│   920                                                                                            │
│   921 def main():                                                                                │
│                                                                                                  │
│ F:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py:580 in simple_launcher          │
│                                                                                                  │
│   577 │   process.wait()                                                                         │
│   578 │   if process.returncode != 0:                                                            │
│   579 │   │   if not args.quiet:                                                                 │
│ ❱ 580 │   │   │   raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)    │
│   581 │   │   else:                                                                              │
│   582 │   │   │   sys.exit(1)                                                                    │
│   583                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['F:\\kohya_ss\\venv\\Scripts\\python.exe', 'train_network.py', '--enable_bucket', '--pretrained_model_name_or_path=F:/0 models/v1-5-pruned.ckpt',
'--train_data_dir=F:\\test1\\img', '--resolution=512,512', '--output_dir=F:\\test1\\model', '--logging_dir=F:\\test1\\log', '--network_alpha=1', '--save_model_as=safetensors',
'--network_module=networks.lora', '--text_encoder_lr=5e-05', '--unet_lr=0.0001', '--network_dim=128', '--output_name=test1', '--lr_scheduler_num_cycles=8', '--learning_rate=0.0001',
'--lr_scheduler=cosine', '--lr_warmup_steps=260', '--train_batch_size=1', '--max_train_steps=2600', '--save_every_n_epochs=1', '--mixed_precision=bf16', '--save_precision=bf16',
'--cache_latents', '--optimizer_type=AdamW8bit', '--max_data_loader_n_workers=0', '--bucket_reso_steps=64', '--mem_eff_attn', '--gradient_checkpointing', '--xformers', '--bucket_no_upscale']'
returned non-zero exit status 1.

[FurkanGozukara]

this command looks like fixing at least for adamw8bit

pip install bitsandbytes-windows

[sdbds]

you cuda is 11.4，but you can see no 114.dll for windows...
so update to 11.6 or use other bitsandbytes version