RuntimeError: CUDA error: invalid argument

[i2chris]

I'm running this dreambooth extension using all the default settings and only changing these three settings:

Instance prompt: photo of florich girl
Class prompt: photo of girl
Dataset directory: D:\images\flo-output

When I run this I get the error "RuntimeError: CUDA error: invalid argument"

Is there something obvious causing this?

Running on 3090 24GB

Arguments: ('florich', 'D:\\images\\flo-output', '', 'photo of florich girl', 'photo of girl', '', '', 1.0, 7.5, 40.0, 0, 512, False, True, 1, 1, 1, 1000, 1, True, 5e-06, False, 'constant', 0, False, 0.9, 0.999, 0.01, 1e-08, 1, 500, 500, 'no', True, '', False, True, True) {}
Traceback (most recent call last):
  File "C:\github\stable-diffusion-webui\modules\[ui.py](http://ui.py/)", line 185, in f
    res = list(func(*args, **kwargs))
  File "C:\github\stable-diffusion-webui\[webui.py](http://webui.py/)", line 54, in f
    res = func(*args, **kwargs)
  File "C:\github\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\[dreambooth.py](http://dreambooth.py/)", line 265, in start_training
    trained_steps = main(config)
  File "C:\github\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\[train_dreambooth.py](http://train_dreambooth.py/)", line 790, in main
    accelerator.backward(loss)
  File "C:\github\stable-diffusion-webui\venv\lib\site-packages\accelerate\[accelerator.py](http://accelerator.py/)", line 884, in backward
    loss.backward(**kwargs)
  File "C:\github\stable-diffusion-webui\venv\lib\site-packages\torch\[_tensor.py](http://_tensor.py/)", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "C:\github\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\[__init__.py](http://__init__.py/)", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "C:\github\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\[function.py](http://function.py/)", line 253, in apply
    return user_fn(self, *args)
  File "C:\github\stable-diffusion-webui\venv\lib\site-packages\torch\utils\[checkpoint.py](http://checkpoint.py/)", line 146, in backward
    torch.autograd.backward(outputs_with_grad, args_with_grad)
  File "C:\github\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\[__init__.py](http://__init__.py/)", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "C:\github\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\[function.py](http://function.py/)", line 253, in apply
    return user_fn(self, *args)
  File "C:\github\stable-diffusion-webui\venv\lib\site-packages\xformers\[ops.py](http://ops.py/)", line 369, in backward
    ) = torch.ops.xformers.efficient_attention_backward_cutlass(
  File "C:\github\stable-diffusion-webui\venv\lib\site-packages\torch\[_ops.py](http://_ops.py/)", line 143, in __call__
    return self._op(*args, **kwargs or {})
RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[zark119]

did you add --xformers to commandline args?

[Pythonpa]

So, is this need to add --xformers to commandline args or not?

[zark119]

So, is this need to add --xformers to commandline args or not?

without --xformers, I had this error.

[seihoukei]

I have same error with and without --xformers.

[seihoukei]

This is a duplicate of issue #48 , I'll duplicate my finding - enabling adam and fp16 precision made it work, so it's default value for either that causes error.

[i2chris]

Yes, --xformers is added to commandline.

[d8ahazard]

Closing as a duplicate of #48. I'll leave that one open for now until we establish this is fully resolved.