Images all black

[ivantan-ys]

I encountered this after running accelerate launch --config_file 1gpu.yaml test_mvdiffusion_seq.py --config configs/mvdiffusion-joint-ortho-6views.yaml, where the output images are all black:

C:\ProgramData\anaconda3\envs\wonder3d\lib\site-packages\torch\nn\modules\conv.py:456: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.)
  return F.conv2d(input, weight, bias, self.stride,
C:\ProgramData\anaconda3\envs\wonder3d\lib\site-packages\rembg\sessions\base.py:52: RuntimeWarning: invalid value encountered in divide
  im_ary = im_ary / np.max(im_ary)
C:\ProgramData\anaconda3\envs\wonder3d\lib\site-packages\rembg\sessions\u2net.py:44: RuntimeWarning: invalid value encountered in cast
  mask = Image.fromarray((pred * 255).astype("uint8"), mode="L")
1it [08:21, 501.27s/it]

Here's my env information, any advice?

Python version: 3.10.14 | packaged by Anaconda, Inc. | (main, May  6 2024, 19:44:50) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22631-SP0
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 Ti
Nvidia driver version: 531.18
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2601
DeviceID=CPU0
Family=198
L2CacheSize=1536
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2601
Name=Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.3
[pip3] onnxruntime==1.18.0
[pip3] pytorch-lightning==1.9.5
[pip3] torch==2.3.0+cu118
[pip3] torch_efficient_distloss==0.1.3
[pip3] torchaudio==2.3.0+cu118
[pip3] torchmetrics==1.4.0.post0
[pip3] torchvision==0.18.0+cu118
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.8.0               hd77b12b_0
[conda] mkl                       2021.4.0           haa95532_640
[conda] mkl-service               2.4.0           py310h2bbff1b_0
[conda] mkl_fft                   1.3.1           py310ha0764ea_0
[conda] mkl_random                1.2.2           py310h4ed8f06_0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] numpy-base                1.24.3          py310h3caf3d7_0
[conda] pytorch                   2.3.0           py3.10_cuda11.8_cudnn8_0    pytorch
[conda] pytorch-cuda              11.8                 h24eeafa_5    pytorch
[conda] pytorch-lightning         1.9.5                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch                     2.3.0+cu118              pypi_0    pypi
[conda] torch-efficient-distloss  0.1.3                    pypi_0    pypi
[conda] torchaudio                2.3.0                    pypi_0    pypi
[conda] torchmetrics              1.4.0.post0              pypi_0    pypi
[conda] torchvision               0.18.0                   pypi_0    pypi