Fails to load if nvidia mig driver used for gpu

[djjeffr]

How to reproduce this bug?

in values yaml
muti2vec-clip:
...
enable_cuda: true
...
#added this line to make it work
cuda_visible_devices: 0
...
nvidia.com/mig-1g.5gb: 1

Also had to modiify file transformersInferenceDeployment.yaml to allow the line cuda_visible_devices
-helm chart 16.4.0

What is the expected behavior?

Was expecting it to work without adding cuda_visible_devices line, the same as it worked when using nvidia.com/gpu: 1

What is the actual behavior?

If cuda_visible_device is not there it fails to start with error cuda_visible_device must be there to use nvidia mig driver.

Supporting information

No response

Server Version

1.20.5

Code of Conduct

• I have read and agree to the Weaviate's Contributor Guide and Code of Conduct