[tune][rllib] Windows: GPU not recognized

[juliusfrost]

What is the problem?

I'm getting ray.tune.error.TuneError: Insufficient cluster resources to launch trial.
I specified a GPU in my config but ray does not recognize my GPU (RTX 2080) and throws an error.
I can get passed this by setting num_gpus: 0 in my config for now.

https://gist.github.com/juliusfrost/fa7ebbb8d1dfc66eea0bbc4babcbe5aa

Reproduction (REQUIRED)

git clone https://github.com/juliusfrost/rllib-tune-atari.git
cd rllib-tune-atari
pip install -r requirements.txt
python train.py --algo a2c

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[juliusfrost]

#9114 @mehrdadn

[juliusfrost]

@mehrdadn Did you get a chance to look at this? I updated the script to reproduce to remove external dependencies.

[mehrdadn]

@juliusfrost I don't have the full CUDA setup available to test this right now, but perhaps try running this after import ray and let me know if it changes anything?

import ray.resource_spec
import subprocess
# NOTE: This may not work in Windows 11+; see below for another solution.
ray.resource_spec._autodetect_num_gpus = lambda: len([line.rstrip() for line in subprocess.check_output([os.path.join(os.environ['SystemRoot'], 'System32', 'wbem', 'WMIC.exe'), "PATH", "Win32_VideoController", "GET", "AdapterCompatibility"], creationflags=subprocess.CREATE_NO_WINDOW).splitlines()[1:] if line.startswith(b"NVIDIA")])

2022 update: If the above doesn't work, you can try using PowerShell (below) instead.

[juliusfrost]

@mehrdadn Awesome it worked! Great job 👍

[abhayraw1]

I'm still getting this issue. Following the snippet posted by @mehrdadn ray.resource_spec._autodetect_num_gpus() returns 1 but ray.get_gpu_ids() still returns an empty list [].

ray version: 0.9.0.dev0
OS: Windows 10

[mehrdadn]

@wuisawesome Might this be due to _get_gpu_info_string not handling Windows?

[mehrdadn]

Heads up, WMIC is now deprecated so the approach from #9300 may stop working. CC @richardliaw

[fnw]

I am on Ray 1.9.0 and on Windows 11 Insider Preview (Build 22483.1011) and attempting to run GPU experiments results in a "FileNotFoundError: [WinError 2] The system cannot find the file specified" error when trying to run the WMIC command in the GPU auto-detection function, which would seem to indicate that WMIC has already been removed, at least on Insider Preview builds.

[sai-prasanna]

Anyone managed to fix this?

[mehrdadn]

@sai-prasanna @richardliaw @fnw you should be able to use PowerShell instead:

import ray.resource_spec
import subprocess
ray.resource_spec._autodetect_num_gpus = lambda: int(subprocess.check_output([os.path.join(os.environ['SystemRoot'], 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe'), '-Command', "& {@(Get-WmiObject -Query \"SELECT * FROM Win32_VideoController WHERE AdapterCompatibility = 'NVIDIA'\").count}"], stdin=subprocess.DEVNULL, stderr=subprocess.DEVNULL, creationflags=subprocess.CREATE_NO_WINDOW).rstrip())

That should be sufficient if they haven't removed the ability to query WMI from PowerShell.

---

Otherwise, you can do

python3 -m pip install wmi

and then

ray.resource_spec._autodetect_num_gpus = lambda: len(list(filter(lambda c: c.wmi_property('AdapterCompatibility').value == "NVIDIA", __import__('wmi').WMI().Win32_VideoController())))

If it tries to install pywin32 and fails, you might need to first grab the wheel for that from Gohlke's website.

It's probably possible to use ctypes to manually invoke WMI stuff and bypass the need for that, but it's a fair bit more effort.

[mattip]

I wonder if we could use gpustat to get this information instead