Add torch_utils class, auto-detect CUDA availability

[ervteng]

Proposed change(s)

This PR adds a torch_utils class (similiar to tf_utils) and requires importing torch from there. This lets us do a couple things, the first being detect if CUDA is available and set the default tensor type appropriately. Requiring importing torch from here ensures that this will be set before any torch functions are used at all. The PR also adds an is_available() method to torch_utils, and throws a nicer error if torch isn't available when you passed --torch.

In the future we can also set the number of torch threads, etc. from here.

Types of change(s)

• Bug fix
• New feature
• Code refactor
• Breaking change
• Documentation update
• Other (please describe)

Checklist

• Added tests that prove my fix is effective or that my feature works
• Updated the changelog (if applicable)
• Updated the documentation (if applicable)
• Updated the migration guide (if applicable)

Other comments

[ervteng]

I'm pretty sure this isn't the best way to do this, but without it, pylint will complain about torch not having the right members everywhere. I believe this is because torch could be None, though it never actually happens.

[chriselion]

Suggested change

	torch = use mlagents.torch_utils istead (handles GPU detection).
	torch = use mlagents.torch_utils instead (handles GPU detection).

[vincentpierre]

Looks good to me, but I would like ml-agents/mlagents/torch_utils/torch.py to contain a comment stating that the file is temporary and will be removed once torch is required. (To avoid adding functionality to this file in the mean time)

[ervteng]

Looks good to me, but I would like ml-agents/mlagents/torch_utils/torch.py to contain a comment stating that the file is temporary and will be removed once torch is required. (To avoid adding functionality to this file in the mean time)

The file isn't temporary (will still be needed for GPU detection) but yeah the try/except is. Added comment.

[dongruoping]

Only tensors are created on GPU but the models are still on CPU.
Should it also be addressed in this PR?

[dongruoping]

Another point missing here is that when running Torch with CuDNN, if we want to get completely reproducible results, we'll also need to set these besides setting seeds for torch and numpy:

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

while these could potentially hurt the running performance.

[ervteng]

Another point missing here is that when running Torch with CuDNN, if we want to get completely reproducible results, we'll also need to set these besides setting seeds for torch and numpy:

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

while these could potentially hurt the running performance.

How much of a performance hit do we expect by setting these flags?

[dongruoping]

How much of a performance hit do we expect by setting these flags?

Not sure, it could be case by case, depending on how much the model changes during training and how much optimization cudnn does.
I haven't really try to quantify the impact, but it's a general warning on PyTorch doc.

[dongruoping]

ml-agents/mlagents/trainers/policy/torch_policy.py: line 61 should be removed. It reset the default tensor type to non-cuda, which is already properly set in utils .

I tested with removing that line. It works well.