Do we able to use image as input for IsaacGym ?

[sizhe-wang]

Dear, author:
we have seen resnet_actor_critic setting for training the RL algorithm. Does IsaacGym support image type input ? Thanks advance. looking forward your reply !

[ViktorM]

Hi @sizhe-wang,

Yes, Isaac Gym supports camera sensors. We don't currently have examples of RL training environments using camera input but might consider adding a simple one as it is a pretty common request to provide an example. @gavrielstate for visibility.

[gavrielstate]

We're hoping to add an additional basic example using camera input for RL control later this winter.

[noshaba]

@gavrielstate How about controller inputs? :)

[gavrielstate]

We won't be adding direct support for controllers to Isaac Gym I'm afraid, but it should be available now within Isaac Sim. We will have more RL capabilities in future versions of Isaac Sim that include the Gym-style tensor physics APIs.

[erwincoumans]

See my fork: https://github.com/erwincoumans/IsaacGymEnvs/tree/cartpole_camera
It has task=CartpoleCamera (thanks to @ViktorM )