Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Error] [carb.gym.plugin] cudaImportExternalMemory failed on rgbImage buffer with error 999 #10

Open
WanyingMo opened this issue Nov 8, 2023 · 3 comments

Comments

@WanyingMo
Copy link

WanyingMo commented Nov 8, 2023

Hello, when I run the distillation policy, this error was returned, while everything goes fine during base policy training. Does anyone know how to solve this? Thanks a lot!
I know that it might be the problem of ubuntu version, is there any solution not to downgrade system?QAQ

command: python train.py --exptid 111-11 --resume --resumeid 000-00 --delay --use_camera --no_wandb

Importing module 'gym_38' (/home/user/apps/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/user/apps/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.10.0+cu113
Device count 1
/home/user/apps/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/user/.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Emitting ninja build file /home/user/.cache/torch_extensions/py38_cu113/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
Setting seed: 1
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled

Start creating ground...
Converting heightmap to trimesh...
iteration 0 - triangles 3414402 threshold 2.187e-06
iteration 5 - triangles 1776036 threshold 0.00209715
iteration 10 - triangles 458906 threshold 0.0627485
simplified mesh in 2.6172 seconds from 3414402 to 170720 triangles
Created 88159 vertices
Created 170720 triangles
Adding trimesh to simulation...
Trimesh added
Finished creating ground. Time taken 4.11 s

Creating env...
0%| | 0/192 [00:00<?, ?it/s][Error] [carb.gym.plugin] cudaImportExternalMemory failed on rgbImage buffer with error 999
Segmentation fault (core dumped)

I fixed it with downgrade my ubuntu22.04 to 20.04. Seems the only way to fix it.

@WanyingMo
Copy link
Author

I fixed it with downgrade my ubuntu22.04 to 20.04. Seems the only way to fix it.

@lienbo
Copy link

lienbo commented Jun 28, 2024

add this in your code or export the environment
os.environ['VK_ICD_FILENAMES'] ='/usr/share/vulkan/icd.d/nvidia_icd.json'

it works for me ubuntu22.04 conda python3.8

@Ytydt-Reuz
Copy link

add this in your code or export the environment os.environ['VK_ICD_FILENAMES'] ='/usr/share/vulkan/icd.d/nvidia_icd.json'

it works for me ubuntu22.04 conda python3.8

THIS really works ! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants