Report a bug: warp?(codes can be run well in my workstation) #2
Comments
|
I got the same problem. I am not sure if that's the warp context problem. It cannot recognize the cuda path. |
|
same problem |
|
When I upgraded the warp version to 0.15.1, this issue was resolved. At the same time, replace the tape. py file with the one mentioned by the author in another issue. |
|
Sorry for the late reply. I tried but I cannot reproduce the error on my end though. Could you attach more details about your workstation? Does |
|
Thanks for your reply.
system: Ubuntu 20.04.4 LTS
GPU: NVIDIA-SMI 470.57.02 , 0 Quadro RTX 6000
In the "conda list":
nvidia-cublas-cu11 11.10.3.66 pypi_0 pypi
nvidia-cuda-cupti-cu11 11.7.101 pypi_0 pypi
nvidia-cuda-nvrtc-cu11 11.7.99 pypi_0 pypi
nvidia-cuda-runtime-cu11 11.7.99 pypi_0 pypi
nvidia-cudnn-cu11 8.5.0.96 pypi_0 pypi
nvidia-cufft-cu11 10.9.0.58 pypi_0 pypi
nvidia-curand-cu11 10.2.10.91 pypi_0 pypi
nvidia-cusolver-cu11 11.4.0.1 pypi_0 pypi
nvidia-cusparse-cu11 11.7.4.91 pypi_0 pypi
nvidia-nccl-cu11 2.14.3 pypi_0 pypi
nvidia-nvtx-cu11 11.7.91 pypi_0 pypi
python 3.10.14
torch 2.0.1 pypi_0 pypi
torchaudio 2.0.2 pypi_0 pypi
torchvision 0.15.2 pypi_0 pypi
Cuda:0 works well for torch
Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>> import torch
>> if torch.cuda.is_available():
... print("CUDA is available")
... else:
... print("CUDA is not available")
...
CUDA is available
>> device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
>> print("Using device:", device)
Using device: cuda:0
Actually, I also tried another new DELL workstation(nvidia driver:535, A6000), it reported the same error info.
…________________________________
From: Pingchuan Ma ***@***.***>
Sent: Wednesday, August 28, 2024 01:55
To: PingchuanMa/NCLaw ***@***.***>
Cc: ***@***.*** ***@***.***>; Author ***@***.***>
Subject: Re: [PingchuanMa/NCLaw] Report a bug: warp?(codes can be run well in my workstation) (Issue #2)
This is an external email.
Sorry for the late reply. I tried but I cannot reproduce the error on my end though. Could you attach more details about your workstation? Does cuda:0 works for your torch.device or it's only broken in warp?
—
Reply to this email directly, view it on GitHub<#2 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALYH3ZBMCO36AWKYZIYMNYDZTS4R5AVCNFSM6AAAAABJGNCWICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJTGE4DKNZYGY>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
did you install warp from scratch or from pypi? could you also attach the output of bash command |
|
Yes.
Nvcc version is 11.3. But another workstation is 11.8, both report same error.
On 30 Aug 2024, at 00:02, Pingchuan Ma ***@***.***> wrote:
This is an external email.
did you install warp from scratch or from pypi? could you also attach the output of bash command which nvcc please?
—
Reply to this email directly, view it on GitHub<#2 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALYH3ZFDCS42I2Q67I56DQTZT5AYHAVCNFSM6AAAAABJGNCWICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJYGIZTKMBYGM>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
I do presume that this is a warp installation problem. Could you try to compile warp from scratch? The warp version used in this repo is attached in |
|
OK, thanks.
l will let you know after once try.
…Sent from my iPhone
On 31 Aug 2024, at 07:26, Pingchuan Ma ***@***.***> wrote:
This is an external email.
I do presume that this is a warp installation problem. Could you try to compile warp from scratch? The warp version used in this repo is attached in third_party/warp. I will also try to reproduce this on a computer with cuda 11.x to see if it's a cuda problem. Will let you know.
—
Reply to this email directly, view it on GitHub<#2 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALYH3ZEVEF3M3U5YSYSJO4TZUD5RPAVCNFSM6AAAAABJGNCWICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRSGU4TKNRZGI>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Hi pingchuan, Sry for late reply. I didn't work these days. Many thanks! |
when I excute the command:
xxx$python experiments/scripts/dataset/main.pyI got this, as follow:
`env:
blob:
bsdf_pcd:
type: diffuse
reflectance:
type: rgb
value:
- 0.92941176
- 0.32941176
- 0.23137255
material:
elasticity:
cls: CorotatedElasticity
E: 100000.0
nu: 0.2
requires_grad: false
random: false
plasticity:
cls: IdentityPlasticity
requires_grad: false
random: false
name: jelly
ckpt: null
shape:
type: cube
name: dataset
center:
- 0.5
- 0.5
- 0.5
size:
- 0.5
- 0.5
- 0.5
resolution: 10
mode: uniform
sort: null
vel:
random: false
lin_vel:
- 1.0
- -1.5
- -2.0
ang_vel:
- 4.0
- 4.0
- 4.0
name: jelly
rho: 1000.0
span:
- 0
- 1000
clip_bound: 0.5
render:
spp: 32
width: 512
height: 512
skip_frame: 25
bound: 1.75
mpm_mul: 6
sph_version: cuda_ad_rgb
pcd_version: cuda_ad_rgb
has_sphere_emitter: true
fps: 10
sim:
quality: low
num_steps: 1000
gravity:
bc: freeslip
num_grids: 20
dt: 0.0005
bound: 3
eps: 1.0e-07
skip_frame: 1
train:
teacher:
strategy: cosine
start_lambda: 25
end_lambda: 200
num_epochs: 300
batch_size: 128
elasticity_lr: 1.0
plasticity_lr: 0.1
elasticity_wd: 0.0
plasticity_wd: 0.0
elasticity_grad_max_norm: 0.1
plasticity_grad_max_norm: 0.1
name: jelly/dataset
seed: 0
cpu: 0
num_cpus: 128
gpu: 0
overwrite: false
resume: false
gt_name: null
dataset: true
Warp 0.6.1 initialized:
CUDA not available
Devices:
"cpu" | x86_64
Kernel cache: /home/xxxx/.cache/warp/0.6.1
Error executing job with overrides: ['overwrite=False', 'resume=False', 'gpu=0', 'cpu=0', 'env=jelly', 'render=debug', 'sim=low', 'name=jelly/dataset', 'dataset=True']
Traceback (most recent call last):
File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 137, in
main()
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in
lambda: hydra.run(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 35, in main
wp_device = wp.get_device(f'cuda:{cfg.gpu}')
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1676, in get_device
return runtime.get_device(ident)
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1484, in get_device
return self.device_map[ident]
KeyError: 'cuda:0'
env:
blob:
bsdf_pcd:
type: diffuse
reflectance:
type: rgb
value:
- 0.96078431
- 0.88235294
- 0.63529412
material:
elasticity:
cls: SigmaElasticity
E: 1000000.0
nu: 0.2
requires_grad: false
random: false
plasticity:
cls: DruckerPragerPlasticity
E: 1000000.0
nu: 0.2
friction_angle: 25.0
cohesion: 0.0
requires_grad: false
random: false
name: sand
ckpt: null
shape:
type: cube
name: dataset
center:
- 0.5
- 0.5
- 0.5
size:
- 0.5
- 0.5
- 0.5
resolution: 10
mode: uniform
sort: null
vel:
random: false
lin_vel:
- 1.0
- -1.5
- -2.0
ang_vel:
- 4.0
- 4.0
- 4.0
name: sand
rho: 1000.0
span:
- 0
- 1000
clip_bound: 0.5
render:
spp: 32
width: 512
height: 512
skip_frame: 25
bound: 1.75
mpm_mul: 6
sph_version: cuda_ad_rgb
pcd_version: cuda_ad_rgb
has_sphere_emitter: true
fps: 10
sim:
quality: low
num_steps: 1000
gravity:
bc: freeslip
num_grids: 20
dt: 0.0005
bound: 3
eps: 1.0e-07
skip_frame: 1
train:
teacher:
strategy: cosine
start_lambda: 25
end_lambda: 200
num_epochs: 300
batch_size: 128
elasticity_lr: 1.0
plasticity_lr: 0.1
elasticity_wd: 0.0
plasticity_wd: 0.0
elasticity_grad_max_norm: 0.1
plasticity_grad_max_norm: 0.1
name: sand/dataset
seed: 0
cpu: 0
num_cpus: 128
gpu: 0
overwrite: false
resume: false
gt_name: null
dataset: true
Warp 0.6.1 initialized:
CUDA not available
Devices:
"cpu" | x86_64
Kernel cache: /home/xxxx/.cache/warp/0.6.1
Error executing job with overrides: ['overwrite=False', 'resume=False', 'gpu=0', 'cpu=0', 'env=sand', 'render=debug', 'sim=low', 'name=sand/dataset', 'dataset=True']
Traceback (most recent call last):
File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 137, in
main()
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in
lambda: hydra.run(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 35, in main
wp_device = wp.get_device(f'cuda:{cfg.gpu}')
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1676, in get_device
return runtime.get_device(ident)
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1484, in get_device
return self.device_map[ident]
KeyError: 'cuda:0'
env:
blob:
bsdf_pcd:
type: diffuse
reflectance:
type: rgb
value:
- 0.59607843
- 0.98431373
- 0.59607843
material:
elasticity:
cls: SigmaElasticity
E: 300000.0
nu: 0.25
requires_grad: false
random: false
plasticity:
cls: VonMisesPlasticity
E: 300000.0
nu: 0.25
sigma_y: 5000.0
requires_grad: false
random: false
name: plasticine
ckpt: null
shape:
type: cube
name: dataset
center:
- 0.5
- 0.5
- 0.5
size:
- 0.5
- 0.5
- 0.5
resolution: 10
mode: uniform
sort: null
vel:
random: false
lin_vel:
- 1.0
- -1.5
- -2.0
ang_vel:
- 4.0
- 4.0
- 4.0
name: plasticine
rho: 1000.0
span:
- 0
- 1000
clip_bound: 0.5
render:
spp: 32
width: 512
height: 512
skip_frame: 25
bound: 1.75
mpm_mul: 6
sph_version: cuda_ad_rgb
pcd_version: cuda_ad_rgb
has_sphere_emitter: true
fps: 10
sim:
quality: low
num_steps: 1000
gravity:
bc: freeslip
num_grids: 20
dt: 0.0005
bound: 3
eps: 1.0e-07
skip_frame: 1
train:
teacher:
strategy: cosine
start_lambda: 25
end_lambda: 200
num_epochs: 300
batch_size: 128
elasticity_lr: 1.0
plasticity_lr: 0.1
elasticity_wd: 0.0
plasticity_wd: 0.0
elasticity_grad_max_norm: 0.1
plasticity_grad_max_norm: 0.1
name: plasticine/dataset
seed: 0
cpu: 0
num_cpus: 128
gpu: 0
overwrite: false
resume: false
gt_name: null
dataset: true
Warp 0.6.1 initialized:
CUDA not available
Devices:
"cpu" | x86_64
Kernel cache: /home/xxxx/.cache/warp/0.6.1
Error executing job with overrides: ['overwrite=False', 'resume=False', 'gpu=0', 'cpu=0', 'env=plasticine', 'render=debug', 'sim=low', 'name=plasticine/dataset', 'dataset=True']
Traceback (most recent call last):
File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 137, in
main()
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in
lambda: hydra.run(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 35, in main
wp_device = wp.get_device(f'cuda:{cfg.gpu}')
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1676, in get_device
return runtime.get_device(ident)
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1484, in get_device
return self.device_map[ident]
KeyError: 'cuda:0'
env:
blob:
bsdf_pcd:
type: diffuse
reflectance:
type: rgb
value:
- 0.45490196
- 0.8
- 0.95686275
material:
elasticity:
cls: VolumeElasticity
E: 100000.0
nu: 0.3
mode: taichi
requires_grad: false
random: false
plasticity:
cls: SigmaPlasticity
requires_grad: false
random: false
name: water
ckpt: null
shape:
type: cube
name: dataset
center:
- 0.5
- 0.5
- 0.5
size:
- 0.5
- 0.5
- 0.5
resolution: 10
mode: uniform
sort: null
vel:
random: false
lin_vel:
- 1.0
- -1.5
- -2.0
ang_vel:
- 4.0
- 4.0
- 4.0
name: water
rho: 1000.0
span:
- 0
- 1000
clip_bound: 0.5
render:
spp: 32
width: 512
height: 512
skip_frame: 25
bound: 1.75
mpm_mul: 6
sph_version: cuda_ad_rgb
pcd_version: cuda_ad_rgb
has_sphere_emitter: true
fps: 10
sim:
quality: low
num_steps: 1000
gravity:
bc: freeslip
num_grids: 20
dt: 0.0005
bound: 3
eps: 1.0e-07
skip_frame: 1
train:
teacher:
strategy: cosine
start_lambda: 25
end_lambda: 200
num_epochs: 300
batch_size: 128
elasticity_lr: 1.0
plasticity_lr: 0.1
elasticity_wd: 0.0
plasticity_wd: 0.0
elasticity_grad_max_norm: 0.1
plasticity_grad_max_norm: 0.1
name: water/dataset
seed: 0
cpu: 0
num_cpus: 128
gpu: 0
overwrite: false
resume: false
gt_name: null
dataset: true
Warp 0.6.1 initialized:
CUDA not available
Devices:
"cpu" | x86_64
Kernel cache: /home/xxxx/.cache/warp/0.6.1
Error executing job with overrides: ['overwrite=False', 'resume=False', 'gpu=0', 'cpu=0', 'env=water', 'render=debug', 'sim=low', 'name=water/dataset', 'dataset=True']
Traceback (most recent call last):
File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 137, in
main()
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in
lambda: hydra.run(
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/home/xxxx/xxxx_research/NCLaw/experiments/eval.py", line 35, in main
wp_device = wp.get_device(f'cuda:{cfg.gpu}')
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1676, in get_device
return runtime.get_device(ident)
File "/home/xxxx/miniconda3/envs/nclaw/lib/python3.10/site-packages/warp/context.py", line 1484, in get_device
return self.device_map[ident]
KeyError: 'cuda:0'`
Btw, I'm sure the CUDA has been installed correctly follow your provided instruction, so could your give me some advices to tackle this issue? Maybe the warp is the trouble maker?
Looking forward to your reply!!!
The text was updated successfully, but these errors were encountered: