Geometry buffer script throwing error

[darthgera123]

Hi i am trying to run the geometry buffer script on my dataset and its firstly taking 2-3 hrs to show any output and is then throwing errors.
My command:
bash geometry_from_nerf_run.sh 0 --data_root="/scratch/darthgera123/nerf/woman_data/" --trained_nerf="/scratch/darthgera123/nerf/woman_nerf/lr5e-4/" --out_root="/scratch/darthgera123/nerf/woman_geometry/" --imh=512 --scene_bbox=-0.3,0.3,-0.3,0.3,-0.3,0.3 --occu_thres=0.5 --mlp_chunk=3750
The error:

2021-08-09 16:04:55.509013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10210 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:03:00.0, compute capability: 7.5)
2021-08-09 16:04:55.512131: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55f55866a9d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-08-09 16:04:55.512172: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
Views (train):   0%|          | 0/304 [00:00<?, ?it/s]2021-08-09 16:05:05.594129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-08-09 17:51:19.840422: E tensorflow/core/grappler/clusters/utils.cc:87] Failed to get device properties, error code: 999

As mentioned I initially ket the mlp_chunk high till I got OOM error but now its throwing this weird error. Please help @xiumingzhang @cdibona @dberlin

[xiumingzhang]

Sorry for the delayed reply. Looks like a CUDA issue to me. Have you tried making a fresh conda environment using the provided `environment.yml`?

…

On Mon, Aug 9, 2021 at 9:01 AM Pulkit gera ***@***.***> wrote: Hi i am trying to run the geometry buffer script on my dataset and its firstly taking 2-3 hrs to show any output and is then throwing errors. My command: bash geometry_from_nerf_run.sh 0 --data_root="/scratch/darthgera123/nerf/woman_data/" --trained_nerf="/scratch/darthgera123/nerf/woman_nerf/lr5e-4/" --out_root="/scratch/darthgera123/nerf/woman_geometry/" --imh=512 --scene_bbox=-0.3,0.3,-0.3,0.3,-0.3,0.3 --occu_thres=0.5 --mlp_chunk=3750 The error: 2021-08-09 16:04:55.509013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10210 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:03:00.0, compute capability: 7.5) 2021-08-09 16:04:55.512131: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55f55866a9d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-08-09 16:04:55.512172: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5 Views (train): 0%| | 0/304 [00:00<?, ?it/s]2021-08-09 16:05:05.594129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2021-08-09 17:51:19.840422: E tensorflow/core/grappler/clusters/utils.cc:87] Failed to get device properties, error code: 999 As mentioned I initially ket the mlp_chunk high till I got OOM error but now its throwing this weird error. Please help @xiumingzhang <https://github.com/xiumingzhang> @cdibona <https://github.com/cdibona> @dberlin <https://github.com/dberlin> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXICPQAU44ZNGYQMVYPXOTT37GT7ANCNFSM5BZ7DHCA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[xiumingzhang]

Closing due to no response. Please reopen this if the problem persists.