You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Traceback (most recent call last):
File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/gpfs/home/jj323/infinigen/infinigen/datagen/manage_jobs.py", line 812, in <module>
main(args)
File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
raise proxy.with_traceback(exception.__traceback__) from None
File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/gpfs/home/jj323/infinigen/infinigen/datagen/manage_jobs.py", line 675, in main
manage_datagen_jobs(all_scenes, elapsed=(now-start_time).total_seconds())
File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
raise proxy.with_traceback(exception.__traceback__) from None
File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/gpfs/home/jj323/infinigen/infinigen/datagen/manage_jobs.py", line 603, in manage_datagen_jobs
LocalScheduleHandler.instance().poll()
File "/gpfs/home/jj323/infinigen/infinigen/datagen/util/submitit_emulator.py", line 181, in poll
total = self.total_resources()
File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
raise proxy.with_traceback(exception.__traceback__) from None
File "/gpfs/home/jj323/mambaforge-pypy3/envs/infini/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/gpfs/home/jj323/infinigen/infinigen/datagen/util/submitit_emulator.py", line 158, in total_resources
visible = [int(s.strip()) for s in os.environ[CUDA_VARNAME].split(',')]
File "/gpfs/home/jj323/infinigen/infinigen/datagen/util/submitit_emulator.py", line 158, in <listcomp>
visible = [int(s.strip()) for s in os.environ[CUDA_VARNAME].split(',')]
ValueError: invalid literal for int() with base 10: 'GPU-c5aa4d0e-51b8-94f4-8799-3f17f14b64ea'
In call to configurable 'total_resources' (<function LocalScheduleHandler.total_resources at 0x14f7115e6c20>)
In call to configurable 'manage_datagen_jobs' (<function manage_datagen_jobs at 0x14f7113f7370>)
In call to configurable 'main' (<function main at 0x14f7113f7250>)
I tried to comment the code in infinigen/infinigen/datagen/util/submitit_emulator.py from line 163~166 (link) and it fixed.
I wonder if this is a correct solution.
The text was updated successfully, but these errors were encountered:
It seems your CUDA_VISIBLE_DEVICES contains long strings like GPU-c5aa4d0e-51b8-94f4-8799-3f17f14b64ea where I (incorrectly) had assumed it was only integers. Could you try removing the int(..) from int(s.strip()) so it just looks like [s.strip() for s in os.environ[CUDA_VARNAME].split(',')]? If this works for you I will patch it into main. I should probably find and follow some kind of specification for what these are allowed to look like.
python -m infinigen.datagen.manage_jobs --output_folder outputs/my_videos --num_scenes 50 --pipeline_config local_128GB.gin monocular_video cuda_terrain opengl_gt --cleanup big_files --warmup_sec 60000 --config video high_quality_terrain
I tried to comment the code in infinigen/infinigen/datagen/util/submitit_emulator.py from line 163~166 (link) and it fixed.
I wonder if this is a correct solution.
The text was updated successfully, but these errors were encountered: