How to generate videos without camera moving automatically.

[bugByteMe]

I want to generate video with a fixed camera position and direction. But infinigen automatically moves camera.

Is there an option to make the camera fixed with the position and direction defined in the camera rig config?

Thanks !

[araistrick]

Hello, you can add -p compose_scene.animate_cameras_enabled=False or just comment out the animate cameras part of generate_nature.py. I dont believe theres a commandline config for setting a particular camera pose, but you can just add code to generate_nature.py to modify the positions.

For example you can comment out our pose_cameras code, then add something like:

cam_rigs[0].location = (1, 2, 3)
cam_rigs[0].rotation_euler = np.deg2rad([90, 0, 30])

[bugByteMe]

@araistrick Thanks for your detailed reply! This is a quite interesting application. I'm exploring it!

[bugByteMe]

@araistrick I have a another problem here. By setting -p compose_scene.animate_cameras_enabled=False, when I generate a stereo video, the output folder /Image only contains camera_0, without other cameras in the camera rig. Before setting this to false, it generates stereo video correctly.

It seems that I should comment of these lines to disable camera animation?
p.run_stage('animate_cameras', lambda: cam_util.animate_cameras( camera_rigs, scene_preprocessed, pois=pois), use_chance=False)

[araistrick]

Could you provide the exact command you ran, and the output logs?

[bugByteMe]

I have tried to run render task separately, with the following command.

python -m infinigen_examples.generate_nature -- --input_folder \
/mnt/remote/D002/home/jrguo/outputs/water_video_3/4b0d6f42/fine --output_folder \
/mnt/remote/D002/home/jrguo/outputs/water_video_3/4b0d6f42/frames_0_0_0001_0 --seed 4b0d6f42 --task render \
--task_uniqname renderbackup_0_0_0001_0 -g forest under_water.gin simple.gin video.gin \
-p render.render_image_func=@full/render_image \
LOG_DIR='/mnt/remote/D002/home/jrguo/outputs/water_video_3/4b0d6f42/logs' \
execute_tasks.frame_range=[1,1] execute_tasks.camera_id=[1,1] execute_tasks.resample_idx=0

And got the error:

File "/mnt/data/interns/gid-guojiarui/infinigen/infinigen/core/execute_tasks.py", line 330, in execute_tasks
    camera = cam_util.set_active_camera(*camera_id)
  File "/mnt/data/interns/gid-guojiarui/infinigen/infinigen/core/placement/camera.py", line 171, in set_active_camera
    camera = get_camera(rig_id, subcam_id)
  File "/mnt/data/interns/gid-guojiarui/infinigen/infinigen/core/placement/camera.py", line 161, in get_camera
    raise ValueError(f'Could not get_camera({rig_id=}, {subcam_id=}). {list(col.objects.keys())=}')
ValueError: Could not get_camera(rig_id=1, subcam_id=1). list(col.objects.keys())=['CameraRigs/0', 'CameraRigs/0/0', 'CameraRigs/0/1']
  In call to configurable 'execute_tasks' (<function execute_tasks at 0x7f0d77e189d0>)

So I think this is the reason why it can't generate other camera views.
It can't get cam_id 1 (when setting camera_id to [0,0], everything is fine), but in base.gin, there are actually two camera configs.

[araistrick]

You should use camera id (0,1). This will render the second "eye" of the 0th stereo camera. 1,1 would be the second eye of the second stereo camera, which doesn't exist in your scene

[bugByteMe]

Oh, I thought this was an interval. I checked camera.py for a better understanding. Thanks!

[araistrick]

Sorry yeah it is actually a max for two different python range()s. It really needs splitting up into two well named variables with docs etc

[bugByteMe]

I have another question about camera here.
In base.gin, I found this camera rig config.

camera.spawn_camera_rigs.camera_rig_config = [
    {'loc': (0, 0, 0), 'rot_euler': (0, 0, 0)},
    {'loc': (0.075, 0, 0), 'rot_euler': (0, 0, 0)}
]

This configures cameras in one camera rig. Then how to declare configs for other camera rigs?
I want to define cameras with large camera rig translations or rotations. It is recommended in base.gin that I should treat my camera rig as multiple camera rigs each with one camera. Is this possible to realize in .gin file or I should change the code?
By the way, what's the unit of loc.
Thanks!

[araistrick]

Our default code has all cameras use the same camera rig config. So if there are 5 stereo camera rigs they will all have the same baseline. If you want to control the position / rotation of these rigs you will need to add python code like I showed above to set their '.location' and '.rotation_euler' via the blender API. All locations are ~meters, all rotations are radians

[bugByteMe]

Thanks! I'll work on this.

[bugByteMe]

By setting camera rig on myself, I've got assertion errors in the rendering stage in postprocessing part.

[15:06:49.157] [infinigen.times] [INFO] | [Post Processing]
[15:06:49.157] [infinigen.times] [INFO] | [Post Processing] failed with <class 'AssertionError'>
[15:06:49.157] [infinigen.times] [INFO] | [Render Frames] failed with <class 'AssertionError'>
[15:06:49.157] [infinigen.times] [INFO] | [MAIN TOTAL] failed with <class 'AssertionError'>
Traceback (most recent call last):
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/guojiarui/infinigen/infinigen_examples/generate_nature.py", line 446, in <module>
    main(args)
  File "/home/guojiarui/infinigen/infinigen_examples/generate_nature.py", line 417, in main
    execute_tasks.main(
  File "/home/guojiarui/infinigen/infinigen/core/execute_tasks.py", line 418, in main
    execute_tasks(
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/guojiarui/infinigen/infinigen/core/execute_tasks.py", line 380, in execute_tasks
    render(
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/guojiarui/infinigen/infinigen/core/execute_tasks.py", line 227, in render
    render_image_func(frames_folder=Path(output_folder), camera_id=camera_id)
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 670, in scoping_wrapper
    return fn_or_cls(*args, **kwargs)
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/guojiarui/miniconda3/envs/infinigen/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/guojiarui/infinigen/infinigen/core/rendering/render.py", line 367, in render_image
    camera_ids=cam_util.get_cameras_ids(),
  File "/home/guojiarui/infinigen/infinigen/core/placement/camera.py", line 149, in get_cameras_ids
    assert subcam.name == camera_name(i, j)
AssertionError: 
  In call to configurable 'render_image' (<function render_image at 0x718881cd8670>) in scope 'full'
  In call to configurable 'render' (<function render at 0x718881cd9750>)
  In call to configurable 'execute_tasks' (<function execute_tasks at 0x718881cd9f30>)

My code modifications are (after pose cameras in generate_nature.py):

    loc_base = camera_rigs[0].location
    rot_base = camera_rigs[0].rotation_euler
    for index, cam_rig in enumerate(camera_rigs):
        cam_rig.location = (loc_base[0] - index*0.1, loc_base[1], loc_base[2])
        cam_rig.rotation_euler = (rot_base[0], rot_base[1], rot_base[2])

In .gin, I configure:

camera.spawn_camera_rigs.n_camera_rigs = 4

I run with the command:

python -m infinigen.datagen.manage_jobs --output_folder /home/guojiarui/gen/under_water_syn_mono2 --num_scenes 1 \
--configs under_water.gin simple.gin \
--pipeline_configs local_32GB.gin monocular_video.gin \
--pipeline_overrides LocalScheduleHandler.use_gpu=False \
-p compose_scene.max_fish_schools=5 compose_scene.animate_cameras_enabled=False execute_tasks.camera_id=[3,0]

I didn't figure out why this assertion fails.

[araistrick]

Please try again on main, 1.3.3 contains some code which may (?) fix this although it was not written by me so I am not fully positive. If it does not work, please provide me code/commands to reproduce your issue