Optimization using depth AOV not converging

[fnoriegaz]

First, apologies since I have basic understanding of computer graphics and i might misuse some terms. Though I try to be as clear as possible.
I want to run the following workflow:

• Create a init scene with a deformed version of the cube in blender. Export the scene using mitsuba add-on for blender
• Create a gt scene( basically the state i want to reach with optimization ). I reuse the same elements and the mesh is now rotated just for having a simple example that should work in principle.
• In blender, I set also the render engine to mitsuba and configure aov with depth as integrator before exporting to XML file.
• With a python script, I load both scenes and run an optimization problem using a rotation as latent variable and mi.ad.Adam as optimizer.

This is the python script:

import drjit as dr
import mitsuba as mi
import json


mi.set_variant("cuda_ad_rgb")


gt_scene = mi.load_file("gt_cube.xml")
init_scene = mi.load_file("init_cube.xml")

gt_params = mi.traverse(gt_scene)
init_params = mi.traverse(init_scene)

print(f"{init_params=}")

n_cams = 6
gt_renders = []
for i in range(n_cams):
    render = mi.render(scene=gt_scene, params=gt_params, sensor=i)
    mi.util.write_bitmap(f"gt_cam_{i}.exr", render)
    gt_renders.append(render)
    render = mi.render(scene=init_scene, params=init_params, sensor=i)
    mi.util.write_bitmap(f"init_cam_{i}.exr", render)

init_vertices = dr.unravel(mi.Vector3f, init_params["elm__4.vertex_positions"])

opt = mi.ad.Adam(lr=0.1)
opt["rot"] = mi.Vector3f(90,0,0)
dr.enable_grad(opt["rot"])

iterations = 200
for it in range(iterations):
    #T = mi.Transform4f().rotate([1,0,0], opt["rot"][0]).rotate([0,1,0], opt["rot"][1]).rotate([0,0,1], opt["rot"][2])
    T = mi.Transform4f().rotate([1,0,0], opt["rot"][0])

    init_params["elm__4.vertex_positions"] = dr.ravel(T @ init_vertices)
    init_params.update()

    total_loss = 0
    idx = f"{it}".zfill(5)
    for i in range(n_cams):
        render = mi.render(scene=init_scene, params=init_params, sensor=i)
        mi.util.write_bitmap(f"modeled_cam_{i}_{idx}.exr", render)
        loss = dr.mean(dr.square(gt_renders[i] - render))
        total_loss += loss

    dr.backward(total_loss)

    opt.step()

    print(f"{it=}, {loss=}")

When running the mentioned script i see that the error goes from:
it=0, loss=1.78326
to:
it=199, loss=2.04629

Finally, I need to mention that if instead of AOV, I use other inegrators like direct or direct projective, the optimization works and the error is reduced up to one order of magnitude. Therefore, I believe that the implementation of the script could be correct.

The main questions after this description would be:

1. Are the integrator AOV + Depth mean to be used the way I want? i.e. Optimize mesh rotation(and potentially translation)
2. Is there something obvious that I am missing so my optimization is not converging? e.g. My loss function only works for RGB images but not for Depth data?

Any hint that could lead me to solve or understand betteer the issue would be super appreciated.
Thanks in advance!

[njroussel]

Hi @fnoriegaz

Even if you provide depth information, there are still discontinuities in the pixel integrals which requires using one of the *_projective integrators. See the top of this tutorial and the papers that it references for more information about visibility discontinuities: https://mitsuba.readthedocs.io/en/stable/src/inverse_rendering/object_pose_estimation.html

I am surprised that simply using direct works at all, given what I said above.

In theory, depth information could be used. But the code would need to apply the same algorithms we have in *_projective integrators to the aov integrator too. Although theoretically simple, the implementation would require a rather large refactor.