[🐛 bug report] Unable to run differentiable rendering example with spectral mode

[vucenovic]

Hello, I am trying to run the cbox example given in the docs, and changed the mitsuba2 variant to spectral as I want to work with this variant. Neither my own, nor the cbox example are running in this variant and I get the following error when trying to run both:

cuda_check(): driver API error = 0700 "CUDA_ERROR_ILLEGAL_ADDRESS" in D:/BAC/mitsuba2/ext/enoki/src/cuda/jit.cu:1319.

I am kinda stuck to as where the error might come from, as the underlying code does not give me enough information.

[merlinND]

Hi @vucenovic,

By any chance, does it work if you lower the resolution and spp significantly? At some point I remember that OOM errors could manifest as a segfault.

[vucenovic]

Hi @merlinND ,

Thanks for the quick answer. When i lower the spp to 1, and the film's resolution in the provided .xml file to 64x64, i get the following error, a bit further down in the jit.cu file:

cuda_check(): runtime API error = 0700 "cudaErrorIllegalAddress" in D:/BAC/mitsuba2/ext/enoki/src/cuda/jit.cu:1680.

[vucenovic]

Hello, sorry to bother, but are there any ideas why this could happen? Has the spectral_differentiable mode been tested successfully on your side? I tried to run my example with the lowest possible samples, resolution, and only 10 frequencies, and it still had the same error.

[merlinND]

Hi @vucenovic,

I have just tried this without issues from the current master state:

mitsuba -m gpu_autodiff_spectral ../resources/data/scenes/cbox/cbox.xml
python3 ../docs/examples/10_inverse_rendering/invert_cbox.py

It worked without crashing after some small tweaks to the invert_cbox.py to target a spectral color instead of RGB.

Could you try with the default 4 spectral bands instead of 10?

[vucenovic]

Could you tell me what your tweaks were, so we can be sure we are running the same setup?

[vucenovic]

I managed now to get the example running with the current master branch. It seems that our fork is running into a problem..

[Speierers]

Hi @vucenovic,
If this is a problem on your end, I suppose this issue can be closed?

[vucenovic]

Hi @Speierers , yes, i also got the fork running now by merging the latest commits into it. I am a bit baffled though, what has changed since november, that could have resolved the issue?

Also I would still like to see the code tweaked for running the example, as i get this now: RuntimeError: cuda_trace_append(): arithmetic involving arrays of incompatible size (76 and 3). The instruction was "sub.rn.ftz.$t1 $r1, $r2, $r3".

[merlinND]

Hi @vucenovic,

Here's an updated invert_cbox.py script. It just needed a couple changes in places that were hardcoded for RGB:

Expand

# Simple inverse rendering example: render a cornell box reference image,
# then replace one of the scene parameters and try to recover it using
# differentiable rendering and gradient-based optimization.

import os
from os.path import join, dirname, realpath

import enoki as ek
import mitsuba

USE_SPECTRAL = True
mitsuba.set_variant('gpu_autodiff_spectral' if USE_SPECTRAL else 'gpu_autodiff_rgb')

from mitsuba.core import Thread, Float, Color3f
from mitsuba.core.xml import load_file
from mitsuba.python.util import traverse
from mitsuba.python.autodiff import render, write_bitmap, Adam
import time

# Load the Cornell Box
extra = realpath(join(dirname(__file__), '../../../resources/data/scenes/cbox'))
fr = Thread.thread().file_resolver()
fr.append(extra)
scene = load_file(str(fr.resolve('cbox.xml')))

# Find differentiable scene parameters
params = traverse(scene)

# Discard all parameters except for one we want to differentiate
key = 'red.reflectance.values' if USE_SPECTRAL else 'red.reflectance.value'
params.keep([key])

# Print the current value and keep a backup copy
param_tp = type(params[key])
param_ref = param_tp(params[key])
print('Reference value:', param_ref)

# Render a reference image (no derivatives used yet)
image_ref = render(scene, spp=8)
crop_size = scene.sensors()[0].film().crop_size()
write_bitmap('out_ref.png', image_ref, crop_size)

# Change the left wall into a bright white surface
params[key] = ek.full(param_tp, 0.9, ek.slices(params[key]))
params.update()

# Construct an Adam optimizer that will adjust the parameters 'params'
opt = Adam(params, lr=(1e-2 if USE_SPECTRAL else 2e-1))

time_a = time.time()

iterations = 100
for it in range(iterations):
    # Perform a differentiable rendering of the scene
    image = render(scene, optimizer=opt, unbiased=True, spp=1)
    write_bitmap('out_%03i.png' % it, image, crop_size)

    # Objective: MSE between 'image' and 'image_ref'
    ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)

    # Back-propagate errors to input parameters
    ek.backward(ob_val)

    # Optimizer: take a gradient step
    opt.step()

    # Enforce a valid range
    if USE_SPECTRAL:
        # Currently there's an exception thrown in distr_1d as soon as
        # any spectral coefficient goes below zero. It may be more
        # convenient to disable that exception if the constraint
        # is enforced manually.
        params[key] = ek.max(params[key], 1e-2)
        params.update()

    # Compare iterate against ground-truth value
    err_ref = ek.hsum(ek.sqr(param_ref - params[key]))
    # print('Iteration %03i: error=%g' % (it, err_ref[0]), end='\r')
    print('Iteration %03i: error=%g' % (it, err_ref[0]))

time_b = time.time()

print()
print('%f ms per iteration' % (((time_b - time_a) * 1000) / iterations))

One detail: currently there's an exception thrown in distr_1d as soon as any spectral coefficient goes below zero. It may be more convenient to disable that exception if the constraint is enforced manually.
To avoid triggering that exception I didn't use the right LR and clamping parameters, if you handle that you can get full convergence.

[vucenovic]

Thank you!

[vucenovic]

Hello, I am trying to replicate the example with my own scene and parameter. The code is kept the same and I get the following message: RuntimeError: set_gradient(): no gradients are associated with this variable (a prior call to requires_gradient() is required.) when calling ek.backwards(ob_val). What could be the reason for this? The forward propagation doc also uses this call.

[vucenovic]

        self.params[param] = ek.full(param_tp, val, ek.slices(self.params[param]))
        self.params.update()

        opt = Adam(self.params, lr=1e-2)

        time_a = time.time()

        filename_output.replace("Reference", "Iteration")

        for it in range(0, iterations):
            image = render(self.mitsuba_scene, spp=spp, optimizer=opt, unbiased=True)
            write_bitmap(self.render_directory + "/" + filename_output + ("_%i" % it) + ".png", image_ref, crop_size)
            ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)
            ek.backward(ob_val)
            opt.step()

            err_ref = ek.hsum(ek.sqr(param_ref -self.params[param]))
            print('Iteration %03i: error=%g' % (it, err_ref[0]))

Here is the code snippet

[Speierers]

Hi @vucenovic,

Do you run param.keep() at some point to make sure that it only contains the differentiable parameters that you are looking to optimize?

It would be easier to understand the problem here if you share the whole script 😉

[vucenovic]

Hello, yes i did, i checked all the params with is_differentiable and kept those. Now i modified my script to only keep the parameter in question, here is the full function and setup of params:

Function

 def render_differentiable_with_iterations(self, param, val, spp=1, pixels=2048, sampler="independent", suffix=None, iterations=100):
        param_0 = self.params[param]
        param_tp = type(self.params[param])
        param_ref = param_tp(self.params[param])
        self.params.keep([param])

        # Because we reference the Meshes via relative path from the xml file, tell Mitsuba where to look for those files
        Thread.thread().file_resolver().append(os.path.dirname(self.scenefile))
        
        # Load the Scene file and pass the parameters to the xml file
        self.mitsuba_scene = load_file(self.scenefile, rot=self.rotation, od=self.distance_to_object, dd=self.distance_to_camera, sd=self.distance_to_arealight, md=self.max_depth, rr=self.rr_depth,
                                       inte=self.integrator, os=self.object_size, ds=self.detector_size, spp=spp, px=pixels, fc=self.distance_to_far,
                                       fov=self.fov, matobj=self.object_spectra, scalematobj=self.object_spectra_scale, spectra_area=self.arealight_spectra, stype=sampler, fd=self.focus_distance)
        
        # Get the sensor from our scene
        detector = self.mitsuba_scene.sensors()[0]

        tic = time.perf_counter()
        image_ref = render(self.mitsuba_scene, spp=spp)

        film = detector.film()
        crop_size =film.crop_size()

        filename_output = "sphereReference_" + str(pixels) + "_" + str(spp) + "_" + str(
            self.SOD) + "_" + str(sampler) + "_" + str(self.rotation)
        if suffix is not None:
            filename_output = filename_output + "__" + suffix
        write_bitmap(self.render_directory + "/" + filename_output + ".png", image_ref, crop_size)
        toc = time.perf_counter()

        elapsed_time = toc-tic

        print("Elapsed time : {}".format(elapsed_time))

        self.rendered_scenes.append(
            (self.render_directory + filename_output + '.png', elapsed_time))

        self.params[param] = ek.full(param_tp, val, ek.slices(self.params[param]))
        self.params.update()

        opt = Adam(self.params, lr=1e-2)

        time_a = time.time()

        filename_output.replace("Reference", "Iteration")

        for it in range(0, iterations):
            image = render(self.mitsuba_scene, spp=spp, optimizer=opt, unbiased=True)
            write_bitmap(self.render_directory + "/" + filename_output + ("_%i" % it) + ".png", image_ref, crop_size)
            ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)
            ek.backward(ob_val)
            opt.step()

            err_ref = ek.hsum(ek.sqr(param_ref -self.params[param]))
            print('Iteration %03i: error=%g' % (it, err_ref[0]))
        time_b = time.time()

        print()
        print('%f ms per iteration' % (((time_b - time_a) * 1000) / iterations))

Setup of Params

def find_differentiable_params(self):
        """
        Find differentiable scene parameters
        """
        if(self.mitsuba_scene is None):
            print("No Scene set, cannot find differentiable params.\nMake sure to call setup_scene first!")
        self.params = traverse(self.mitsuba_scene)
        print("The following scene parameters have been found: \n")
        print(self.params)
        print("\n")
        print("The following scene parameters can be differentiated: \n")
        params_to_keep = []
        for k in self.params.keys():
            if is_differentiable(self.params[k]):
                params_to_keep.append(k)
                print(k)
        self.params.keep(params_to_keep)

[Speierers]

What is the param you are optimizing for here?

Also could you try to explicitly call ek.set_require_gradient(self.params[param]) before calling self.params.update()?

[vucenovic]

hello, trying to run this line gives me: RuntimeError: set_gradient(): no gradients are associated with this variable (a prior call to requires_gradient() is required.) '. And I am trying to work with the 'Rectangle.emitter.radiance.values' parameter.

[Speierers]

I think you misstyped the function. You should call set_requires_gradient and not set_gradient

[vucenovic]

Here is the snippet:

        ek.set_requires_gradient(self.params[param])
        self.params.update()

        opt = Adam(self.params, lr=1e-2)

        time_a = time.time()

        filename_output.replace("Reference", "Iteration")

        for it in range(0, iterations):
            image = render(self.mitsuba_scene, spp=spp, optimizer=opt, unbiased=True)
            write_bitmap(self.render_directory + "/" + filename_output + ("_%i" % it) + ".png", image_ref, crop_size)
            ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)
            ek.backward(ob_val)
            opt.step()

            err_ref = ek.hsum(ek.sqr(param_ref -self.params[param]))
            print('Iteration %03i: error=%g' % (it, err_ref[0]))
        time_b = time.time()

        print()
        print('%f ms per iteration' % (((time_b - time_a) * 1000) / iterations))

Unfortunately it does throw the above error when calling ek.backward(ob_val) again.

[Speierers]

It would be great to be able to see the whole scene XML to understand exactly what is the parameter you are trying to optimize for. (e.g. regular or irregular spectrum)

[vucenovic]

I have resolved the issue now, it seems that there was something wrong with the order of how I called the functions for traversing the scene and loading it. Thanks for the help!