No normal gradient (and others I believe) from DSS class

[bango123]

I am interested in using this repo for my research and have been digging through the DSS rendering code. I found that when using the SmapeLoss in a toy rendering problem with a single point and single normal (similar to Figure 6 of your paper) the gradient of the loss with respect to the normal came back as None. After some more digging, I notice that in the render() function line 855-859 the projPoints, cameraPoints, and cameraNormals have their gradients detached when computing rho. Therefore, it looks like the partial derivatives drho/dn and drho/dp from equations 8 and 9 of your paper are not being computed.

Also I dug into the DSS rasterize backward function and it looks like dRho is always returned as None. It looks like only dI/dw and (dI/dh)*(dh/dp) from equations 8 and 9 are returned.

It would be very helpful if I could get assistance with recovering these gradients or clarifying my understanding of the code.

Thank you!

[Wu-Xiuchao]

Hi, I have same problem as you. And I found another problem https://github.com/yifita/DSS/issues/9

[bango123]

I implemented my own dRho computation. I cannot guarantee it is correct, but it hasn't messed anything up for me so far. I replaced lines 276-287 from the rasterize function with this:

        if ctx.needs_input_grad[0]:
            # dI/drho(x) = ( WsMap_ - pixels*h)/(sumRho)
            # Note that this gradient is back-proped to rho, not \overline{rho} (like the paper says)
            # This can be done because computationally the same gradient would occur since the
            # gradient for \overline{rho} at pixels outside the ellipse of rho are all 0.
            WsMap_ = torch.where(isBehind.unsqueeze(-1),
                                 torch.zeros(1, 1, 1, 1, 1, device=WsMap.device, dtype=WsMap.dtype),
                                 WsMap)
            h = (rhoMap_filtered > 0).double() # Easy way to get h function!
            dRho = (WsMap_ - pixels.unsqueeze(3).repeat(1, 1, 1, 5, 1)*h.unsqueeze(-1))/sumRho.unsqueeze(-1)
            dRho = gradPixels.unsqueeze(3)*dRho

            # Invert the mapping so the gradient is back-propped to appropriate BB rho
            # rhoMap = _gather_maps(rho.reshape(batchSize, -1, 1), totalIdxMap, 0.0).squeeze(-1)
            # BATCHES MUST HAVE POINTS IN THE SAME ORDER... OTHERWISE THIS RE-ORDER WILL NOT WORK
            dRho = _guided_scatter_maps(numPoint*bbWidth*bbHeight, dRho,
                                        totalIdxMap, torch.repeat_interleave(boundingBoxes, bbWidth*bbHeight, 1))
            dRho = dRho.reshape(batchSize, -1, bbHeight, bbWidth)
        else:
            dRho = None

Then I removed all "detach()" when computing rho. Hope this helps others!

[yifita]

Thank you @bango123 !
I'm curious about your motivation to derive \rho or \bar{\rho}? Do you want to further derive \rho w.r.t. point positions and normals?

[bango123]

Hello @yifita! Thank you for releasing this code base and your paper is really high quality btw.

I am doing a research project which is reconstructing from point based geometry and your work seemed like a perfect fit for it. So I put in the time to really understand your work and code implementation.

From your paper, it looks like these are the derivatives wanted (after approximating dh/dn = 0):

And from my understanding of your code, this function (DSS rasterize) is where dI/dw, dI/ \bar{rho}, dI/h are all computed. I noticed dI/ \bar{rho} is never computed, so I tried my hand at implementing it (code above) in an effort to better understand how things work.

However, as you probably already know, it looks like adding \bar{rho} w.r.t to points and normals (i.e. dI/ \bar{rho}) makes very little difference to the gradients. I am guessing that is because your approximated gradient for dI/dh is so much larger in magnitude.

tl;dr wanted to learn about your code/paper, so I tried to fix something. The "fix" appears to make little difference.