Normal maps collapsed

[mbaradad]

Hi,
Why are normal maps in the paper so homogeneous (collapsed to a single color, corresponding to a planar and frontal depth map)?

The ones I get after backprojecting depth:

vs the ones in the paper:

I have had similar issues when depth is too small numerically and computing normals from depth/point-cloud. The issue in my case (which produced similar bad normal maps) was that when normalizing vectors before the cross product, I had an additive epsilon to avoid dividing over 0. This additive epsilon dominates if the depth values are small, so it requires adjusting the epsilon.

[Ilya-Muromets]

Yeah I think the scaling on my normal maps is pretty bad (as you can see the results are dominated by discontinuities along object edges). I tried to find a factor that highlighted the mm-scale features we recovered without amplifying all the single-pixel noise from color matching (all that grainy texture on the surface of the rocks you posted above).

Are you able to share your normal visualization code?

[mbaradad]

yes, the one I typically use is this one:

def compute_normals_from_closest_image_coords(coords, mask=None):
    assert coords.shape[1] == 3 and len(coords.shape) == 4
    assert mask is None or (mask.shape[0] == coords.shape[0] and mask.shape[2:] == coords.shape[2:])

    x_coords = coords[:,0,:,:]
    y_coords = coords[:,1,:,:]
    z_coords = coords[:,2,:,:]

    if type(coords) is torch.Tensor or type(coords) is torch.nn.parameter.Parameter:
      ts = torch.cat((x_coords[:, None, :-1, 1:], y_coords[:, None, :-1, 1:], z_coords[:, None,:-1,1:]), dim=1)
      ls = torch.cat((x_coords[:, None, 1:, :-1], y_coords[:, None, 1:, :-1], z_coords[:, None, 1:, :-1]), dim=1)
      cs = torch.cat((x_coords[:, None, 1:, 1:], y_coords[:, None, 1:, 1:], z_coords[:, None,1:,1:]), dim=1)

      n = torch.cross((ls - cs),(ts - cs), dim=1) * 1e10

      # if normals appear incorrect, it may be becuase of the 1e-20, if the scale of the pcl is too small,
      # it was giving errors with a constant of 1e-5 (and it was replaced to 1e-20). We also need an epsilon to avoid nans when using this function for training.
      n_norm = n/(torch.sqrt(torch.abs((n*n).sum(1) + 1e-20))[:,None,:,:])
    else:
      ts = np.concatenate((x_coords[:, None, :-1, 1:], y_coords[:, None, :-1, 1:], z_coords[:, None,:-1,1:]), axis=1)
      ls = np.concatenate((x_coords[:, None, 1:, :-1], y_coords[:, None, 1:, :-1], z_coords[:, None, 1:, :-1]), axis=1)
      cs = np.concatenate((x_coords[:, None, 1:, 1:], y_coords[:, None, 1:, 1:], z_coords[:, None,1:,1:]), axis=1)

      n = np.cross((ls - cs),(ts - cs), axis=1)
      n_norm = n/(np.sqrt(np.abs((n*n).sum(1) + 1e-20))[:,None,:,:])

    if not mask is None:
      assert len(mask.shape) == 4
      valid_ts = mask[:,:, :-1, 1:]
      valid_ls = mask[:,:, 1:, :-1]
      valid_cs = mask[:,:, 1:, 1:]
      if type(mask) is torch.Tensor:
        final_mask = valid_ts * valid_ls * valid_cs
      else:
        final_mask = np.logical_and(np.logical_and(valid_ts, valid_ls), valid_cs)
      return n_norm, final_mask
    else:
      return n_norm

[Ilya-Muromets]

Got it, thanks! I'll take a look at that for future visualization.
My visualized normals were simply a scaled depth gradient:

def get_normal(depth):
    zy, zx = np.gradient(depth)  

    normal = np.dstack((-zx, -zy, np.ones_like(depth)))
    n = np.linalg.norm(normal, axis=2)
    normal[:, :, 0] /= n
    normal[:, :, 1] /= n
    normal[:, :, 2] /= n

    # offset and rescale values to be in 0-255
    normal += 1
    normal /= 2
    normal *= 255
    
    return normal