Issue related to paper published and implementation

[vinayver198]

Hi @lahavlipson ,
I was able to run demo on my dataset and the results were amazing. Thanks for this wornderful work . I was trying to understand the codebase and the paper. I came across some i,j,k conventions . In paper it is mentioned as i = patch index, k= other frame, and j = source frame whereas in code k = patch index, i = source frame and j = other frame. I have added below snippet of code and paper reference. Can you please clarify whether I have understood wrong or there is some difference.

Thanks,
Vinay

Code Snippet

def transform(poses, patches, intrinsics, ii, jj, kk, depth=False, valid=False, jacobian=False, tonly=False):
    """ projective transform """

    # backproject
    X0 = iproj(patches[:,kk], intrinsics[:,ii])

    # transform
    Gij = poses[:, jj] * poses[:, ii].inv()

    if tonly:
        Gij[...,3:] = torch.as_tensor([0,0,0,1], device=Gij.device)

    X1 = Gij[:,:,None,None] * X0

    # project
    x1 = proj(X1, intrinsics[:,jj], depth)


    if jacobian:
        p = X1.shape[2]
        X, Y, Z, H = X1[...,p//2,p//2,:].unbind(dim=-1)
        o = torch.zeros_like(H)
        i = torch.zeros_like(H)

        fx, fy, cx, cy = intrinsics[:,jj].unbind(dim=-1)

        d = torch.zeros_like(Z)
        d[Z.abs() > 0.2] = 1.0 / Z[Z.abs() > 0.2]

        Ja = torch.stack([
            H,  o,  o,  o,  Z, -Y,
            o,  H,  o, -Z,  o,  X, 
            o,  o,  H,  Y, -X,  o,
            o,  o,  o,  o,  o,  o,
        ], dim=-1).view(1, len(ii), 4, 6)
        
        Jp = torch.stack([
             fx*d,     o, -fx*X*d*d,  o,
                o,  fy*d, -fy*Y*d*d,  o,
        ], dim=-1).view(1, len(ii), 2, 4)

        Jj = torch.matmul(Jp, Ja)
        Ji = -Gij[:,:,None].adjT(Jj)
        
        Jz = torch.matmul(Jp, Gij.matrix()[...,:,3:])

        return x1, (Z > 0.2).float(), (Ji, Jj, Jz)

    if valid:
        return x1, (X1[...,2] > 0.2).float()
        
    return 

Paper ref :

[lahavlipson]

Hi @vinayver198, you are correct - this is a notation difference between the paper and code. Sorry for the confusion.

We have an updated paper draft with notation that reflects the code and will update the version on Arxiv shortly.

[vinayver198]

Hi @lahavlipson , thanks for the clarification.

I have one more doubt related to patch graph. I am trying to understand the complete flow of the code . I am stuck in understanding the way forward and backward edges are working . Actually I wanted to visualise the connections how they form a graph. I have provided some screenshots from which I am trying to understand . It would be great help if you can provide any explanation or reference to understand them.

Thanks,
Vinay

[lahavlipson]

Here is a basic illustration of the patch graph: patch_graph.pdf

Each time a new frame is popped from the queue, its patches are given 'backward' edges to the previous frames (e.g. purple to blue, in the illustration).

At the same timestep, patches from previous frames are also given 'forward' edges to the new frame (e.g. blue to purple).

[vinayver198]

Hi @lahavlipson ,

Thanks a lot . It was of great help. Now I can visualise the graph clearly.

Thanks,
Vinay