Warping Result

[zhangyahu1]

Hi @AliaksandrSiarohin,

May I ask a question about the warping result like this:

It seems the moved region is filled in the warping image (the third column) based on the estimated optical flow. I try to use the off-the-self model to predict the optical flow, however, the warping result have the issue about double region like this:

I would appreciate it if you can give me any hint to solve this problem.

[AliaksandrSiarohin]

Don't understand what you mean, but you should use occlusion mask to get rid of double regions.

[zhangyahu1]

For example, the arrow part has the right leg in the source image but the arrow apart in the warping result is filled with the background. However, the warping results for my implementation have double regions.

[zhangyahu1]

This is the code:

bsz, c, h, w = inp.size()
xx = torch.arange(0, w).view(1 ,-1).repeat(h ,1)
yy = torch.arange(0, h).view(-1 ,1).repeat(1 ,w)
xx = xx.view(1,1,h,w).repeat(bsz,1,1,1)
yy = yy.view(1,1,h,w).repeat(bsz,1,1,1)
grid = torch.cat((xx,yy), dim=1).float().cuda()
vgrid = grid + flow
# scale to [-1, 1]
vgrid[:,0,:,:] = 2.0 * vgrid[:,0,:,:].clone()/max(w-1,1) - 1.0
vgrid[:,1,:,:] = 2.0 * vgrid[:,1,:,:].clone()/max(h-1,1) - 1.0
vgrid = vgrid.permute(0,2,3,1)
warping = F.grid_sample(inp, vgrid, mode='bilinear', padding_mode='zeros', align_corners = True)

[AliaksandrSiarohin]

It is filled with some random staff nearby, the generator should not consider this part. Occlusion mask will instruct the generator to exclude this part from consideration.

[zhangyahu1]

Thanks!

[zhangyahu1]

So do you mean the part is filled with background by 'inacccurate' estimated optical flow (but may contribute to the final performance) instead of interpolation operation? If not, could you please tell me which part in your code does this operation?

[AliaksandrSiarohin]

Yes. Optical flow in these regions is undefined, this is just an accedent that it is decently warped.

[zhangyahu1]

I see. Thanks a lot!