RIFE 4.0 causes tensor size mismatch errors for some resolutions

[n00mkrad]

Interpolating 720p or 1440p video with RIFE 4.0 throws an error after some frames:

File "D:\Code\GitHub\flowframes\Code\bin\x64\Release\FlowframesData\pkgs\rife-cuda\arch\RIFE_HDv3.py", line 59, in inference
flow, mask, merged = self.flownet(imgs, timestep, scale_list)
File "D:\Software\Python38\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Code\GitHub\flowframes\Code\bin\x64\Release\FlowframesData\pkgs\rife-cuda\arch\IFNet_HDv3.py", line 99, in forward
f0, m0 = block[i](torch.cat((warped_img0[:, :3], warped_img1[:, :3], timestep, mask), 1), flow, scale=scale_list[i])
RuntimeError: Sizes of tensors must match except in dimension 2. Got 768 and 736 (The offending index is 2)

1080p works fine, it seems to be some scaling/cropping issue?

I can reliably reproduce this problem on 4.0, but not on 3.9, so it seems to be caused by IFNet_HDv3.py changes.

[hzwer]

Practical-RIFE/inference_video.py

Line 196 in 66a4e29

tmp = max(128, int(128 / args.scale))

Increasing this padding number to 128 can fix this issue.

[SportsmanLee]

Hi, is this RIFE 4.0 model trained with perceptual loss? Thx.

[hzwer]

@SportsmanLee Yes, 0.5 l1 loss + 0.5 perceptual loss ->

Practical-RIFE/model/loss.py

Line 98 in 66a4e29

class VGGPerceptualLoss(torch.nn.Module):