Artefacts described in this ticket are improved in SVP with Rife v4.15 but still present

[netExtra]

When using SVP, Rife tends to have less artefacts than the other mode. I notice that Rife 4.7 has even less artefacts than 4.6. But the problem is that the artefacts that remain tend to be worse. The biggest one I've seen recently is a character appearing and disappering every other frame. Is there any use for me to point out the location of these artefacts ? Is there anyone I should send them to?

[jensdraht1999]

Hello,

I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.

If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.

But yeah, you could add your videos probably to here and we could if those errors still appear.

[netExtra]

Hello,

I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.

If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.

But yeah, you could add your videos probably to here and we could if those errors still appear.

Thanks. I separate SVP errors from Rife errors. As you said, SVP errors with Rife tend to happen on scene changes. But these are not the same as Rife errors which happen less often but generally look much worse. Rife 4.7 uses about 20% more GPU memory for me, but it has improved things with one major artefact gone and another has been reduced.

Right now, there is one horrible error where the character literally blinks in and out each alternate frame. I have attached a video showing the issue. You can't miss it.

Rife.error.mov

[jensdraht1999]

@netExtra I'd say, the best thing is to test out https://nmkd.itch.io/flowframes.

If you test out all different models and all of them exhibit this error, then this might be a rife problem.

It probably is, because most of the rife problems are very fast motion problems, where the ai has to guess, what happens.

BTW: You can also test out all other methods beside rife very easily and if this still happens, then this might even be an univerisal problem that cannot be solved probably.

[netExtra]

@netExtra I'd say, the best thing is to test out https://nmkd.itch.io/flowframes.

If you test out all different models and all of them exhibit this error, then this might be a rife problem.

It probably is, because most of the rife problems are very fast motion problems, where the ai has to guess, what happens.

BTW: You can also test out all other methods beside rife very easily and if this still happens, then this might even be an univerisal problem that cannot be solved probably.

I've tested 4.6 and 4.7 , both have the same issue and since 4.7 > 4.6 > 4.4 I'm guessing that 4.4 will be the same. Maybe I'm missing something here but even the pay version of Flowframes seems to be only using 4.4 which has even more artefacts and errors.

[jensdraht1999]

FlowFrames Pay Version uses up to 4.6. But I'd say, try out RIFE 3/2/1 in FlowFrames Free versions and perhaps they are not present there.

[netExtra]

FlowFrames Pay Version uses up to 4.6. But I'd say, try out RIFE 3/2/1 in FlowFrames Free versions and perhaps they are not present there.

Installed Flowframes. Says GPU: Virtual Desktop Monitor even though I have an Nvidia GPU. Tried changing GPU id makes no difference. When interpolating CPU is basically idling for scene extraction and frame extraction. If it's going to be this slow I really would need a realtime mode.

[jensdraht1999]

@netExtra OK if all of them have it, then it's probably RIFE Model, probably needs more training.

[Q8sh2ing]

Hello,

I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.

If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.

But yeah, you could add your videos probably to here and we could if those errors still appear.

Thanks. I separate SVP errors from Rife errors. As you said, SVP errors with Rife tend to happen on scene changes. But these are not the same as Rife errors which happen less often but generally look much worse. Rife 4.7 uses about 20% more GPU memory for me, but it has improved things with one major artefact gone and another has been reduced.

Right now, there is one horrible error where the character literally blinks in and out each alternate frame. I have attached a video showing the issue. You can't miss it.

Rife.error.mov

cases like this are pretty common for large movements and you pretty much can't do anything. there are several algorithms out there that are trained for large movement objects where you can test it out. There is also a paid software called SVFI available on steam, their team has brought out a new feature where you can mask out the objects that are not moving for example crosshair and game map, in your case it would be the character that are barely moving in the centre.

[netExtra]

cases like this are pretty common for large movements and you pretty much can't do anything. there are several algorithms out there that are trained for large movement objects where you can test it out. There is also a paid software called SVFI available on steam, their team has brought out a new feature where you can mask out the objects that are not moving for example crosshair and game map, in your case it would be the character that are barely moving in the centre.

Thanks but there were other ugly errors in Rife 4.6 that have been fixed in 4.7/4.8 so I would hope that this eventually gets fixed soon. Especially since these large and ugly errors don't occur in SVP when not using Rife.

[netExtra]

4.9 is better. One remainder of an artefact that was improved in 4.7 is now completely gone in 4.9. The Spider-man artefact I show above seems to have improved slightly. I have another similar artefact that has also improved to the point where I can remove it by decreasing the scene detection threshold. It's also a little more efficient than 4.7 too Rife is getting better and better 👍

[netExtra]

Update 04/11/2023 I don't know if this helps but I can remove the Spider-man artefact shown above by significantly reducing rife-scene change threshold in SVP. But it is so low that it cause stuttering issues elsewhere.

[hooke007]

The 'scene change threshold in SVP' you mentioned is out of rife's control. Because it use a different 'solution' to detect scene changes. Once some frames were marked by this 'solution', rife do nothing on them.

[hooke007]

Ideally, choose a clip without scene-changes to test. If you have to use SVP, then totally disable it's scene change algorithm.

[netExtra]

Ideally, choose a clip without scene-changes to test. If you have to use SVP, then totally disable it's scene change algorithm.

I disabled the sc algorithm. I double checked it was off by setting the sc threshold to 5 which I know fixes the issue in SVP manager GUI. When I run the clip it's broken just like the video attached above. When I re-enable the algorithm the artefact disappears but causes major problems elsewhere so it's not a solution.

I'm not sure how testing a clip without scene-changes would help because there are no issues with those and never have been.

[hooke007]

choose a clip without scene-changes to test.

It's not talking about the movement of len.

Could you give the raw clip of your test src?

[netExtra]

choose a clip without scene-changes to test.

It's not talking about the movement of len.

Could you give the raw clip of your test src?

Hi. Sorry I am confused about your request for a clip without scene changes. I don't have any issues with that. What do you mean by "movment of len"?. Also, my raw clip is the one that is attached to this issue. Thanks.

[hooke007]

I only see the screen recording here. #34 (comment)

I want to test its raw clip on my side.

[netExtra]

I only see the screen recording here. #34 (comment)

I want to test its raw clip on my side.

OK I'm doing it now. Will upload it in a couple of minutes. Thanks.

[netExtra]

Attached. I had to convert it to MP4.

TAS-001.mp4

[hooke007]

Thanks. It could be reproduced by inference_video.py with model4.9.
https://github.com/hzwer/Practical-RIFE/assets/41094733/6414e4cd-a169-47bd-a23a-d67244210687

[netExtra]

Thanks. It could be reproduced by inference_video.py with model4.9. https://github.com/hzwer/Practical-RIFE/assets/41094733/6414e4cd-a169-47bd-a23a-d67244210687

Now it can be replicated does that help? BTW the attached video doesn't work :)

[hooke007]

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.

I didn't try legacy v3/2 models so I have no ideas more.

[netExtra]

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.

I didn't try legacy v3/2 models so I have no ideas more.

OK Thanks. I've raised a new issue for Rife 4.11 but I'm not sure if I should be since it probably doesn't affect the model training.

[netExtra]

OK so this artefact seems to be gone now and I am confused because up until this morning it was still there with Rife 4.9. I have another annoying artefact similar to this which was getting better but seems to have regressed a little. SVP has been updated and new code added so I will have to investigate further.

[carycary246]

Any way to reduce panning artefacts? My TV motion interpolation never has issues with straight pans like this but rife often has issues with these kind of shots.

mpv_2024-03-11_13-50-52.mp4

[netExtra]

@carycary246 Maybe I'm missing something but I can't see any major artefacts in that clip. But if it is there, then it looks very minor. Also which version of Rife are you using?

[carycary246]

@carycary246 Maybe I'm missing something but I can't see any major artefacts in that clip. But if it is there, then it looks very minor. Also which version of Rife are you using?

Look at the right side with the building fences. I think Rife doesn't know how to handle those kind of objects (thin lines?) even when it is a straight pan with no other motion. I am using 4.14 lite now, but all previous versions are the same. Here is another example, in motion it is distracting because it warps a large area over several frames.

mpv_2024-03-11_15-42-38.mp4

[netExtra]

@carycary246 The artefacts are not being captured clearly in your clips, but they are clear in the still image. I don't watch anime, but in live action this type of horizontal and vertical pan straight line warping is much reduced by Rife v4.9 and mostly gone by Rife v4.13, with only a couple of minor artefacts visible if you look for them. The main issue remaining is fast movement artefacts but v4.15 is an improvement. Sorry I can't be more help.

[hiredd]

4 frame interpolation model may resolve the problem

[hexiaoyi95]

@carycary246 can you share these two original videos without interpolation?

[carycary246]

@hexiaoyi95 I've uploaded the two clips without interpolation here.

https://mega.nz/file/9ToS0KgQ#S03DLc5YPMCPnDhPmsjN-bLpb4O3MR2xQjVJpC4HNhg
https://mega.nz/file/Zf5X1IAR#HesJScsyt7iBT0b4zM8Rtp7iwDXvJL-Z8Crc0Bgd0pM

[netExtra]

@hexiaoyi95 I've uploaded the two clips without interpolation here.

https://mega.nz/file/9ToS0KgQ#S03DLc5YPMCPnDhPmsjN-bLpb4O3MR2xQjVJpC4HNhg https://mega.nz/file/Zf5X1IAR#HesJScsyt7iBT0b4zM8Rtp7iwDXvJL-Z8Crc0Bgd0pM

SVP can't play "Your Name" clip properly because it might be HFR?. But I was able to play the "Solo" clip and the artefacts are identical to the ones I used to see in the introduction to Blade Runner 2023 before Rife 4.9.

[carycary246]

SVP can't play "Your Name" clip properly because it might be HFR?. But I was able to play the "Solo" clip and the artefacts are identical to the ones I used to see in the introduction to Blade Runner 2023 before Rife 4.9.

Your Name is a standard 24fps film, not sure why you wouldn't be able to play it. Panning motion wouldn't be any different from any other kind of 24 fps media.

[netExtra]

SVP can't play "Your Name" clip properly because it might be HFR?. But I was able to play the "Solo" clip and the artefacts are identical to the ones I used to see in the introduction to Blade Runner 2023 before Rife 4.9.

Your Name is a standard 24fps film, not sure why you wouldn't be able to play it. Panning motion wouldn't be any different from any other kind of 24 fps media.

SVP doesn't like it and it's default config treats it like HFR but I still got a good idea of the artefacts for both clips.

[netExtra]

Rife v4.15 was a major improvement for fast motion but Rife v4.16 is a major regression. I hope @hzwer does NOT use this as the backbone for future versions because for live action fast motion, it would be like going back to Rife 4.9 or worse.

[TNTwise]

4.16 is a lite model, so it is similar to 4.9 in terms of quality. I'm assuming he means that it will be the base for future lite models.

[netExtra]

Maybe. But 4.16 being pretty much like 4.9 for quality is going in the wrong direction. Rife 4.9 is a nice all round model but is pretty bad for live action. Rife 4.15 is the best for quality.

[TNTwise]

Well, in terms of speed, it's faster than 4.13-lite, the lite models are more for speed than quality. Keep using 4.15 for the best results.

[hzwer]

@netExtra Interesting, since recently, friends who care about 2d or 3d scenes almost always give me the opposite feedback. I plan to release a heavier version to see the effect.

[netExtra]

@netExtra Interesting, since recently, friends who care about 2d or 3d scenes almost always give me the opposite feedback. I plan to release a “heavy” version to see the effect.

I've checked the SVP forums and the consensus is that v4.16 lite is a good all round replacement for v4.9. But v4.15 has the least artefacts, especially for fast motion. Maybe your friends are talking about animation? I'm talking about video and movies.

[netExtra]

@netExtra Interesting, since recently, friends who care about 2d or 3d scenes almost always give me the opposite feedback. I plan to release a “heavy” version to see the effect.

Rife v4.15 lite arrived today and is almost as good as v4.15 (heavy) and is much better than v4.16 for movies with fast motion. Rife v4.15 (heavy) still handles the artefacts I have shown above on Nov. 5th 2023 with Spider-man much better.

[DisplayTalk]

2024-01-22.23-07-11.mp4

2024-01-29.11-46-16.mp4

I uploaded here some recordings of the display motion testing software Smoothfrog(https://www.aperturegrille.com/software/) at both 30fps and 24fps. Both use a 3840px/sec pan speed(for the 1920 width recording this is 1 screen width per 0.5sec).
I realize that this test is not similar to the training data, and will not reflect all content. However, it can be considered a "torture test" for fast panning. It reflects a part of VFI performance(fast simple motions with fine patterns)

for 24fps(which is a more strict test due to lower base fps) 2x:

4.7

4.8

4.9

4.10

4.11

4.12

4.13

4.14

4.15

4.16 lite

As you can see 4.7 beats all later models on this test. I still think that 4.15 is the best overall performing rife model, but for certain cases it's outperformed by 4.7. In the movie "Spirited Away" the frame slightly jitters up and down on each frame on the original. With all models 4.8 and later this causes all pans to jitter in the rife output. With all models 4.7 and previous this jitter is not there. Also not a problem in softsplat based interpolation GMFSS pg104/UMSS v1/VFSS Anime v1.

Here is a link to a 25 second clip of one of the pans from the intro: https://gofile.io/d/WJDoap

Please eyetrack the stained glass windows, and compare 4.7 to 4.15 on this clip. On 4.15 you should see what looks like a microstutter periodically during the motion, which is not present on 4.7.

If anyone is using scdetect for the verification/testing, please turn it off. It will turn off the interpolation in most fast motion if you have it adjusted to sensitive values(anything lower than like .3). That goes for these clips and also any other verification of interpolation.. Well, unless you're using a transitionchooser list to manually select the correct scene changes.

I hope this can be useful feedback for rife @hzwer, the blendy/blurry appearance in fast motion(and especially the panning jitter on some sources) has bothered me since I first noticed it with later models. Rife 4.15 is very very good for general watching both of live action and anime, but I think it can still be better if the training(and especially verification) includes more extremely fast motions. Especially fast continuous motions without any motion blur on the source, like game footage recorded at 24 or 30fps for example. The biggest weaknesses are the multiple layers, fine patterns, and fast motion.
Anyways, thanks for all the work so far, you're doing a good job and I still eagerly await future iterations of Rife.

[netExtra]

@DisplayTalk what you say about 4.7 is interesting. I will test with my normal test scenes and compare.

Update: 4.7 is actually very good for fast movement but it's not quite as good as the two v4.15 models. It still suffers a little from similar artefacts to v4.6. But v4.15 lite is just as good for GPU usage (if not a little better) and doesn't suffer from the same artefacts as models pre v4.9.

[DisplayTalk]

Yes, 4.15 is still the better model generally, but in some situations 4.7 is better. I think this can be fixed by changing training/validation a bit to prioritize simple fast continuous motions, handling of fine details/patterns, and stability of panning.

[netExtra]

It's been a while since I've used Rife v4.15 v2 so I thought I should have a proper look again. It minimises or removes every fast motion artefact I have come across with the default SCT in SVP. Maybe I'm repeating myself a little here but Rife v4.15 lite is pretty close to 4.15 but uses much less CPU. So if you don't have the same artefacts it makes more sense to use the lite version.

[netExtra]

I'm not sure if this helps but:
SVP with TensorRT 9.2 - Rife v4.15 (heavy) at 60fps works best with fast motion.
SVP with TensorRT 8.5.1 - Rife v4.15 lite at 72fps and above works best. Rife v4.15 (heavy) artefact handling gets worse.

i.e.
TRT 8.5.1 with Rife v4.15 lite/72fps = TRT 9.2 with Rife 4.15/60fps.