Color shift after interpolation

[Shardz0]

after I interpolate my video
the reds, magentas, and blues get shifted to a darker value while the yellows, greens, and cyans gets shifted to a whiter value.
Greens, cyans, magentas, also get shifted ~ -3 degrees

[mirh]

This is probably a duplicate of #52
In the developer options for ffmpeg, add -bsf:v h264_metadata=matrix_coefficients=6 as additional argument.

[DestroyTargetNonBasicLand]

Don't know what exactly causes this but both Flowframes and VEAI suffer from this same issue.

AFAIK the only real solution here is to transcode your inputs to YUV444/RGB output colorspace beforehand. (I personally use ProRes 4444 via StaxRip.)

Unsure if this can be automated @n00mkrad (a seamless real-time conversion of sorts), but it's worth looking into IMO.

Comparisons - click on "Comparison 1/3" for more examples.

[mirh]

-.-
Just follow the links and you see everything explained thoroughly.
There's also an additional ffmpeg bug when importing jpeg pictures.

[DestroyTargetNonBasicLand]

Just follow the links and you see everything explained thoroughly..

Trust me, I've tried everything under the sun including the flags from that link, but the issue still persisted for me unfortunately.
Since VEAI also suffers from this same exact problem, the colors/gamma shifts start to stack up when upscaling motion interpolated videos for example.. >_>

...It finally dawned on me that these AI engines may have some sort of distaste for chroma subsampling when recalling that the rife vapoursynth plugin requires you to adjust the colorspace to RGB:

I gave a whirl, and to my complete disbelief - It actually worked!

https://slow.pics/c/EkBFc32i

Example 1: Original video (YUV420, BT.709) converted to ProRes 4444 prior to feeding it to RIFE, with no added arguments.

Example 2: Original video + the following flags: "-vf scale=out_color_matrix=bt709 -colorspace bt709 -color_primaries bt709 -color_trc bt709".

Example 3: Same as 2 but adding your: "-bsf:v h264_metadata=matrix_coefficients=6" flag instead.

[mirh]

"AI engines" work internally with RGB for simplicity reasons (and VS makes no effort to hide that), and that's ironically the deal here. They don't do anything with chroma themselves, it's not them to be wrong (and RGB is just a pixel format, not a colorspace).
Subsampling has something to do with our problem here, in the sense that it's about specifying how the compression works, but it's bugs in how ffmpeg handles it to make people go bonkers.
(and 99% of videos are 4:2:0 btw, duh... it's not some odd edge case)

And just about everything on this planet uses ffmpeg.. so it stands to reason that you may see similar glitches (even though developers may eventually workarounds the limitations, or be more or less lucky).

Example 1: Original video (YUV420, BT.709) converted to ProRes 4444 prior to feeding it to RIFE, with no added arguments.

I'm pretty damn sure RIFE itself doesn't support post-production codecs (or video codecs, in general tbh).
So.. we are talking about it inside flowframes here, right?

Example 2: Original video + the following flags: "-vf scale=out_color_matrix=bt709 -colorspace bt709 -color_primaries bt709 -color_trc bt709".

The original video is untagged BT.709 then, if this fixes it (otherwise converting tagged BT.709 to BT.709 again would make no difference).
When ffmpeg decodes the source nothing tells it what's the right approach (which video players may otherwise try to guess), so it's applying the default BT.601->BT.709 coefficients to what is instead already BT.709. And it marks the metadata as BT.709 (this should look wrong if you check it separately).

This will then properly inform the usual video -> png -> video pipeline into its implicit conversion to BT.601 (colorspace information that is then lost).
If decoded as BT.601 (as you'd be expected) you'd still have the wrong hues that you have forced above, but since 55c0f59 flowframes marks the final video with whatever it detected in the metadata of the input. video players would have to guess it again.
So crazily enough, two wrongs makes a right. Misreading BT.601-stored colours as BT.709, applies the inverse matrix of your first operation and what we could call BT.709² appears on your screen just right.

Example 3: Same as 2 but adding your: "-bsf:v h264_metadata=matrix_coefficients=6" flag instead.

No shit it's bad if you are forcing BT.601 to what you just converted to bt709? Drop that part, or use `1' at least.

[DestroyTargetNonBasicLand]

And just about everything on this planet uses ffmpeg.. so it stands to reason that you may see similar glitches (even though developers may eventually workarounds the limitations, or be more or less lucky).

Ah of course - VEAI also rely on FFmpeg (backend filters + encoding), so I guess all similar apps suffer from this issue..
This sucks, but we can at least report this to the devs I reckon.

The original video is untagged BT.709 then, if this fixes it (otherwise converting tagged BT.709 to BT.709 again would make no difference).

It's redundant I know - The examples I provided are just a way of proving that these aren't colormatrix/metadata fuck-ups by Flowframes,
and as much as I wish for a simpler alternative/workaround, the only solution atm is to transcode every video with chroma subsampling to yuv444/RGB pixel formats.

[mirh]

This sucks, but we can at least report this to the devs I reckon.

https://trac.ffmpeg.org/ticket/9167
https://trac.ffmpeg.org/ticket/979
https://trac.ffmpeg.org/ticket/9693
https://trac.ffmpeg.org/ticket/9672

It's redundant I know - The examples I provided are just a way of proving that these aren't colormatrix/metadata fuck-ups by Flowframes,

I just explained to you how all of this works.
It's fuck-ups by flowrames stacked over fuck-ups by ffmpeg, on top of whoever encoded your file being too lazy to actually tag the proper color information.

and as much as I wish for a simpler alternative/workaround, the only solution atm is to transcode every video with chroma subsampling to yuv444/RGB pixel formats.

No, simply run

ffmpeg.exe -i "input.mp4" -c copy -bsf:v h264_metadata=matrix_coefficients=1 "output.mp4"

and then add this to the ffmpeg flowframes developer options arguments (also disable JPEG image extraction)

-bsf:v h264_metadata=matrix_coefficients=6:video_full_range_flag=0 -vf scale=flags=accurate_rnd

This was at least enough to make the second sample from #84 happy (that doesn't require extra tinkering, but here

---

Jesus christ, flowframes color handling only triggers when the source has all the information provided.
Meaning that videos could have enough information to fall into the dooms of ffmpeg, but not enough to trigger the half-assed safeguard of flowframes.

So now I understand what you meant in example 2-3: you didn't reprocess the source video with ffmpeg before using it as an input, those are simply the options inside flowframes... And specifying the matrix_coefficients didn't help...
....
Did you remember to uncheck the damn "HQ JPEGs" setting together with that?