Video scenes are choppy in newer FFmpeg versions

[mrfixit2001]

The video scenes play and are not garbled, but they are choppy/laggy. I've tried virtually all available video settings and none have any affect that I've found. But... when I run libretro-ppsspp from retropie the videos play perfectly. The libretro gameplay doesn't perform as well as this core emulator does, so I'd prefer to use this one.

The retropie version is based from an older commit of ppsspp and sourced in this repo: https://github.com/RetroPie/ppsspp/tree/libretro

I wrote a patch (attached) which specifically reverts /Core/HLE/sceMpeg.cpp back as close to the commit from the retropie repo as I can get it. It compiles but it doesn't resolve the problem. So some time between then and now something with the video scenes player broke, as it's clearly not the environment.

ffmpeg-revert.zip

[unknownbrackets]

You'll need to test more of the commits between among those 8000 to narrow it down - it won't really be possible to figure out what happened.

That said, a few things:

1. What games? If you think it's every game, try at least 10 - it could just be a common problem to a single developer's games or a certain genre of game with common libraries.

2. What device? If you're using a video core GPU / raspberry pi, it's been reported that the latest changes made it slower (Bad performance on Raspberry Pi3 with new 1.6.3 version #11265), even though they made virtually every other device much faster.

3. What operating system?

4. What FPS shows in PPSSPP? Is PPSSPP getting full speed, but the video is skipping? Or, more likely probably, is emulation just not reaching 100% performance and so you're lagging?

5. If you set frameskip to 8 and turn off auto frameskip, does it run full speed (just low framerate)? Is the audio smooth?

6. If you're using Linux, are you using the SDL port, Qt port, or something else? Are you using system ffmpeg or the submodule version of ffmpeg that is older but has a compatible ABI?

-[Unknown]

[mrfixit2001]

Good feedback! Thanks! Some answers: Obviously testing 8k commits is unrealistic lol but I realize it's a large gap for sure. I'll try and find the time to test each release and see where it first started. Curious if anyone else sees this same issue? I can confirm that gameplay reaches 100 percent and sometimes higher (as long as frameskip is 2). It averages at 99.5%. I can confirm the issue exists at both 720p and 1080p resolutions. I'm testing it on Linux with rockchip kernel 4.4 with multiple boards. Auto frame skip is off and I've tested with a frame skip of 10 without resolve. Audio is perfect. It's PPSSPPSDL and it's using system ffmpeg. Same exact settings as I'm using on the libretro version compiled.

[unknownbrackets]

It's not The Warriors is it? I think that game specifically is not currently working right: #8991

I know that videos in general work great on Android and Windows in most games. A good game to validate against is Crisis Core, since it's popular. Have not tested Mac or Linux as recently, but I know they were working since v1.6.3 (maybe around 300 commits after) at least for sure. No recent changes to video handling, really.

Even trying a build in between would cut that down to 4000. I think if we were looking at a range between stable releases, we could make some vague guesses. That said, of course 8000 commits can be bisected in probably 13 tries or so for someone experiencing the issue.

It might also be worth trying FFmpeg 3.0.2 (this is the version we test against, and link on Android/Windows/etc. official binaries - no patches, but significantly reduced configure options to make it lighter.)

Of course, FFmpeg is not careful about backwards compatible API breaks, but an issue was recentlyish (iirc after v1.5.4?) fixed that was causing horrible corruption during video playback in new system FFmpeg versions. If such an old PPSSPP is working fine on a newish system FFmpeg, it makes me suspicious that it might actually be using a static FFmpeg (build script error) or be using some unknown patch.

-[Unknown]

[mrfixit2001]

I'll test some of those games when I'm able and report back, thanks!

I'll also test some different release versions and see if I can determine when it started. I know that video in 1.5.4 was definitely all garbled, so I'll start with v1,6.

I had to patch sceMpeg.cpp in order to make the older ppsspp versions work with the newer ffmpeg versions. Can you advise what other source files in this project are involved in ffmpeg playback besides that one? I can test reverting some of them to see if I can identify the problematic file, which should narrow the commit scope down dramatically as well.

[unknownbrackets]

Some things are interrelated, but Core/HW/MediaEngine.cpp is probably involved as well. Depending on what game is playing the video, it may also use Core/HLE/scePsmf.cpp not sceMpeg.

-[Unknown]

[mrfixit2001]

That was stellar info, thanks! For some reason I haven't been able to compile versions 1.6 or 1.6.3 without EGL, no matter what flags or cmakelists changes I tried. Oh well, I was able to fix it against the master commit from the other detail you provided.

After testing, I can confirm that reverting the changes made to MediaEngine.cpp fixes the stuttering issue. It's not a complete revert, in order to successfully compile, but mostly. So the fix for this issue is somewhere in the attached patch. I haven't tested individual segments to narrow it down to root cause, but maybe you'll be able to have an idea when you look at it. If you want to send over a modified patch for me to try in my environment that doesn't revert everything included then feel free.

revert_mediaengine.zip

[unknownbrackets]

Which game is this?

-[Unknown]

[mrfixit2001]

I have a hand full I'm testing. Not in front of the computer right now but I know both need for speed and tekken are in there. All the ones I've tested had playback issues and the patch has resolved.

[mrfixit2001]

I know NFS is 30fps and tekken is 60fps, just FYI, that's why I know they are in my test group. But no settings were changed, just this patch applied.

[mrfixit2001]

FYI - Crisis Core is in my test list, per your recommendation. Runs perfectly now. Stuttered before.

[unknownbrackets]

You'll want to identify what change you're reverting.

A few things to make sure you haven't lost track:

• This works for other people. Presumably, it's just not working for you since you're using a newer (or older?) version of FFmpeg than we support (we support FFmpeg 3.0.2.)

• Various games play videos differently. Sometimes, a change that fixes one game may break another game.

At this point, you seem to just want to "undo" all changes that have been made since v1.0.1 (4 years of changes) to anything related to videos, without regard to what it might break or why. I'm sure it's a worthy goal to make a newer version of FFmpeg work, but it's probably some API breakage (like us relying on a pts field that isn't being consistently filled anymore, or something.)

For example, your proposed change breaks support for mono audio. It will also reintroduce #9262 and make video playback slower on mobile devices.

I don't think this is pull request ready, no.

-[Unknown]

[mrfixit2001]

Agreed, definitely not ready for a PR on this, and you're correct that at this time my concentration is to resolve the issue for my specific platform without regard to how it affect others (such as mobiles). My "fix" was simply based off of knowing it used to work and using that working version to compare what had changed.

Yes I am definitely using an FFMPEG version later than 3.0.2.

As time permits I will continue testing and see if I'm able to narrow the patch down to the specific segment(s) necessary to resolve the issue so that it's not just a blanket revert, as it is in it's current state.

Thanks for all your help on this, I'll let you know if I have time to work more on this. For now the issue can be closed :)

[unknownbrackets]

Okay, well, it's still an issue so we're probably better off leaving it open.

-[Unknown]