Depending on exact moment playback or fullscreen starts, OpenGL playback may stutter

[Cyberbeing]

Randomly ~10% of the time when starting mpv, playback or fullscreen with the OpenGL renderer will stutter badly. This is usually resolved by exiting mpv and restarting playback, though occasionally it takes multiple attempts to restore normal behavior.

This behavior can be reproduced with --ao=null --no-audio in additional to normal audio output drivers, suggesting this problem is unaffected by audio.

Reproduced on:
Windows 7 SP1 x64
GTX 770 2GB
mpv git-c5e5f38

[ghost]

Does pausing and unpausing change it? Does it change after playing for a while?

[Cyberbeing]

Yes, the issue can also be reproduced or resolved by pause+unpause or seeking the video.

Once this stuttering issue is triggered, the amount of stuttering which occurs does not always seem to remain constant during playback. On a test I just did right now, it alternated between horrible stuttering and minor stuttering every 10 seconds or so. I'd expect this could vary from test to test. Though unfortunately, the stuttering does not seem resolve itself as playback continues without manual user intervention.

I'd suspect the problem I'm seeing is vsync jitter, which can also affect directshow video renderers on Windows unless they actively manage vsync/presentation behavior similar to madVR or Reclock. That said, I'm unfamiliar with the means mpv may already currently be using for managing vsync in regards to video playback rate, a/v sync, and frame presentation on Windows to avoid this.

Though it does seems strange that mpv is unable to automatically calculate realtime --display-fps (refresh rate) information on Windows, and utilize that information to optimize presentation times in regards to video framerate?

Maybe it would be useful for debugging purposes if mpv implemented something similar to the realtime ctrl+j osd from mpc-hc evr-cp which contains a jitter graph along with various other renderer timing (min/max/stddev) and framerate statistics?

[Soukyuu]

I'm having the same issue on win8.1 x64 + 260GTX, it seems. Starting the video stutters, pause->unpause seems to make things better. I always thought that was because my GPU is rather old and mpv doesn't use a buffer.

[ghost]

I'd suspect the problem I'm seeing is vsync jitter, which can also affect directshow video renderers on Windows unless they actively manage vsync/presentation behavior similar to madVR or Reclock.

Why would vsync jitter? I'd expect it to be pretty much synchronized to display refresh, which in turn must be pretty stable.

What I could see happening is that the fact that mpv only flips all 2-3 vsyncs (with 23 fps video on 60 fps display) brings the driver out of cadence. This would lead to mistiming, so that instead of 2/3/2/3 the vsync intervals are 1/4/2/4 or something like this. If this applies, then I hoped this could be fixed by drawing the frame every vsync (even if it doesn't change), but since smoothmotion does something similar, and doesn't improve things for you, it probably won't help.

The main problem here is that OpenGL doesn't have any (standard/widely supported) APIs to handle frame timing.

Also, mpv normally adjusts video timestamps to audio speed. But since you get the same problem with --no-audio, this can't be the issue. (Although it is with --ao=dsound, because dsound's playback position reporting is coarse and useless.) Normally the audio/video/system clocks are sufficiently in sync that this can't become an issue. From what I know, Reclock changes the video and audio speed if video and display FPS don't match exactly, but even without I'd expect a mistimed frame only every few seconds or minutes, not heavy stutter all the time, so I think this is not related either.

Though it does seems strange that mpv is unable to automatically calculate realtime --display-fps (refresh rate) information on Windows, and utilize that information to optimize presentation times in regards to video framerate?

I think smoothmotion effectively does this, as long as you give it an approximately correct --display-fps value. I don't know how you expect mpv to calculate the refresh rate on its own. Win32 APIs don't return anything useful (I've only seen one API which returns rounded integers), and some drivers don't block on vsync, or if you tell them to do so, ruin performance.

Maybe it would be useful for debugging purposes if mpv implemented something similar to the realtime ctrl+j osd from mpc-hc evr-cp which contains a jitter graph along with various other renderer timing (min/max/stddev), refreshrate and framerate statistics?

I'd rather stay with --dump-stats. Drawing the graph at runtime would require tons of complex code and would falsify the rendering times.

I always thought that was because my GPU is rather old and mpv doesn't use a buffer.

As long as you use simple rendering (plain --vo=opengl), I wouldn't expect this to happen.

[Soukyuu]

If this applies, then I hoped this could be fixed by drawing the frame every vsync (even if it doesn't change), but since smoothmotion does something similar, and doesn't improve things for you, it probably won't help.

Well, it does improve things in the sense that the stuttering goes down a lot compared to no smoothmotion. It doesn't solve the stuttering completely, though, as described in #1505

As long as you use simple rendering (plain --vo=opengl), I wouldn't expect this to happen.

Yes, but it happens with plain --vo=opengl as well. Less so with --vo=opengl:smoothmotion.
Anyway, the GPU usage is too low to be the cause. I just never really looked into it because I was waiting for mpv to introduce smoothmotion.

[Cyberbeing]

What I could see happening...mistiming

I'll preface this that I personally am only testing with refresh rates which are near-exact multiples of the video framerate, usually ~95.904Hz or ~119.88Hz. So in terms of mpv, it's just a matter of ensuring playback is synced to my refresh rate as much as possible to ensure even display of video frames without extra repeats or drops.

mpv currently does not appear to wait for the end of a vsync cycle to start playback, nor correct sync between video playback rate and monitor refresh rate afterwords.

Why would vsync jitter?

That's just what I refer to this general problem, for lack of a better term. I honestly can no longer remember the exact technical explanation of this problem, and have no idea how much would apply to mpv in a non-directshow environment.

With DirectShow, the audio clock is used as the reference timer which determines playback rate. Yet the audio clock is often has lots of jitter which needs to be averaged out, not to mention it often runs slightly too slow or fast. The video framerate is then synced to the playback rate of the audio, along with any slowdown or speedup. GPU refresh rate and vsync and most other software are based on the cpu/system clock, which is rarely perfectly stable itself. Depending on exactly when you flip buffers during each vsync interval, smoothness of tearing-free playback can also vary.

This relationship between the audio clock (dshow reference timer), refresh rate, video framerate, and system clock is essentially "clock deviation" & estimated "drop/repeated frames every X" measures in madVR's ctrl+j osd stats.

For example:
Encoded Video Framerate: 23.9760 fps video (24000/1001 NTSC)
Clock deviation: +0.00163%
Actual dshow a/v playback rate: 23.976415 fps ( 24000/1001 * 1.0000163)
Optimal refresh rates: 23.976415 Hz, 47.952830 Hz, 71.929244 Hz, 95.905659 Hz, 119.882074 Hz

This dshow explanation probably has little direct relationship to the in mpv though, except of the basic concept that you'll have dropped/repeated frames if you have a non-perfect match between a/v playback rate and refresh rate. Currently, I have my refresh rates fine-tuned for directshow so there will only be a dropped/repeated frame every couple days of non-stop playback with 23.976 fps video. Assuming the video renderer ensures new video frame is displayed exactly every 4 vsync intervals (96hz) or 5 vsync intervals (120hz) without variation, there should be no smoothness or stuttering issues. If you have an unexpected minor decoding delay, you could do something like madVR and delay a video frame to the next vsync interval and then recover. Delaying a single frame is less noticeable than the alternative of dropping or repeating a single frame.

If this applies, then I hoped this could be fixed by drawing the frame every vsync (even if it doesn't change), but since smoothmotion does something similar, and doesn't improve things for you, it probably won't help.

When does the OpenGL render draw a frame every vsync interval? Does this only occur when :smoothmotion is enabled? What happens if I just set --display-rate on Windows with smoothmotion disabled? The only additional problems you should run into when presenting frame every vsync interval are, not displaying video frames with even durations, or what madVR likes to call presentation glitches (presentation is requested, but does not occur for some reason).

Win32 APIs don't return anything useful

If you want a user friendly value, you are probably correct. Otherwise you probably aren't looking hard enough, since I'm positive there are ways to calculate the actual refresh rate on Windows since madVR, reclock, and powerstrip all do so. I believe the method is something along the lines querying the total number of scanlines, setting a timer, and measuring how long it takes a single scanline to return. Average these raw measurements over a few seconds and you'll have the actual refresh rate value in terms of the system clock. My description could be slightly off, but I'm sure you could find some opensource code which does this if you cannot figure it out. The most common way to do this is probably with GDI and DirectDraw, but I'd assume something similar is also possible with Direct3D and OpenGL.

I'd rather stay with --dump-stats. Drawing the graph at runtime would require tons of complex code and would falsify the rendering times.

I'd really like something which it would be possible to easily see during playback, and doesn't require scrolling. What about expanding the console statistics to relay additional debug information below the AV: line?

[Cyberbeing]

Anyway, I don't really want to lead this discussion way off-topic from the issue at hand:

Why depending on the exact moment of playback start, unpause, or seek, does MPV OpenGL playback have the potential to stutter constantly, even when the video framerate should be a near-perfect match with the monitor refresh rate? What can be done to fix or improve on this?

[ghost]

mpv currently does not appear to wait for the end of a vsync cycle to start playback, nor correct sync between video playback rate and monitor refresh rate afterwords.

But that is the problem: it can't know when vsyncs happen or how often. How should it know? There are no guarantees by OpenGL.

If you have an unexpected minor decoding delay, you could do something like madVR and delay a video frame to the next vsync interval and then recover.

See above.

When does the OpenGL render draw a frame every vsync interval?

See above.

Does this only occur when :smoothmotion is enabled? What happens if I just set --display-rate on Windows with smoothmotion disabled?

If smoothmotion is enabled, it renders frames as quickly as possible, either repeating or interpolating frames. This will make the driver either lock to vsync (by blocking the caller if it tries to "queue" too many frames), or just wastes CPU/GPU. We still don't know when the frames that are displayed. If tripple buffering is enabled, the driver could even drop frames.

The only additional problems you should run into when presenting frame every vsync interval are, not displaying video frames with even durations, or what madVR likes to call presentation glitches (presentation is requested, but does not occur for some reason).

I don't know how exactly madVR does this, e.g. IDirect3DDevice9::Present has no timestamp parameter. Maybe IDirect3DSwapChain9Ex::GetPresentStat can be used to guess better about this. No idea. It's also possible that video APIs or video renderer APIs have something better. Anyway, all this is not relevant to OpenGL.

If you want a user friendly value, you are probably correct. Otherwise you probably aren't looking hard enough, since I'm positive there are ways to calculate the actual refresh rate on Windows since madVR, reclock, and mpc-hc all do so. I believe the method is something along the lines querying the total number of scanlines, setting a timer, and measuring how long it takes a single scanline to return.

Feel free to tell me the actual mechanisms, instead of vague descriptions.

I'd really like something which it would be possible to easily see during playback, and doesn't require scrolling. What about expanding the console statistics to relay additional debug information below the AV: line?

If it becomes possible to retrieve reliable statistics, maybe. If you want to debug, just add it to your copy of the source code.

Anyway, I don't really want to lead this discussion way off-topic from the issue at hand:

How is it offtopic?

Why depending on the exact moment of playback start, unpause, or seek, does MPV OpenGL playback have the potential to stutter constantly, even when the video framerate should be a near-perfect match with the monitor refresh rate?

That's what I'm trying to find out?

What can be done to fix or improve on this?

Install Linux and use the vdpau VO.

[Cyberbeing]

Feel free to tell me the actual mechanisms, instead of vague descriptions.

Unfortunately vague descriptions are the extent of my knowledge regarding this. You'll need to research it yourself or ask another developer for assistance.

How is it offtopic?

Only you can answer that. If it's given you some kind of useful idea, I guess it wasn't. I just felt like my previous long comment was getting somewhat off-topic with somewhat vague directshow information which may not even relate to mpv, especially since this issue occurs even without audio. And while I'm curious about how mpv functions and find it interesting, even if I knew, I don't have enough technical background to give you any further tips about how to resolve this.

That's what I'm trying to find out?

And hopefully you do. I'm probably not much more use to that end, unless you have something you'd like me to test.

[ghost]

Only you can answer that.

All these issues sound related to me.

[rossy]

I don't know how you expect mpv to calculate the refresh rate on its own. Win32 APIs don't return anything useful (I've only seen one API which returns rounded integers), and some drivers don't block on vsync, or if you tell them to do so, ruin performance.

DwmGetCompositionTimingInfo can give a fractional refresh rate. I think it's the right API to use when mpv is running on the desktop. It might also return other information that would be useful for syncing mpv's presentation times with the monitor. To get a fractional refresh rate in full screen (without D3D/DXGI,) I found the QueryDisplayConfig API. There's some code to use these in rossy@e24e3ef, but it's a bit messy at the moment.

As for VSync, maybe DwmFlush() could be used in windowed mode instead of relying on the swapinterval. #698 says that setting swapinterval=1 can cause judder with composition enabled and this definitely happens on my machines, though I'm not sure why there are issues when mpv is fullscreen.

[Cyberbeing]

Feel free to tell me the actual mechanisms

It would appear that madVR may use the D3D method GetRasterStatus() for this purpose.
https://msdn.microsoft.com/en-us/library/windows/desktop/bb174402%28v=vs.85%29.aspx

[Cyberbeing]

If using DirectDraw instead of D3D, it would appear to be a combination of:

GetVerticalBlankStatus
https://msdn.microsoft.com/en-us/library/windows/desktop/gg426151%28v=vs.85%29.aspx

GetScanLine
https://msdn.microsoft.com/en-us/library/windows/desktop/gg426149%28v=vs.85%29.aspx

[ghost]

@Cyberbeing: thanks. The DirectDraw APIs are redundant to the D3D9 one. Not sure how this could be used, though. Other than literally polling the display.

@rossy: this looks interesting. I assume the dynamic function loading is needed for XP compatibility?

[Cyberbeing]

The DirectDraw APIs are redundant to the D3D9 one.

While they are redundant, I expect using the DirectDraw method via GDI would probably be faster in terms of resource use when called standalone. Using Direct3D probably wouldn't make as much sense without an already existing Direct3D device to poll. Information about the DWM (forced vsync) Windows desktop can probably only be done with a DWM API like the one rossy mentioned.

Other than literally polling the display.

FWIW, I believe that constantly polling the display is exactly what madVR does, and then uses that information as the basis of its vsync presentation logic.

[ghost]

FWIW, I believe that constantly polling the display is exactly what madVR does, and then uses that information as the basis of its vsync presentation logic.

I guess it could just waste CPU time by polling it... but somehow I doubt that. An API that actually returns absolute times of when certain events happen or happened would be much more useful. Though it seems like DwmGetCompositionTimingInfo() returns some of that, e.g. fields like qpcVBlank. But I'm not sure if this call always works, or if it does something only if DWM is somehow active. It also doesn't answer the question how frames can be scheduled.

[Cyberbeing]

I guess it could just waste CPU time by polling it... but somehow I doubt that

Well you could always ask @madshi to confirm, but it's not exactly a secret that madVR does waste of bit of CPU time (+1-3%) constantly polling multimedia timers, display vsync information, and other such things.

Here is a quote from an old Doom9 post by madshi:

madVR calls GetRasterStatus() all the time to decide in which exact moment to present frames. Especially in windowed and overlay modes the information collected from GetRasterStatus() is extremely important. In FSE mode it's a bit less important, but still a bit.

[ghost]

@rossy: also, if you intend to clean this up, most of the definitions and the loader code should probably go into a different file. It's starting to get cluttered...

[rossy]

I assume the dynamic function loading is needed for XP compatibility?

Vista too, unfortunately.

I guess it could just waste CPU time by polling it... but somehow I doubt that. An API that actually returns absolute times of when certain events happen or happened would be much more useful. Though it seems like DwmGetCompositionTimingInfo() returns some of that, e.g. fields like qpcVBlank. But I'm not sure if this call always works, or if it does something only if DWM is somehow active.

Yeah, it only works when the desktop is composited. I think it's also possible to get a VSync timestamp from a DXGI swap chain, but that's not particularly helpful for OpenGL. It doesn't seem like it's possible to get the timestamp from a IDXGIOutput when the program doesn't have a swap chain running on the same screen.

if you intend to clean this up, most of the definitions and the loader code should probably go into a different file. It's starting to get cluttered

Will do. I was hoping to be able to move stuff to separate files. It seems like the cocoa and wayland code is in separate files already.

[ghost]

Anyway, I guess it comes down to:

1. Having a clever heuristic to dynamically adjust the moment when SwapBuffers is called, and
2. better retrieval of vsync statistics

Looking at some other projects, they do partially seemingly absurd things, like creating a thread only to run an endless loop waiting for (or even polling for) vsync events. On Linux, some even go as far as creating a hidden dummy window with dummy GLX context.

[avih]

Randomly ~10% of the time when starting mpv, playback or fullscreen with the OpenGL renderer will stutter badly.

Not sure I understand this statement. Does it only happen in full screen? only in windowed mode? or both?

If you can build mpv yourself, could you please try this branch https://github.com/avih/mpv/tree/js-duk , with --vo=opengl:dwmflush=1 and report back? This should try to sync with the DWM vsyncs only in windowed mode, or set it to 2 to do the same also in full screen.

Optimal refresh rates: 23.976415 Hz, 47.952830 Hz, 71.929244 Hz, 95.905659 Hz, 119.882074 Hz

Note to self: add the 47 and 95 exceptions to the list (the rest are already covered, and also 29, 59).

[mia-0]

Is this still an issue with --video-sync=display-resample?

[quilloss]

--video-sync=display-resample with --vo=opengl-hq:interpolation:dwmflush=windowed seems to solve the stutter issue for me.