Fix timing of AVI files dumped on Linux #1345

sgadrat · 2014-10-20T09:30:43Z

Scenario

Uses a Linux build of Dolphin-Emu.
Emulate a game that runs at 30 FPS instead of the targetted 60 FPS.
Start dumping frames.
Stop dumping frames after some times.
Play the dumped AVI file with a media player.

Expected result

The video is played at 30 FPS

Actual result

The video is played at 60 FPS, making it accelerated.

The patch

The timing information is set on s_scaled_frame->pts, giving precise timing information to the encoder. Frames arriving too early (less than one tick after the previous frame) are droped.

The setting of packet's timestamps and flags is done after the call to avcodec_encode_video2()
as this function resets these fields according to its documentation.

Source/Core/VideoCommon/AVIDump.cpp

@@ -335,6 +336,8 @@ static int s_height;
 static int s_size;
 static u64 s_last_frame;
 bool b_start_dumping = false;
+static Common::Timer s_timer;
+static int64_t s_last_pts;


RisingFog · 2014-10-20T13:03:39Z

Does this handle save states and dumping in the middle of emulation?

sgadrat · 2014-10-20T13:19:32Z

Dumping in the middle of emulation works.

About save states. I tried this scenario :

Start emulation
Start frame dumping
Save the state
Wait a little then load the saved state
Stop frame dumping

The resulting AVI contains the video with as output by the emulator. We notice only the time at wich we loaded the state, as de game returns to the past. It seems to be the normal behaviour to me. Do you expect somthing else ?

RisingFog · 2014-10-20T13:30:51Z

That's the expected output. How about games like Crazy Taxi and Super
Monkey Ball? Do those games provide a proper output with regular dumping?

On Mon, Oct 20, 2014 at 9:19 AM, sgadrat notifications@github.com wrote:

Dumping in the middle of emulation works.

About save states. I tried this scenario :

Start emulation

Start frame dumping

Save the state

Wait a little then load the saved state

Stop frame dumping

The resulting AVI contains the video with as output by the emulator. We
notice only the time at wich we loaded the state, as de game returns to the
past. It seems to be the normal behaviour to me. Do you expect somthing
else ?

—
Reply to this email directly or view it on GitHub
#1345 (comment).

sgadrat · 2014-10-20T13:54:28Z

I will test with these games as soon as I get my hands on it. Before testing I can only say that the patch should be resilient irregular framerates. The point of the patch is to timestamp frames to be encoded by watching the wallclock when a frame is output, making it resilient to framerate variance being due to slowdowns in emulation or naturally irregular.

Note that I considered that these games provide irregular framerate by reading your comment, but I may be wrong. Is there any specific trap I should considere when testing these games ?

JMC47 · 2014-10-20T13:55:34Z

The Spyro: Enter the Dragonfly game likes to jump between 20, 30 and 60 fps. That game is scary for video dumping.

RisingFog · 2014-10-20T14:12:06Z

Those games have a splash screen at the beginning where it's actually just
a single frame which then is held there by a VI interrupt (same with now
loading, etc. for crazy taxi)

On Mon, Oct 20, 2014 at 9:54 AM, sgadrat notifications@github.com wrote:

I will test with these games as soon as I get my hands on it. Before
testing I can only say that the patch should be resilient irregular
framerates. The point of the patch is to timestamp frames to be encoded by
watching the wallclock when a frame is output, making it resilient to
framerate variance being due to slowdowns in emulation or naturally
irregular.

Note that I considered that this games provide irregular framerate by
reading your comment, but I may be wrong. Is there any specific trap I
should considere when testing these games ?

—
Reply to this email directly or view it on GitHub
#1345 (comment).

sgadrat · 2014-10-20T14:41:17Z

Tested crazy taxi and spyro.

About crazy taxi, the "now loading" lonely frame is held on screen by the player an amount of time that seems correct. Tested with VLC and ffplay.

About Spyro, I had some slowdowns during kinematics in the emulator itself so it is repercuted in the video file. But the consistent ingame framerate changing is unnoticeable on the video file.

Source/Core/VideoCommon/AVIDump.cpp

@@ -311,6 +311,7 @@ bool AVIDump::SetVideoFormat()
 #include "Common/FileUtil.h"
 #include "Common/StringUtil.h"
 #include "Common/Logging/Log.h"
+#include "Common/Timer.h"


lioncash · 2014-10-20T17:07:28Z

Can you squash two change commits back into the original one?

RisingFog · 2014-10-20T17:20:50Z

@dolphin-emu-bot rebuild

RisingFog · 2014-10-21T01:30:06Z

EDIT: nvm, read below

RachelBryk · 2014-10-21T22:32:04Z

Nope, this is totally wrong behavior. Frames need to be dumped based on emulated time, not host time.

RisingFog · 2014-10-21T22:41:22Z

Ah didn't notice that. Thanks for pointing that out. If you could base it
off coretiming tick counts that would be a lot better
On Oct 21, 2014 6:32 PM, "RachelBryk" notifications@github.com wrote:

Nope, this is totally wrong behavior. Frames need to be dumped based on
emulated time, not host time.

—
Reply to this email directly or view it on GitHub
#1345 (comment).

sgadrat · 2014-10-22T01:28:29Z

Here is the patch for dumping frames based on emulated time. It is indeed a better idea than using host system as the video should be smooth event when the emulator slow down.

I have to some questions about the patch itself. I reuse s_last_frame that where effectively unused while apparently there for this kkind of synchronization. Is it a good idea ? And s_last_pts could be removed and recomputed for each frame. Do you prefere with the s_last_pts variable or recompute it ?

sgadrat · 2014-10-22T01:30:56Z

Oops did not saw that "AVIDump::DoState()" was shared between Linux and Windows.

sgadrat · 2014-10-22T01:56:41Z

Hmmm with this patch, the video playback is accelerated.

RisingFog · 2014-10-22T01:58:25Z

@sgadrat in what way?

Source/Core/VideoCommon/AVIDump.cpp

+	else
+		s_scaled_frame->pts = 0;
+	s_scaled_frame->pts += (delta * s_stream->codec->time_base.den) / SystemTimers::GetTicksPerSecond();
+	s_last_frame += delta;


sgadrat · 2014-10-22T22:56:25Z

@RisingFog It was due to the use of a little delta converted in 1/60th of a second then basing the next computing on the imprecise value. It is fixed int the last patch.

Source/Core/VideoCommon/AVIDump.cpp

-	int error = avcodec_encode_video2(s_stream->codec, &pkt, s_scaled_frame, &got_packet);
+	int got_packet = 0;
+	int error = 0;
+	u64 delta = CoreTiming::GetTicks() - s_last_frame;


RisingFog · 2014-10-23T13:17:11Z

I noticed some audio sync issues with some Wii/VC titles (similar to the issues I ran into initially with frame dumping). Hopefully those changes I suggested should be able to fix those issues.

Also once the changes are tested and committed, combine all the commits into a single commit.

The timing information is set on s_scaled_frame->pts, giving precise timing information to the encoder. Frames arriving too early (less than one tick after the previous frame) are droped. The setting of packet's timestamps and flags is done after the call to avcodec_encode_video2() as this function resets these fields according to its documentation.

sgadrat · 2014-10-23T21:42:59Z

I blindly fixed the points you commented as I am currently unable to test this Audio/Video sync issue. Is the reproduction scenario something like that ?

Check "Dump Frames" and "Dump audio" from "Movie" menu
Start the emulation of a Wii title [1]
Play the game making noises with obvious timing (like shot with a gun)
Stop the emulation
Merge the audio and video file
Play the resulting file watching for A/V synchronization

[1] Which games in particular may be good candidates ?

RisingFog · 2014-10-23T22:21:48Z

Yes. Good examples include Super Mario Galaxy (1 & 2) and Doc Louis Punch
Out.

On Thu, Oct 23, 2014 at 5:43 PM, Sylvain Gadrat notifications@github.com
wrote:

I blindly fixed the points you commented as I am currently unable to test
this Audio/Video sync issue. Is the reproduction scenario something like
that ?

Check "Dump Frames" and "Dump audio" from "Movie" menu

Start the emulation of a Wii title [1]

Play the game making noises with obvious timing (like shot with a
gun)

Stop the emulation

Merge the audio and video file

Play the resulting file watching for A/V synchronization

[1] Which games in particular may be good candidates ?

—
Reply to this email directly or view it on GitHub
#1345 (comment).

RisingFog · 2014-10-24T00:36:08Z

@dolphin-emu-bot rebuild

RisingFog · 2014-10-24T02:14:46Z

lgtm

Fix timing of AVI files dumped on Linux

lioncash reviewed Oct 20, 2014
View reviewed changes

Source/Core/VideoCommon/AVIDump.cpp

@@ -335,6 +336,8 @@ static int s_height;

static int s_size;

static u64 s_last_frame;

bool b_start_dumping = false;

static Common::Timer s_timer;

static int64_t s_last_pts;

This comment was marked as off-topic.

Sign in to view

lioncash reviewed Oct 20, 2014
View reviewed changes

Source/Core/VideoCommon/AVIDump.cpp

@@ -311,6 +311,7 @@ bool AVIDump::SetVideoFormat()

#include "Common/FileUtil.h"

#include "Common/StringUtil.h"

#include "Common/Logging/Log.h"

#include "Common/Timer.h"

This comment was marked as off-topic.

Sign in to view

sgadrat force-pushed the fix-avidump-framerate branch from e08fe3c to d42e379 Compare October 20, 2014 17:19

sgadrat force-pushed the fix-avidump-framerate branch from 1971ebd to c95a50c Compare October 22, 2014 01:51

RisingFog reviewed Oct 22, 2014
View reviewed changes

RisingFog reviewed Oct 23, 2014
View reviewed changes

Source/Core/VideoCommon/AVIDump.cpp

int error = avcodec_encode_video2(s_stream->codec, &pkt, s_scaled_frame, &got_packet);

int got_packet = 0;

int error = 0;

u64 delta = CoreTiming::GetTicks() - s_last_frame;

This comment was marked as off-topic.

Sign in to view

sgadrat force-pushed the fix-avidump-framerate branch from 316d455 to 3a12c50 Compare October 23, 2014 21:32

skidau added a commit that referenced this pull request Oct 28, 2014

Merge pull request #1345 from sgadrat/fix-avidump-framerate

b13ba06

Fix timing of AVI files dumped on Linux

skidau merged commit b13ba06 into dolphin-emu:master Oct 28, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix timing of AVI files dumped on Linux #1345

Fix timing of AVI files dumped on Linux #1345

sgadrat commented Oct 20, 2014

This comment was marked as off-topic.

RisingFog commented Oct 20, 2014

sgadrat commented Oct 20, 2014

RisingFog commented Oct 20, 2014

sgadrat commented Oct 20, 2014

JMC47 commented Oct 20, 2014

RisingFog commented Oct 20, 2014

sgadrat commented Oct 20, 2014

This comment was marked as off-topic.

lioncash commented Oct 20, 2014

RisingFog commented Oct 20, 2014

RisingFog commented Oct 21, 2014

RachelBryk commented Oct 21, 2014

RisingFog commented Oct 21, 2014

sgadrat commented Oct 22, 2014

sgadrat commented Oct 22, 2014

sgadrat commented Oct 22, 2014

RisingFog commented Oct 22, 2014

This comment was marked as off-topic.

This comment was marked as off-topic.

sgadrat commented Oct 22, 2014

This comment was marked as off-topic.

RisingFog commented Oct 23, 2014

sgadrat commented Oct 23, 2014

RisingFog commented Oct 23, 2014

RisingFog commented Oct 24, 2014

RisingFog commented Oct 24, 2014

Fix timing of AVI files dumped on Linux #1345

Fix timing of AVI files dumped on Linux #1345

Conversation

sgadrat commented Oct 20, 2014

This comment was marked as off-topic.

RisingFog commented Oct 20, 2014

sgadrat commented Oct 20, 2014

RisingFog commented Oct 20, 2014

sgadrat commented Oct 20, 2014

JMC47 commented Oct 20, 2014

RisingFog commented Oct 20, 2014

sgadrat commented Oct 20, 2014

This comment was marked as off-topic.

lioncash commented Oct 20, 2014

RisingFog commented Oct 20, 2014

RisingFog commented Oct 21, 2014

RachelBryk commented Oct 21, 2014

RisingFog commented Oct 21, 2014

sgadrat commented Oct 22, 2014

sgadrat commented Oct 22, 2014

sgadrat commented Oct 22, 2014

RisingFog commented Oct 22, 2014

This comment was marked as off-topic.

This comment was marked as off-topic.

sgadrat commented Oct 22, 2014

This comment was marked as off-topic.

RisingFog commented Oct 23, 2014

sgadrat commented Oct 23, 2014

RisingFog commented Oct 23, 2014

RisingFog commented Oct 24, 2014

RisingFog commented Oct 24, 2014