Skip to content

Commit

Permalink
Improve frame accuracy when pausing and seeking.
Browse files Browse the repository at this point in the history 
When pausing, sometimes the consumer thread would output a newer frame before
the video thread was done. Then, the video thread would output an older frame.

Also, when seeking, the consumer would reload a bunch of the same frame into
the buffer.
  • Loading branch information
@bmatherly
bmatherly committed Feb 25, 2015
1 parent 06bca9b commit ab3ce28
Show file tree
Hide file tree
Showing 3 changed files with 46 additions and 20 deletions.
6 changes: 1 addition & 5 deletions src/glwidget.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -513,11 +513,7 @@ int GLWidget::reconfigure(bool isMulti)
m_consumer->connect(*m_producer);
// Make an event handler for when a frame's image should be displayed
m_consumer->listen("consumer-frame-show", this, (mlt_listener) on_frame_show);
if (m_glslManager) {
m_consumer->set("real_time", property("realtime").toBool()? 1 : -1);
} else {
m_consumer->set("real_time", property("realtime").toBool()? 1 : -MLT.realTime());
}
m_consumer->set("real_time", MLT.realTime());
m_consumer->set("mlt_image_format", "yuv422");
m_consumer->set("color_trc", Settings.playerGamma().toLatin1().constData());

Expand Down
8 changes: 2 additions & 6 deletions src/mainwindow.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1895,17 +1895,13 @@ void MainWindow::onShuttle(float x)

void MainWindow::on_actionRealtime_triggered(bool checked)
{
MLT.videoWidget()->setProperty("realtime", checked);
Settings.setPlayerRealtime(checked);
if (Settings.playerGPU())
MLT.pause();
if (MLT.consumer()) {
int threadCount = QThread::idealThreadCount();
threadCount = threadCount > 2? (threadCount > 3? 3 : 2) : 1;
threadCount = Settings.playerGPU()? 1 : threadCount;
MLT.consumer()->set("real_time", checked? 1 : -threadCount);
MLT.restart();
}
Settings.setPlayerRealtime(checked);

}

void MainWindow::on_actionProgressive_triggered(bool checked)
Expand Down
52 changes: 43 additions & 9 deletions src/mltcontroller.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -188,8 +188,11 @@ void Controller::play(double speed)
{
if (m_producer)
m_producer->set_speed(speed);
// If we are paused, then we need to "unlock" sdl_still.
if (m_consumer) {
// Restore real_time behavior and work-ahead buffering
m_consumer->set("real_time", realTime());
m_consumer->set("buffer", 25);
m_consumer->set("prefill", 1);
m_consumer->start();
refreshConsumer();
}
Expand All @@ -200,10 +203,18 @@ void Controller::play(double speed)
void Controller::pause()
{
if (m_producer && m_producer->get_speed() != 0) {
if (m_consumer && m_consumer->is_valid()) {
// Flush all frames from the video task in the consumer.
m_consumer->stop();
// Disable real_time behavior and buffering for frame accurate seeking.
m_consumer->set("real_time", 0);
m_consumer->set("buffer", 0);
m_consumer->set("prefill", 0);
}
m_producer->set_speed(0);
if (m_consumer->is_valid()) {
m_consumer->purge();
m_producer->seek(m_consumer->position() + 1);
m_producer->seek(m_consumer->position() + 1);
if (m_consumer && m_consumer->is_valid()) {
m_consumer->start();
}
}
if (m_jackFilter)
Expand Down Expand Up @@ -315,18 +326,28 @@ void Controller::onWindowResize()
void Controller::seek(int position)
{
if (m_producer) {
// Always pause before seeking (if not already paused).
if (m_consumer && m_consumer->is_valid() && m_producer->get_speed() != 0) {
// Flush all frames from the video task in the consumer.
m_consumer->stop();
// Disable real_time behavior and buffering for frame accurate seeking.
m_consumer->set("real_time", 0);
m_consumer->set("buffer", 0);
m_consumer->set("prefill", 0);
}
m_producer->set_speed(0);
m_producer->seek(position);
if (m_consumer && m_consumer->is_valid()) {
if (m_consumer->is_stopped())
if (m_consumer->is_stopped()) {
m_consumer->start();
else
} else {
m_consumer->purge();
refreshConsumer();
}
}
}
if (m_jackFilter)
mlt_events_fire(m_jackFilter->get_properties(), "jack-seek", &position, NULL);
refreshConsumer();
}

void Controller::refreshConsumer()
Expand Down Expand Up @@ -565,6 +586,10 @@ void Controller::setOut(int out)
void Controller::restart()
{
if (!m_consumer) return;
if (m_producer && m_producer->is_valid() && m_producer->get_speed() != 0) {
// Update the real_time property if not paused.
m_consumer->set("real_time", realTime());
}
if (m_producer && m_producer->is_valid() && Settings.playerGPU()) {
const char* position = m_consumer->frames_to_time(m_consumer->position());
double speed = m_producer->get_speed();
Expand Down Expand Up @@ -654,8 +679,17 @@ bool Controller::isAudioFilter(const QString &name)

int Controller::realTime() const
{
int threadCount = QThread::idealThreadCount();
return threadCount > 2? qMin(threadCount - 1, 4) : 1;
int realtime = 1;
if (!Settings.playerRealtime()) {
if (Settings.playerGPU()) {
return -1;
} else {
int threadCount = QThread::idealThreadCount();
threadCount = threadCount > 2? qMin(threadCount - 1, 4) : 1;
realtime = -threadCount;
}
}
return realtime;
}

void Controller::setImageDurationFromDefault(Service* service) const
Expand Down

5 comments on commit ab3ce28

@ddennedy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This commit broke GPU processing. It it rather heavy-handed. Changing a bunch of consumer properties when pausing and playing? Is that what a MLT app needs to do properly handle play/pause? Ugly.

@ddennedy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, when I said it "broke GPU processing" I mean it crashes with GPU processing enabled. The following changes fixes it and still provides clean, dead-stop behavior most of the time:

diff --git a/src/mltcontroller.cpp b/src/mltcontroller.cpp
index 669c34f..63a94a8 100644
--- a/src/mltcontroller.cpp
+++ b/src/mltcontroller.cpp
@@ -204,8 +204,6 @@ void Controller::pause()
 {
     if (m_producer && m_producer->get_speed() != 0) {
         if (m_consumer && m_consumer->is_valid()) {
-            // Flush all frames from the video task in the consumer.
-            m_consumer->stop();
             // Disable real_time behavior and buffering for frame accurate seeking.
             m_consumer->set("real_time", 0);
             m_consumer->set("buffer", 0);
@@ -214,6 +212,7 @@ void Controller::pause()
         m_producer->set_speed(0);
         m_producer->seek(m_consumer->position() + 1);
         if (m_consumer && m_consumer->is_valid()) {
+            m_consumer->purge();
             m_consumer->start();
         }
     }
@@ -328,10 +327,9 @@ void Controller::seek(int position)
     if (m_producer) {
         // Always pause before seeking (if not already paused).
         if (m_consumer && m_consumer->is_valid() && m_producer->get_speed() != 0) {
-            // Flush all frames from the video task in the consumer.
-            m_consumer->stop();
             // Disable real_time behavior and buffering for frame accurate seeking.
-            m_consumer->set("real_time", 0);
+            if (!Settings.playerGPU())
+                m_consumer->set("real_time", 0);
             m_consumer->set("buffer", 0);
             m_consumer->set("prefill", 0);
         }

The removal of the consumer->stop()s might also be necessary for decklink output. I still have to check that. Also, if you are doing Encode > Stream (broken at the moment), then pause and seek are supported, and we do not want to interrupt the stream during that by calling stop.

@bmatherly
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Dan,

Thanks for the review. With your patch, I had some trouble resuming when GPU is enabled. I had better luck with this modified change:

diff --git a/src/mltcontroller.cpp b/src/mltcontroller.cpp
index 6cf2655..14f76b8 100644
--- a/src/mltcontroller.cpp
+++ b/src/mltcontroller.cpp
@@ -202,20 +202,19 @@

 void Controller::pause()
 {
     if (m_producer && m_producer->get_speed() != 0) {
         if (m_consumer && m_consumer->is_valid()) {
-            // Flush all frames from the video task in the consumer.
-            m_consumer->stop();
             // Disable real_time behavior and buffering for frame accurate seeking.
-            m_consumer->set("real_time", 0);
+            m_consumer->set("real_time", -1);
             m_consumer->set("buffer", 0);
             m_consumer->set("prefill", 0);
         }
         m_producer->set_speed(0);
         m_producer->seek(m_consumer->position() + 1);
         if (m_consumer && m_consumer->is_valid()) {
+            m_consumer->purge();
             m_consumer->start();
         }
     }
     if (m_jackFilter)
         m_jackFilter->fire_event("jack-stop");
@@ -334,14 +333,12 @@
 void Controller::seek(int position)
 {
     if (m_producer) {
         // Always pause before seeking (if not already paused).
         if (m_consumer && m_consumer->is_valid() && m_producer->get_speed() != 0) {
-            // Flush all frames from the video task in the consumer.
-            m_consumer->stop();
             // Disable real_time behavior and buffering for frame accurate seeking.
-            m_consumer->set("real_time", 0);
+            m_consumer->set("real_time", -1);
             m_consumer->set("buffer", 0);
             m_consumer->set("prefill", 0);
         }
         m_producer->set_speed(0);
         m_producer->seek(position);

With both your patch and the one above, I see that seeking is still frame accurate. But when pausing the last frame is output twice. I see this by adding a debug line at the begingin of GLWidget::on_frame_show():

diff --git a/src/glwidget.cpp b/src/glwidget.cpp
index b5f2a89..c19d2b9 100644
--- a/src/glwidget.cpp
+++ b/src/glwidget.cpp
@@ -618,10 +618,11 @@
 }

 // MLT consumer-frame-show event handler
 void GLWidget::on_frame_show(mlt_consumer, void* self, mlt_frame frame_ptr)
 {
+   qDebug() << mlt_frame_get_position(frame_ptr);
     GLWidget* widget = static_cast<GLWidget*>(self);
     int timeout = (widget->consumer()->get_int("real_time") > 0)? 0: 1000;
     if (widget->m_frameRenderer && widget->m_frameRenderer->semaphore()->tryAcquire(1, timeout)) {
         Mlt::Frame frame(frame_ptr);
         if (frame.get_int("rendered")) {

When pausing, I frequently see this result:

[Debug  ] <static void Mlt::GLWidget::on_frame_show(mlt_consumer, void*, mlt_frame)> 121 
[Debug  ] <static void Mlt::GLWidget::on_frame_show(mlt_consumer, void*, mlt_frame)> 122 
[Debug  ] <static void Mlt::GLWidget::on_frame_show(mlt_consumer, void*, mlt_frame)> 123 
[Debug  ] <static void Mlt::GLWidget::on_frame_show(mlt_consumer, void*, mlt_frame)> 124 
[Debug  ] <static void Mlt::GLWidget::on_frame_show(mlt_consumer, void*, mlt_frame)> 125 
[Debug  ] <static void Mlt::GLWidget::on_frame_show(mlt_consumer, void*, mlt_frame)> 126 
[Debug  ] <static void Mlt::GLWidget::on_frame_show(mlt_consumer, void*, mlt_frame)> 126

I suppose that is tolerable for now.

Regarding "heavy handed":
I would have thought the same thing until I did some profiling. With the previous code, when the user pauses, the consumer pulls 25 duplicate frames into the consumer buffer. Then, when you seek, those 25 frames are purged and 25 more duplicate frames are loaded. That's 25 image malloc/memcpy operations for video and 25 more for audio - and then another 50 free operations - for no reason. But by chaning the consumer properties that all gets reduced to one unnecessary frame loaded in the buffer.

I'll commit my patch above if you're OK with that.

@ddennedy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, your version of the patch works for me too. Go ahead.

@bmatherly
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
377eb4f

Please sign in to comment.