Movie import #1258

scribblemaniac · 2019-08-31T08:15:22Z

This PR adds the ability to import movies. From the usual import menu, there are now two options, "Movie Video..." and "Movie Audio..." which are meant to be used to extract the frames to a bitmap layer and the audio to a sound layer respectively.

Honestly I was feeling a bit lazy while doing this so I know there is room for improvement. My priority was just getting a working version ready for the next release as this is a very helpful feature for those users who aren't already experts with ffmpeg 😉 There is still the opportunity to add an import dialog with additional features to this such as: only importing from a specific start and end time (easy for ffmpeg), audio/video channel selection, or even some kind of preview. There could be some progress bar for the audio import, but you'd have to parse the duration out from the ffmpeg output which is not as easy as the frame number for the video import. The error messages are very unhelpful. There are a few things in the import code I came across that probably need refactoring, particularly moving some things out of mainwindow. Just ideas for the future, I personally think this should be good enough for use as-is.

Closes #1070

ffmpeg apparently doesn't set the exit code sometimes

davidlamhauge · 2019-08-31T08:47:21Z

It is a very basic implementation, but it looks very promising.
When testing it, I accidentally started importing a 10 minute movie, and I had no chance to abort it. This should be fixed.
Otherwise I haven't found problems, but there is room for improvement.
I've not done a thorough test, but aside from the ability to abort the import, there are nothing wrong with merging this, and see what happens when the users use it.

Jose-Moreno · 2019-08-31T15:43:07Z

Excellent work! This has been a most requested over the years, so it is a welcome addition!

I think however that this one could wait at least for a progress bar with a cancel button before merging. Indeed having an import dialog with preview and trimming options (start / end time) would be great for future enhancements, but for the bare minimum of visual feedback, a progress bar, and the ability to cancel the operation are required to communicate to the user what is happening and that there is are no (obvious) errors, or if there's a user mistake to roll back the operation before waiting too long (as David commented).

I honestly can't tell you how many times I've thought a program has locked up because I didn't see visual feedback for long import operations and closed the program thinking it crashed 😂

With the new undo / redo scheme this should also have an accompanying action that rolls back all the imported frames, or reapplies the imported frames for a movie clip in case there was an unattended user error and the amount of frames is too long to delete manually frame-by-frame (and because consistency).

Since I know your time is limited, maybe if someone else want to contribute to your branch and consequently to this PR to add these qualifying specs, it should be possible. Either way thanks again for tackling this, hopefully the others can chime in with their comments 🙏

scribblemaniac · 2019-08-31T17:57:50Z

All the operations are practically cancellable now.

MrStevns

It's a nice addition for sure but the fact that we're getting another dependency, though static, will nudge the application size 70 mb or so extra... this will probably take pencil2d into 200 mb range, which I'm not exactly excited about. The application is still relatively simple and yet the size of the program enter the range of older adobe software.

scribblemaniac · 2019-09-01T08:12:28Z

@candyface There's plenty of room for improvement in that regard for whomever is adventurous enough. The ffmpeg/ffprobe binaries could be decreased significantly by compiling a more minimal version. The Qt components could probably be reduced with Qt Lite. You could remove the ffprobe dependency entirely if you parsed the duration from the ffmpeg output, but that would be a pain and much more prone to breaking in the future.

Of the things we can optimize with the application (speed, memory, and size), I think size is the least important of them. Of course we shouldn't be reckless with it, but space is a fairly abundant resource for most modern systems, and app sizes are naturally larger than they used to be.

scribblemaniac · 2019-09-07T23:58:11Z

I've sweetened the deal 😁 Now this also adds support for many more audio file formats (*.wma *.ogg *.flac *.opus *.aiff *.aac *.caf) and "fixes" the mp3 playback issue on windows by converting it to a wav on import.

MrStevns · 2019-09-08T08:39:37Z

I've sweetened the deal 😁 Now this also adds support for many more audio file formats (*.wma *.ogg *.flac *.opus *.aiff *.aac *.caf) and "fixes" the mp3 playback issue on windows by converting it to a wav on import.

Awesome. I think it was always a good idea to convert the audio files to wav internally when importing, to make sure codecs wouldn't be a problem. The user doesn't notice anyway so it just makes it easier for us.

This doesn't make use of FFprobe though does it?

MrStevns

LGTM code wise. Haven't tested it.

Jose-Moreno · 2019-09-08T16:34:34Z

@scribblemaniac Awesome! A quick question though, will these formats be only for videos with audio containers with those codecs or will this improvement expand to the sound import dialog as well? (I hope it does 🙏 )

scribblemaniac · 2019-09-08T20:29:54Z

This doesn't make use of FFprobe though does it?

No it does not, but that's mostly because there is no progress bar for sound import.

A quick question though, will these formats be only for videos with audio containers with those codecs or will this improvement expand to the sound import dialog as well?

They are are part of the Import > Sound action, but they use the same function as the movie audio import behind the scenes. It should work with virtually any audio codec in videos already.

davidlamhauge

Apart from the comment on a maximum amount of frames, I have no issues here.
LGTM.

davidlamhauge · 2019-09-09T16:00:46Z

core_lib/src/interface/editor.cpp

+        return false;
+    }
+
+    // Get frame estimate


You make a frame estimate here. I think it should be used to limit the amount of frames to be imported.
I imported a mp4, that lasted 13 seconds. It took a minute.
After that I tried to import a mp4 that lasted 3:56 minutes. After 4-5 minutes it started importing pictures. After importing about 350 frames, the software crashed.
What is the biggest video you've tried to import?
Shouldn't we set a maximum of 300-500 frames?

There is no technical limit. If you are on Windows, it probably crashed because you ran out of memory, but that's entirely dependent on the system and is difficult for us to set a good limit on. I can't test this right now, but I will later.

One solution is to output the frame from ffmpeg directly to the temporary folder (or moving files into the temporary folder), which prevents having to have all the frames loaded into memory and would cut the import time in half, however this would not be good for pcl files where writing to the temporary folder is practically the equivalent of saving.

We could load in chunks at a time, since we can't rely on people understanding that their video might generates thousands of images, this will still take a hit in loading time but imo. it's better than crashing the program and won't be as bad as moving frames to the temporary folder for each frame.

Also allow only the text of the image import filter to be translated.

I'm not in favor of relying upon ffmpeg for retrieving the duration. It is more difficult to parse and highly suseptible to changes in format between ffmpeg versions and possibly locatalizations. I have made the ffmpeg duration code a fallback if ffprobe is absent, that way if it does cause issues, we can just tell users to throw the ffprobe binary in the plugins folder and it will just work.

scribblemaniac · 2019-09-14T07:49:20Z

FFprobe is optional now 😑

MrStevns · 2019-09-28T18:07:41Z

Can be merged as soon as the conflicts has been resolved.

scribblemaniac · 2019-09-28T21:03:12Z

I will resolve the conflicts soon, but I'm actually going to move this to [WIP] for now. I have an idea to reduce the memory burden of this feature.

This change is so that all frames that have not been modified since importing from a movie can be unloaded from memory and the frame cache. Saving is done by copying files on the disk. This should prevent out of control memory consumption during movie import, and should make saving said frames very fast.

The previous method worked on metadata tags as far as I can tell, which are often not present in the output. This uses the stream duration which should be about as reliable as you can get with the wrong tool.

scribblemaniac · 2020-03-04T10:12:14Z

I've just implemented the idea mentioned earlier. Frames are saved to the disk right away (but are not part of the project file until it is actually saved). This means lower memory usage, less crashing, and faster saving. Unfortunately this only works for frames being added to a space without an existing keyframe and there's no way around that since the import frame has to be merged with the existing frame.

I've also resolved the merge conflicts. I believe this feature is ready to be added. There are a few quality of life improvements I would still like to see added including import settings similar to the image sequence import (spacing, alignment, ...), better temporary folder cleanup, better error handling, and warnings for large imports, however I feel that this is good enough now that it should be added to v0.6.5. The new audio import is particularly important with many of our users still having issues with mp3 files. Given that @davidlamhauge and @candyface have already reviewed most of this, I am hopeful that it will not take too long to finish reviewing.

MrStevns · 2020-03-04T10:16:24Z

Awesome, I will review it today.

MrStevns

In general I think it okay, personally I think that error cases should be handled before merging as we otherwise have no way of helping our users if movie import/export fails in a non debug build.

if we create a follow-up issue that highlights the TODO's and the missing error handling, then I'm fine with that too.

…business logic This was a bit more difficult than anticipated but the result now is that the process is completely independent of the progress dialog and instead can notify UI if required. Bonus is also that we avoid polluting Editor further.

Not just a refactor though, I also fixed some potential bugs and improved error logging with DebugDetails and ErrorDialog.

MrStevns · 2020-05-07T05:32:46Z

app/src/actioncommands.cpp

+    if(st == Status::CANCELED)
+    {
+        QMessageBox::warning(mParent,
+                             st.title(),
+                             st.description(),
+                             QMessageBox::Ok,
+                             QMessageBox::Ok);
+    }
+    else if (!st.ok())
+    {
+        ErrorDialog errorDialog(st.title(), st.description(), st.details().html(), mParent);
+        errorDialog.exec();
+    }


This is not correct, now the user will see an empty dialog every time they cancel the operation. There's no explanation, just a window with a exclamation mark. The reason it was "!st.ok() && st != Status::CANCEL" is because it's only meant to trigger a popup if the operation fails.

Right now you'll see this if you cancel, canceling is already an explicit action, we don't need further confirmation that it's done, therefore it should be silent.

Yeah, I was trying to get a message to display when it was cancelled because it exceeds the frame limit so I thought I'd change the user cancelled events to SAFE, but I did not realize how many points there were where it could be cancelled by the user. And I also didn't want to use ErrorDialog for it because we don't need the extra debug information for importing something that is too long. I have updated it with a solution that should work better. Let me know what you think.

MrStevns · 2020-05-07T05:42:32Z

app/src/actioncommands.cpp

+    if(st == Status::CANCELED)
+    {
+        QMessageBox::warning(mParent,
+                             st.title(),
+                             st.description(),
+                             QMessageBox::Ok,
+                             QMessageBox::Ok);
+    }
+    else if (!st.ok())
+    {
+        ErrorDialog errorDialog(st.title(), st.description(), st.details().html(), mParent);
+        errorDialog.exec();
+    }


Same as above..

MrStevns · 2020-05-07T05:43:54Z

app/src/actioncommands.cpp

+        ErrorDialog errorDialog(st.title(), st.description(), st.details().html(), mParent);
+        errorDialog.exec();
+    }
+    return st;


Same as above

MrStevns · 2020-05-07T05:44:56Z

core_lib/src/interface/editor.cpp

@@ -25,6 +25,7 @@ GNU General Public License for more details.
 #include <QImageReader>
 #include <QDragEnterEvent>
 #include <QDropEvent>
+#include <QProgressDialog>


We should remove this again

MrStevns · 2020-05-07T05:45:17Z

core_lib/src/interface/editor.h

@@ -27,6 +27,8 @@ GNU General Public License for more details.

 class QDragEnterEvent;
 class QDropEvent;
+class QProgressDialog;


This too should be removed

MrStevns

Nice improvement to the error handling on top of my own changes, the latest commit introduced a bug however that needs to be fixed before we can merge. See comments :)

MrStevns

LGTM!

Thanks for making this and for the patience, I'm certain the users will find it valuable :)

scribblemaniac added 3 commits August 31, 2019 01:48

Add movie importing features

2109f8f

Add ffprobe to CI builds

ce079a8

Detect failed movie video import

4218c7a

ffmpeg apparently doesn't set the exit code sometimes

scribblemaniac added 3 commits August 31, 2019 10:59

Add cancel to movie import and audio mixing step of movie export

1bb7f1b

Add cancel to movie audio import

036bcd3

Add progress updates and cancel to numbered image sequence import

e8593f1

MrStevns reviewed Sep 1, 2019

View reviewed changes

scribblemaniac added the import-export label Sep 2, 2019

Added support for many new audio formats

5aa5fdc

MrStevns approved these changes Sep 8, 2019

View reviewed changes

davidlamhauge approved these changes Sep 9, 2019

View reviewed changes

scribblemaniac added this to the 0.6.5 milestone Sep 13, 2019

scribblemaniac added the 🔷 Major PR (two reviewers when possible) label Sep 13, 2019

scribblemaniac added 2 commits September 14, 2019 01:00

Create a Movies import format that matches all supported extensions

45aaada

Also allow only the text of the image import filter to be translated.

scribblemaniac changed the title ~~Movie import~~ [WIP] Movie import Sep 28, 2019

Jose-Moreno requested a review from chchwy December 16, 2019 05:16

Jose-Moreno modified the milestones: 0.6.5, 0.6.6 Jan 15, 2020

scribblemaniac added 4 commits March 3, 2020 22:01

Merge branch 'master' into movie-import

3650d8a

Change ffmpeg-based duration detection

d458c10

The previous method worked on metadata tags as far as I can tell, which are often not present in the output. This uses the stream duration which should be about as reliable as you can get with the wrong tool.

Add some checks and TODOs to the movie importer

3b4aa3a

scribblemaniac modified the milestones: 0.6.6, 0.6.5 Mar 4, 2020

scribblemaniac changed the title ~~[WIP] Movie import~~ Movie import Mar 4, 2020

Delete temporary folders on closing without saving

055e626

MrStevns approved these changes Mar 4, 2020

View reviewed changes

MrStevns and others added 4 commits May 4, 2020 16:43

Merge branch 'master' into pr-1258-changes

9804aec

Rename verifyFFMPEG -> verifyFFmpegExists

df12488

Refactor movie importer...again

ea33310

Not just a refactor though, I also fixed some potential bugs and improved error logging with DebugDetails and ErrorDialog.

scribblemaniac force-pushed the movie-import branch from 0cb1e34 to ea33310 Compare May 7, 2020 02:57

MrStevns reviewed May 7, 2020

View reviewed changes

MrStevns requested changes May 7, 2020

View reviewed changes

scribblemaniac added 2 commits May 7, 2020 00:57

Fix blank dialog when cancelling movie import

1dc3ade

Remove unnecessary QProgressDialog reference in Editor

514f130

MrStevns approved these changes May 7, 2020

View reviewed changes

MrStevns merged commit fef4858 into pencil2d:master May 7, 2020

scribblemaniac mentioned this pull request Dec 21, 2020

Mp3 playback issue on Windows #1237

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Movie import #1258

Movie import #1258

scribblemaniac commented Aug 31, 2019

davidlamhauge commented Aug 31, 2019

Jose-Moreno commented Aug 31, 2019

scribblemaniac commented Aug 31, 2019

MrStevns left a comment

scribblemaniac commented Sep 1, 2019

scribblemaniac commented Sep 7, 2019

MrStevns commented Sep 8, 2019 •

edited

Loading

MrStevns left a comment •

edited

Loading

Jose-Moreno commented Sep 8, 2019

scribblemaniac commented Sep 8, 2019 •

edited

Loading

davidlamhauge left a comment

davidlamhauge Sep 9, 2019

scribblemaniac Sep 14, 2019

MrStevns Sep 14, 2019

scribblemaniac commented Sep 14, 2019

MrStevns commented Sep 28, 2019

scribblemaniac commented Sep 28, 2019

scribblemaniac commented Mar 4, 2020

MrStevns commented Mar 4, 2020

MrStevns left a comment •

edited

Loading

MrStevns May 7, 2020 •

edited

Loading

scribblemaniac May 7, 2020

MrStevns May 7, 2020 •

edited

Loading

MrStevns May 7, 2020

MrStevns May 7, 2020

MrStevns May 7, 2020

MrStevns left a comment •

edited

Loading

MrStevns left a comment •

edited

Loading

Movie import #1258

Movie import #1258

Conversation

scribblemaniac commented Aug 31, 2019

davidlamhauge commented Aug 31, 2019

Jose-Moreno commented Aug 31, 2019

scribblemaniac commented Aug 31, 2019

MrStevns left a comment

Choose a reason for hiding this comment

scribblemaniac commented Sep 1, 2019

scribblemaniac commented Sep 7, 2019

MrStevns commented Sep 8, 2019 • edited Loading

MrStevns left a comment • edited Loading

Choose a reason for hiding this comment

Jose-Moreno commented Sep 8, 2019

scribblemaniac commented Sep 8, 2019 • edited Loading

davidlamhauge left a comment

Choose a reason for hiding this comment

davidlamhauge Sep 9, 2019

Choose a reason for hiding this comment

scribblemaniac Sep 14, 2019

Choose a reason for hiding this comment

MrStevns Sep 14, 2019

Choose a reason for hiding this comment

scribblemaniac commented Sep 14, 2019

MrStevns commented Sep 28, 2019

scribblemaniac commented Sep 28, 2019

scribblemaniac commented Mar 4, 2020

MrStevns commented Mar 4, 2020

MrStevns left a comment • edited Loading

Choose a reason for hiding this comment

MrStevns May 7, 2020 • edited Loading

Choose a reason for hiding this comment

scribblemaniac May 7, 2020

Choose a reason for hiding this comment

MrStevns May 7, 2020 • edited Loading

Choose a reason for hiding this comment

MrStevns May 7, 2020

Choose a reason for hiding this comment

MrStevns May 7, 2020

Choose a reason for hiding this comment

MrStevns May 7, 2020

Choose a reason for hiding this comment

MrStevns left a comment • edited Loading

Choose a reason for hiding this comment

MrStevns left a comment • edited Loading

Choose a reason for hiding this comment

MrStevns commented Sep 8, 2019 •

edited

Loading

MrStevns left a comment •

edited

Loading

scribblemaniac commented Sep 8, 2019 •

edited

Loading

MrStevns left a comment •

edited

Loading

MrStevns May 7, 2020 •

edited

Loading

MrStevns May 7, 2020 •

edited

Loading

MrStevns left a comment •

edited

Loading

MrStevns left a comment •

edited

Loading