Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exiftool: Large File Support #1401

Closed
BetaAthe opened this issue Jul 2, 2021 · 20 comments
Closed

Exiftool: Large File Support #1401

BetaAthe opened this issue Jul 2, 2021 · 20 comments
Assignees
Labels
enhancement Optimization, improvement or maintenance task released Available in the stable release

Comments

@BetaAthe
Copy link

BetaAthe commented Jul 2, 2021

Hi. Doing some testing, I am unable to play large videos on my Photoprism instance. With large videos, I mean close to 3GB .mp4 avc1 encoded videos.

I have a folder with more than 10 videos, all encoded the same way. All the videos with size < 2.2 GB are playable, and all of them with size > 2.9GB are not. I don't have any video in the 2.2-2.9 range.

When I try to play that videos, ffmpeg kicks in consuming all the cpu of my server. Disabling ffmpeg transcoding makes the video unplayable.

Also, I saw that those video's metadata was wrong, so digging in a little found out that it was a problem with exiftools, solved by writting

%Image::ExifTool::UserDefined::Options = (
    LargeFileSupport => 1,
);

into the .ExifTool_config file in the /root directory.

@rickysarraf
Copy link
Contributor

Speaking for myself, I have preferred to re-encode my video files, carefully choosing a setting, which keeps the video file's size substantially low. The reason for doing so is simply because those files are rarely to be viewed on a large display. My expected clients of PhotoPrism, in all probability, will mostly be PC, Tablet and Mobile. For these device form factors, I don't really need video files with very high bit rate. But this is purely my personal choice.

File sizes greater than 2 GiB are ignored by PP. I think it logs that information in its system logs.

@BetaAthe
Copy link
Author

BetaAthe commented Jul 2, 2021

Thanks for the response! These videos are more than 3 hours long, as they are VHS digitalizations, and I think they are already compressed to an acceptable threshold. I understand that one possibility is to split them into two or more files, but I would prefer to not resort to that.

To be able to upload them to PhotoPrism, I had to add the PHOTOPRISM_ORIGINALS_LIMIT: 5000 variable.

As I saw that 2.2 GiB videos are working flawlessly, I thought it was a matter of changing one setting or removing one hard limit from the app.

@lastzero
Copy link
Member

lastzero commented Jul 2, 2021

As the name suggests, we've designed PhotoPrism for still pictures and short clips like live photos. It MAY not be the best tool for managing full-length movies, especially if you have a lot of them.

Increasing PHOTOPRISM_ORIGINALS_LIMIT to the max file size in MB and then restarting your instance should work though. If your videos are not encoded with MP4 / AVC1, we recommend you transcode them upfront by running photoprism convert in a terminal to avoid high server load and long delays when playing them in the UI.

@lastzero lastzero self-assigned this Jul 2, 2021
@lastzero lastzero added the technical-support Support request label Jul 2, 2021
@lastzero
Copy link
Member

lastzero commented Jul 2, 2021

Please close this issue if your question is answered with that :)

@BetaAthe
Copy link
Author

BetaAthe commented Jul 2, 2021

Hi! I've added that variable and restarted the instance to be able to index them in the library. All the videos are encoded using the same preset, so they are basically the same. Also, I checked them using exiftools and all of them reported the same codec, AVC1. In the PhotoPrism GUI, large playable videos report "AVC1" and the very large unplayable ones report "MP4", for some reason.

I know that PhotoPrism was not made for large videos in mind, and can understand that this issue could be marked as wontfix because of that, but as I saw that this software is able to reproduce some large videos without trouble, I considered this case a bug as, I think, nothing prevents these very large videos for playing properly.

@lastzero
Copy link
Member

lastzero commented Jul 3, 2021

PhotoPrism uses exiftool to detect the codec, so it should show AVC1 if exiftool does. Did you try a full rescan of the affected folders (don't need to rescan your complete library)?

@BetaAthe
Copy link
Author

BetaAthe commented Jul 3, 2021

Exiftools wasn't able to open this video files at first. I had to follow this forum post to being able to read them [1]. Once done, exiftools read the codec correctly. I've done the same inside the PhotoPrism Docker container but the reported codec is still MP4 in the PhotoPrism web interface after the rescan. Thumbnails are created though.

@lastzero
Copy link
Member

lastzero commented Jul 3, 2021

Then it's probably because we didn't enable large file support in exiftool. Imagine this could lead to memory issues or DOS attacks.

@BetaAthe
Copy link
Author

BetaAthe commented Jul 3, 2021

Ok, got it. I understand your concerns about memory or DOS but, is there a way to tell PhotoPrism to use exiftool with large files support?

@lastzero
Copy link
Member

lastzero commented Jul 3, 2021

Need to change this in our code as the parameters can not be configured, there are already a ton of config options nobody understands.

@lastzero lastzero added enhancement Optimization, improvement or maintenance task and removed technical-support Support request labels Jul 3, 2021
@lastzero lastzero changed the title Large videos support Exiftool: Large File Support Jul 3, 2021
lastzero added a commit that referenced this issue Jul 8, 2021
This also enables large file support to read metadata from
movie length video files.
@lastzero
Copy link
Member

lastzero commented Jul 8, 2021

Added the Exiftool flag as requested, and started a new preview build:

https://drone.photoprism.app/photoprism/photoprism/1456

Let us know if this works for you!

@BetaAthe
Copy link
Author

BetaAthe commented Jul 8, 2021

I checked the photoprism/demo:latest container (don't know if that's the right container) and the issue persists. I want to test more things to try to understand why it doesn't work. I'll comment if I find something.

@lastzero
Copy link
Member

lastzero commented Jul 8, 2021

Use the preview tag as shown in our docs / release notes / example config.

@lastzero
Copy link
Member

lastzero commented Jul 8, 2021

Also start with a fresh installation or delete the existing JSON sidecar files so that they get recreated.

@BetaAthe
Copy link
Author

BetaAthe commented Jul 9, 2021

Ok, I tested removing everything and starting again with the preview tag, but it's not working. I've done some testing:

First, I rendered 6 mp4 test files using AVC1 codec with 1:30, 2:30 and 3:30 hours in length and different sizes. (Size is under "Tamaño" and length under "Duración").
testfiles

This is the import log:

2021-07-09 12:06:06 INFO index: added related jpg file test6.mp4.jpg

2021-07-09 12:06:05 INFO media: test6.mp4.jpg was taken at 2021-07-09 10:06:16.045067349 +0000 UTC (file mod time)

2021-07-09 12:06:05 INFO index: added main mp4 file tests/test6.mp4

2021-07-09 12:06:05 INFO media: test6.mp4 was taken at 2021-07-09 09:58:53.032380405 +0000 UTC (file mod time)

2021-07-09 12:06:05 INFO media: 11 thumbnails created for test6 [203.897678ms]

2021-07-09 12:06:04 INFO converting tests/test6.mp4 to jpg

2021-07-09 12:06:01 INFO index: added related jpg file test4.mp4.jpg

2021-07-09 12:06:00 INFO media: test4.mp4.jpg was taken at 2021-07-09 10:06:00.272474298 +0000 UTC (file mod time)

2021-07-09 12:06:00 INFO index: added main mp4 file tests/test4.mp4

2021-07-09 12:06:00 INFO media: test4.mp4 was taken at 2021-07-09 09:57:55.12619302 +0000 UTC (file mod time)

2021-07-09 12:06:00 INFO media: 11 thumbnails created for test4 [203.03006ms]

2021-07-09 12:05:59 INFO converting tests/test4.mp4 to jpg

2021-07-09 12:05:49 INFO index: added related jpg file test5.mp4.jpg

2021-07-09 12:05:48 INFO index: added main mp4 file tests/test5.mp4

2021-07-09 12:05:48 INFO media: 11 thumbnails created for test5 [200.048314ms]

2021-07-09 12:05:47 INFO converting tests/test5.mp4 to jpg

2021-07-09 12:05:46 INFO folder: added album Vhs (path:vhs public:true)

2021-07-09 12:05:46 INFO index: added related jpg file test2.mp4.jpg

2021-07-09 12:05:45 INFO index: added related jpg file test1.mp4.jpg

2021-07-09 12:05:45 INFO index: added related jpg file test3.mp4.jpg

2021-07-09 12:05:32 INFO index: added main mp4 file tests/test2.mp4

2021-07-09 12:05:32 INFO media: 11 thumbnails created for test2 [273.312459ms]

2021-07-09 12:05:32 INFO converting tests/test2.mp4 to jpg

2021-07-09 12:05:30 INFO index: added main mp4 file tests/test3.mp4

2021-07-09 12:05:30 INFO index: added main mp4 file tests/test1.mp4

2021-07-09 12:05:30 INFO media: 11 thumbnails created for test3 [301.142623ms]

2021-07-09 12:05:30 INFO media: 11 thumbnails created for test1 [281.767971ms]

2021-07-09 12:05:27 INFO converting tests/test3.mp4 to jpg

2021-07-09 12:05:27 INFO converting tests/test1.mp4 to jpg

2021-07-09 12:05:25 INFO index: added folder /tests

Doing this, files test1, test2, test3 and test5 work, but test4 and test6 don't. I tested with Firefox 89 and Microsoft Edge.
works
dont work

This is the data of one working file and one that doesn't:
stats yes
stats no

@lastzero
Copy link
Member

Currently downloading a 10h video from YouTube so that we can figure out how to properly provide multiple values to exiftool -api. Haven't found useful examples in their docs. Going to build a new preview image after having tested it locally.

@lastzero
Copy link
Member

Added -m in addition to -api LargeFileSupport, seems to work with this. Preview build has been started and should be available within 1 or 2 hours (might have to restart if it fails):

https://drone.photoprism.app/photoprism/photoprism/1468

@BetaAthe
Copy link
Author

Looks like it's solved now! Thanks!

@lastzero
Copy link
Member

Excellent, thanks for testing! 🥳

@lastzero
Copy link
Member

Development preview and https://demo.photoprism.org/ have been updated again. Had to remove -api QuickTimeUTC from Exiftool parameters as it turned out not to really fix MP4 / Quicktime video time zones. Now implemented differently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Optimization, improvement or maintenance task released Available in the stable release
Projects
Status: Release 🌈
Development

No branches or pull requests

4 participants