Use multiple CPU processes in scanning and preview generation when importing multiple files

[ollm]

Description of the feature or enhancement you'd like to see in HandBrake

Currently, when I have to import a large number of 4k videos (500-1000, sometimes more), HandBrake takes a long time to scan and generate video previews (1 hour or more), because it only uses a CPU process, would it be possible for HandBrake to scan and generate preview of multiple files at once so that it uses all available CPU processes?

Thanks.

What Operating System are you running?

Windows 10 and Ubuntu 22.04

What version of HandBrake are you running?

v1.5.1 on both

Where did you download HandBrake from?

No response

Activity Log, Crash Log or any other details

No response

[sr55]

While a solid idea in principal, It's unfortunately fraught with potential issues.

The ideal solution would be to simply split the scan up by a few threads but I'd rekon we'd just end up with re-entrant problems with the decoders.
Multi-process has a bunch of state related issues that make it awkward and probably non-viable.

I believe the Linux UI supports multi-instance so you can run multiple GUI's at once. You might try opening up say 2 or more. If your drives are standard HDD, you may need to monitor as there will be a tipping point where performance gets astronomically worse. I'm not 100% sure where that would lie but I'd be surprised if it wasn't fine for at least a few instances.

I believe there may still be an open bug somewhere regarding the performance of "add to queue" on Linux. I believe there was an issue where it bogs down on large sets of files but I can't find the issue at the moment. In theory ~650 should be at most seconds adding to the queue. If it's longer, let me know. (May have been fixed. I'm not sure)

[ollm]

Thanks for the reply.

I believe the Linux UI supports multi-instance so you can run multiple GUI's at once. You might try opening up say 2 or more. If your drives are standard HDD, you may need to monitor as there will be a tipping point where performance gets astronomically worse. I'm not 100% sure where that would lie but I'd be surprised if it wasn't fine for at least a few instances.

I have been trying as you mentioned to run 8 instances of HandBrake at the same time to see how it works, on an internal SSD it does not reach the reading limit, on an external HDD it does reach it, but it uses much more CPU than in 1 single instance, memory usage does not increase too much (1GB - 2GB).

I have tried all this in separated folders in the SSD/HDD, to avoid using RAM cache of Linux.

Memory usage before scanning

Memory usage during scanning

1 HandBrake instance generating previews, CPU usage in SSD

8 HandBrake instances generating previews, CPU usage in SSD

1 HandBrake instance generating previews, CPU usage in External HDD

8 HandBrake instances generating previews, CPU usage in External HDD

8 HandBrake instances generating previews during 10 minutes, CPU usage in External HDD

I believe there may still be an open bug somewhere regarding the performance of "add to queue" on Linux. I believe there was an issue where it bogs down on large sets of files but I can't find the issue at the moment. In theory ~650 should be at most seconds adding to the queue. If it's longer, let me know. (May have been fixed. I'm not sure)

So far I have not had any problems adding the videos to the queue in Linux, it only takes a few seconds.