Multithreading processing chain

[pozitronik]

In the current processing logic all frame processors runs in turns, one after one. The advantages of this logic are obvious: it is simple, there is a possibility to stop and resume processing at any time, e.t.c.
But there are disadvantages too, and they seems to be critical at this moment. There is no possibility to implement any realtime preview with this logic, and there is no place for speed enhancements, because we can't parallelize processing task.
So, there is a big (really big) task to implement multithreading there.

There will be named buffers, each buffer named to a linked processor. Each processor processing will start in the separate thread, it will read a frame from the input buffer, process it, and write to the output buffer (which is input buffer for the next processor).

There are subtasks for this task:

• Implement the buffered processing chain.
• Make *FrameBuffer classes for the different buffers types (memory, file, e.t.c).
• Implement threads manager. Some processors may work faster than others, so their buffers will fill quickly. So it make a sense to pause those processors threads to redistribute computing powers to the other threads.
• Implement states in this approach. Saving and loading buffers as a files can be useful: this logic partially coincides with the current state logic.
• Different modules settings via config file #68
• Implement a memory control code. It is important, because several loaded models may require a lot of memory, and frame buffers are too.
• Allow to control each processor sub-threads count (ThreadPoolExecutor workers). #70
• Show the general progress bar and the every processor progress bar and statistics (memory usage, buffers occupancy, e.t.c).
• Properly handle in-threads exceptions.

[pozitronik]

Intermediate tests on 497 frames 1920x1080 video with the same processing chain (in-memory extracting -> FaceSwapper->FaceEnhancer):
Parallel execution (cpu+cuda execution providers, 2 threads):14% in 12 min -> approximately an hour.
Parallel execution (FaceSwapper uses 16 CPU threads, FaceEnhancer uses 2 CUDA threads): ~724s
Parallel execution (FaceSwapper uses 16 CPU threads, FaceEnhancer uses 2 CUDA threads, frame extractor buffer lock on 10 frames): ~700s (with 1 second thread waiting), ~717s (0.5).
All other parallel tries are even worse.

Current master (cpu+cuda execution providers): ~580s (18% faster of best parallel result).

It is worth to try to play some thread-control logic. it seems, that in some cases parallel execution can speed up the whole processing, and slow it down in other.
Other conclusion: swapper and enhancer can't properly work simultaneously, if they share some hardware (it was predictable, but still worth to try). They can work in parallel with different hardware providers (e.g cpu/gpu), but even in that case the whole process will be slower.
Note: currently i can try my tests only on my laptop, which may redistribute power to cpu/gpu to preserve consumption/heat limits, which can also affect the tests.

[pozitronik]

Due to unsatisfactory experiments result and changes in the common processing logic the task is closed.