Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an option to limit concurent 4K transcodes (and eventually other resolution parallel transcodes...) #936

Closed
silkyclouds opened this issue Dec 24, 2023 · 1 comment

Comments

@silkyclouds
Copy link

silkyclouds commented Dec 24, 2023

Is your feature request related to a problem? Please describe.
Yes, i notice that my server, that has 64GB of RAM, will fail encoding files for the following reason :

2023-12-23T22:55:03.053Z Uh-_nhUa-:Node[MyInternalNode]:Worker[low-loon]:Running FFmpeg failed
2023-12-23T22:55:03.053Z Uh-_nhUa-:Node[MyInternalNode]:Worker[low-loon]:[-error-]
2023-12-23T22:55:03.054Z Uh-_nhUa-:Node[MyInternalNode]:Worker[low-loon]:Error: FFmpeg failed
2023-12-23T22:55:03.054Z Uh-_nhUa-:Node[MyInternalNode]:Worker[low-loon]:"FFmpeg failed"
2023-12-23T22:55:03.055Z Uh-_nhUa-:Node[MyInternalNode]:Worker[low-loon]:"Error: FFmpeg failed\n    at /app/Tdarr_Node/assets/app/plugins/FlowPlugins/CommunityFlowPlugins/ffmpegCommand/ffmpegCommandExecute/1.0.0/index.js:192:27\n    at step (/app/Tdarr_Node/assets/app/plugins/FlowPlugins/CommunityFlowPlugins/ffmpegCommand/ffmpegCommandExecute/1.0.0/index.js:33:23)\n    at Object.next (/app/Tdarr_Node/assets/app/plugins/FlowPlugins/CommunityFlowPlugins/ffmpegCommand/ffmpegCommandExecute/1.0.0/index.js:14:53)\n    at fulfilled (/app/Tdarr_Node/assets/app/plugins/FlowPlugins/CommunityFlowPlugins/ffmpegCommand/ffmpegCommandExecute/1.0.0/index.js:5:58)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
2023-12-23T22:55:03.055Z Uh-_nhUa-:Node[MyInternalNode]:Worker[low-loon]:Flow has failed
2023-12-23T22:55:03.055Z Uh-_nhUa-:Node[MyInternalNode]:Worker[low-loon]:[Step W09] [-error-] Job end
2023-12-23T22:55:03.056Z Uh-_nhUa-:Node[MyInternalNode]:Worker[low-loon]:Transcoding error encountered. Check sections above.
2023-12-23T22:55:03.057Z Uh-_nhUa-:Node[MyInternalNode]:Worker[low-loon]:[Step W10] Worker processing end
2023-12-23T22:55:03.057Z Uh-_nhUa-:Node[MyInternalNode]:Worker[low-loon]:Successfully updated server with verdict: transcodeError
2023-12-23T22:55:03.059Z Uh-_nhUa-:Node[MyInternalNode]:Worker[low-loon]:File processing complete

failures happens when I run two 4K transcodes in parallel, it seems the RAM gets overused, which results in FFMPEG to crash.

Describe the solution you'd like
Add an option to allow only XX parallel transcoded of a specific resolution (Eg: only 1*4K transcode can run at a time) would avoid the RAM usage to raise at a point the job will fail. I noticed that I can run 4K and 1080p in parallel wiuthout filling the RAM, which then encodes my files without any problem.

Describe alternatives you've considered
I am using a single CPU transcode at the moment. But in this case, I am not using my server at its full capacity. my CPU (a 5950x) performs better when transcoding 2 movies at a time at least.

Additional context
Here we can see that single 4K encodes, that runs in parallel with 1080p encodes, are finalizing without any issue. It only crashes when two 4K transcodes are run in parallel :

image

here we can see the 5950x, that currently runs a single 4K transcode, is not maxed out :

image

@HaveAGitGat
Copy link
Owner

HaveAGitGat commented Jun 5, 2024

This will be possible with this change:

#748 (comment)

So you could have a node with tag 4k which only does your 4k transcoding. If workers on other nodes have a file which is 4k, they requeue it for that node.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants