Copy file to temp area before transcoding

[gjannece]

My library is running off of an NFS share and it would be better for Tdarr to copy the file over to
a temp directory before attempting to transcode it.

This would increase the speed of the transcoding ass my box with Tdarr is running off of an SSD
the NAS is much slower.

[HaveAGitGat]

Sorry there isn't a setting to make Tdarr do this - there's no need to as it would be slower overall. Processing is bottlenecked by the CPU, not disk speed. For example:

Tdarr's current workflow:

1. Transcode 4GB H264 file into 2GB H265 file, outputting into transcode cache (60 mins)
2. Copy new file back into library (1 min @ 33MB/s)
   Total = 61 mins

Your suggested workflow:

1. Copy 4GB file into transcode cache (2 min @ 33MB/s)
2. Transcode 4GB H264 file into 2GB H265 file, outputting into transcode cache (60 mins - no difference)
3. Copy new file back into library (1 min @ 33MB/s)
   Total = 63 mins

[gjannece]

I will disagree as in my setup it makes a huge difference. Are you factoring in people who are using NFS or SMB? Are you factoring in if people have it setup Tdarr running on multiple computers to go through their entire library? I would agree with you if I was just using one Tdarr and had a dedicated NAS for just Tdarr, but I use my NAS for other things as well and having open streams eats bandwidth that could be used for other things requiring the NAS.

Just a thought, as I'm sure it is an easy add to Tdarr and could help many users.

[sienar1]

@gjannece as @HaveAGitGat pointed out, video transcoding will certainly not be network bottlenecked. Even running many simultaneous transcodes across multiple servers, it still shouldn't be network bottlenecked. If you are seeing tdarr being able to saturate your network or NAS for long periods of time, then something else needs to be optimized. Perhaps you are using multiple plugins and could look at making a custom plugin that combines steps instead of processing a file multiple times.

Even if you're doing simple steps in tdarr like just removing subtitle streams without actually transcoding video, then you are effectively already doing what you ask. As the streams being copied into the new file (in the transcoding temp folder) are going to happen just as fast as a preparatory file copy. And in the case of simple work like that, if you do a preparatory copy you then still have to make another copy on the tdarr server as it does the work.

[HaveAGitGat]

@gjannece Tdarr v2 is far more efficient as it does not copy the files back and forth between plugin steps. So the first step will transcode from library to cache, then all the other plugin steps will be done in the cache before copying back to the library.

I don't think your method is optimal as if there's an error (corrupt file, invalid transcode arguments etc), it would be a waste of a transfer. Lets take your suggestion:

• Worker picks up an item, 10GB file copied over, transcoding begins, file is corrupt => waste of 10GB transfer

With the current impelentation:

• Worker picks up an item, transcoding begins, 10MB or so of data streamed, file is corrupt => waste of only 10MB transfer.

Now lets look at the new Tdarr_Node system with your suggestion and it's even worse:

• Node1 picks up item, 10GB file copied over, transcoding begins ,invalid arguments, Node1 then moves on to another task => waste of 10GB transfer, arguments fixed, Node2 picks up item, 10GB file copied over, transcoding, file is corrupt => waste of 20GB transfer total

With the current implementation:

• Node1 picks up item, transcoding begins, 10MB or so of data streamed, invalid arguments, Node1 then moves on to another task => waste of 10MB, arguments fixed, Node2 picks up item, transcoding begins, 10MB or so of data streamed, file is corrupt => waste of 20MB transfer total

20GB vs 20MB of wasted transfer is obviously huge so the current implementation is more suitable.

[HaveAGitGat]

Info on v2 here:
http://tdarr.io/docs/

[HaveAGitGat]

2.27.01 just released adds a new node type 'unmapped' nodes, more info here: https://docs.tdarr.io/docs/nodes/nodes#unmapped

[ochompsky]

of course it's paywalled