Information Request: Batch Processing

[ArcAIN6]

This is simply a request for more detailed information of transcoding / batch processing workflow.

Server Spec's
CentOS 6.4 64-bit
16gb DDR2 ECC Ram
1tb x4 Raid mirror system / OS storage
5tb sas storage
2x Intel Xeon quad-core @3.0ghz CPU's

---

Request:
I'm looking to lower the amount of system resources being consumed during batch queue processing (transcoding)

Scenario:
Add 10 video files in mp4 format to dropbox
Transcoding / batch processor innitiates
System resources allocated to ffmpeg skyrocket. almost all free CPU time and memory become allocated to ffmpeg task(s)

This is a severe issue as it causes the system to become so congested that the webserver, webapps, ssh, rsync, heartbeat, and many other mission critical applications become non-responsive.

While not transcoding (when batch processor queue is empty) The system behaves normally. Content delivery resumes to normal without any issue.

---

i guess what i'de like to know, is if there's a way to limit the transcoding processes to utelize a certain percentage of free resources.. i.e. if 8gb of the 16gb is free && if 70% of cpu cores 1 - 8 are free, utelize 4gb of ram, and 50% of free CPU resources to transcode.. thus leaving enough resources available for critical system services.

---

[ArcAIN6]

after a bit of searching on my own, it seems you aren't able to limit memory usage of ffmpeg.. however, you can make ffmpeg use less CPU (the bigger issue here)

from what i read, you can lower the thread count of ffmpeg by adding -theads x .. where x represents the number of cores you wish to utelize.

example:

Process ffmpeg = new Process();
ffmpeg.StartInfo.UseShellExecute = false;
ffmpeg.StartInfo.FileName = "..\ffmpeg.exe";
ffmpeg.StartInfo.Arguments = "-threads 4";  // <=== Add this line
ffmpeg.StartInfo.CreateNoWindow = true;
ffmpeg.Start();

[anatolkaltura]

Right, setting '-threads 1' will limit the number of threads per each ffmpeg processes.

But u still will have several conversion jobs running parallelly.
You can control the number of concurrent conversions jobs via 'maxInstances' param in batch.ini file (or enabledWorkers.KAsyncConvert param in more recent Kaltura versions).

I would u suggest to take another route - try to set the conversion jobs to lower Linux priority. This way the critical jobs will get the resources while they will be needed, and the conversion jobs will be able to exploit the full CPU resources when the critical jobs are idle. I don't know whether this is possible set up, but it is worthwhile to inquire.

[ArcAIN6]

thank you for the info :)