A Python Celery application for queuing and processing video encoding jobs
queueFiles.py will scan a directory defined in dirConfig.py and move/queue all video files ready for processing. The destination directories are also defined in dirConfig.py.
videoTasks.py defines the encode task. This takes a source file (including full path) and a destination directory to output the encoded file to. It uses ffmpeg, and creates a mp4 h264 file according to the settings in the file.
Check out this repo to a system with python3. You'll need a RabbitMQ instance somewhere, and whichever machine runs the workers will need access to ffmpeg.
Configure the RabbitMQ settings in celeryconfig.py.
One machine needs to run queueFiles.py to process and queue videos in a given directory.
Start a celery worker either on the same machine, or on 1 or more other machines. You'll need celery installed using pip install celery.
If more than one machine is used, then there must be shared storage between them, e.g by using NFS, and the video folders mounted to identical paths.
Start a worker like so:
cd /path/to/git/checkout
celery worker -A videoTasks -l info -Ofair
The -l and -O arguments are optional, change them if you like.
The worker process should pick up any messages in the queue and begin processing.