Home

dustinrue edited this page Jan 28, 2011 · 15 revisions
Clone this wiki locally

Welcome to the Dencoder wiki!

0.3.0 released! Read about the changes here Download files below.

What Dencoder Is

Dencoder is a shared job queue that any number of clients can work from, a sort of distributed encoder. Clients take jobs from the queue and then start HandBrakeCLI with the parameters given. There is no limit to the number of clients that can be used. If you have five clients and five movies to encode then the five computers will work in parallel rather than one at a time. Given equally powerful clients the encoding work will be done in about 1/5th the time of using just one.

What Dencoder Is Not

Dencoder isn't able to take a single job and split it up between a number of computers. Dencoder doesn't aim to be that. It aims to provide a simple way to have a number of clients working from a shared job queue.

Installing

Dencoder is currently focused on Mac OS X but porting the clients to Linux and Windows is possible. While Erlang and RabbitMQ are stable software, the scripts I’m using to tie the whole operation are not. This is alpha software. Please review the Known Issues section below.

To use dencoder you need:

  1. a home network, preferably wired
  2. Intel Macs running OS X 10.6, 10.5 might work but I haven't tested, and at least one (though it’s pointless overhead with just one)
  3. a shared file system using apple file protocol or SMB/CIFS
You will also need the following software:
  1. Erlang for OS X – Erlang Installer for OS X
  2. RabbitMQ for OS X – RabbitMQ Installer OS X
  3. Dencoder Client for OS X – Dencoder Client for OS X
  4. Dencoder Client for Ubuntu 10.04 - Dencoder Client for Ubuntu 10.04
Erlang and RabbitMQ have been configured to fit the needs of this project, they are otherwise unchanged and are available from (erlang.org) and (rabbitmq.com) respectively.

Setup

To cut down on the amount of configuration needed create a folder somewhere on the master server and name it Encode. Inside of it create two folders, one called Pending and another called Finished. Share this folder using OS X’s File Sharing option located in System Preferences.

On each client computer connect to this new file share and verify that it is now available via Finder.

On the master, install the Erlang, RabbitMQ and Dencoder Client packages. When finished, reboot the system. This will ensure the RabbitMQ service starts properly.

On each client computer, install the Dencoder Client package. This will install the dencoder scripts, OS X services and required python modules. At the time of this writing these modules are installed: pika, simplejson and pybonjour.

Configuration

At this time, the dencoder client requires some configuration in order to find the master server. A future release will bonjour to find the master server.

To configure dencoder start terminal using spotlight to search for terminal. Once loaded, issue setDencoderHost.py hostname where hostname is the name or IP address of the master server. Repeat this step on all clients and the master. If you don’t know the IP address of your master server start terminal by searching for terminal in spotlight. Enter ifconfig and record the IP address for en0 if you use wired networking or en1 if you use wireless.

On the master only, issue setDencoderBasePath.py path to encode folder where path to encode folder is the path for the encode folder. This step is required because the master doesn’t use file sharing to find the files, they are local. The easiest thing to do is to type setDencoderBasePath.py and drag and drop the Encode folder to the terminal and press enter.

Using Dencoder

At this point Dencoder should ready for use. To encode a file drop the file into the Encode/Pending folder, right click it and pick the Encode using preset preset where preset is the preset you want to encode with. At this time the valid presets are AppleTV, AppleTV 2 and HighProfile. I’ll provide directions on how to customize this in the future. When an encode is finished it’ll be placed in the Encode/Finished directory.

If you like to hack on things

The config files for dencoder are located at /usr/local/etc/dencoder. If you haven’t used setDencoderHost.py or setDencoderBasePath.py then it is documented. The scripts themselves are at /usr/local/bin/dencoder.py and /usr/local/bin/send2dencoder.py. The launchd files are in /Library/LaunchDaemons/com.dustinrue*

It is possible to take the dencoder.py file and config file and move it to a Linux system with virtually no changes. You must manually install pika, simplejson and pybonjour using easy_install

The OS X services are created using Automator. Open /System/Library/Services/Encode* to edit the service.

On the clients you can monitor HandBrake’s progress by issuing tail -f /tmp/hb.log in a terminal window. The error log is at /tmp/hb.err

You can download the source here

Known Issues:

  • Clients aren’t aware if the master goes away
  • There is no way to view what is in the queue
  • Clients immediately dump files into the finished directory, even when they’re not yet finished. The file is finished when the date stamp stops updating :)
  • Finished encodes are not removed from Pending
  • The installers always overwrite any config file modifications you’ve made
  • When in doubt, restart the master and then each of the clients, ensure the shared file system is available on each client