Redis Based Distributed Task Locking
JavaScript Lua
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
lib
scripts
test
.gitignore
.travis.yml
package.json
readme.md

readme.md

Build Status

Tusker - Redis Based Distributed Task Locking

Why? Let's look at following scenario

  • We are transcoding ogg streams into mp3 realtime
  • And oggs are coming as chunks (<15 secs)
  • We have deployed several servers and routing is random and stateless

Lets Cordinate this with Tusker

When we are encoding chunks

var tusker = require('tusker');
var taskManager = tusker.initialize();

//when we are begin to encode a chunk
var lockname = "" + Math.random();
taskManager.lock('the-task', lockname);

//after we do the encoding
taskManager.unlock('the-task', lockname);

At the end of the ogg stream

var tusker = require('tusker');
var taskManager = tusker.initialize();

var info = { some_data: 10 };
var options = {}
taskManager.close('the-task', info, options);

Available options

attempts
  • no of attempts before before mark the task as failed
  • new attempt can be occurred if task completed with an error or timeout occurred
timeout
  • timeout for the task completion in millis
  • if this value exceeded new attempt will be occurred

Receive notification after the all related chunks for a stream has encoded

var tusker = require('tusker');
var taskManager = tusker.initialize();

taskManager.fetchReleased(function(err, taskName, info, completed) {

    //merge chunks and notify back
    var err = null; //if there is an error assign error object 
    completed(err);
});

Invoking a timeout for locks

We cannot assure every lock we create will be unlocked. If the server died before unlock, we have a lock which will never be closed. So we need a timeout machanism.

var tusker = require('tusker
var taskManager = tusker.initialize();

//one minute timeout
taskManager.timeoutLocks(60 * 1000, function(err) {
    
});

NOTE: We've to invoke this method via a cron like tool

Invoking a timeout for task processing

  • We need to timeout long running tasks, and allow another attempt to process it
  • default timeout is 30 seconds otherwise if it is not defined when task closing via .close()
var tusker = require('tusker');
var taskManager = tusker.initialize();

//one minute timeout
taskManager.timeoutProcessing(function(err) {
    
});

NOTE: We've to invoke this method via a cron like tool

Statistics

var tusker = require('tusker');
var taskManager = tusker.initialize();

//one minute timeout
taskManager.stats(function(err, stats) {
    

});

Stats object is formatted as shown below:

{ completed: 101, released: 1, locked: 2, failed: 2, processing: 2 }