Elric: A Simple Distributed Job Scheduler
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
elric
example
README.md
SCHEDULER
settings.py

README.md

Elric

A Simple Distributed Job Scheduler Based on Redis and Apscheduler.

Features:

  • Master-Slave Architecture
  • Distributed Job Based on Redis
  • Support Cron/Data/Interval/Immediate Job

Usage

Setup environment in settings.py:

DISTRIBUTED_LOCK_CONFIG = {
    'server': {
        'host': 'localhost',
        'port': 6379,
        'password': None,
        'db': 1,
    },
    'resource': 'elric_distributed_lock',
    'retry_count': 5,
    'retry_delay': 0.2,
}

JOB_QUEUE_CONFIG = {
    'server': {
        'host': 'localhost',
        'port': 6379,
        'password': None,
        'db': 1,
    },
    'max_length': 100000,
    'buffer_time': 10
}

FILTER_CONFIG = {
    'server': {
        'host': 'localhost',
        'port': 6379,
        'password': None,
        'db': 0,
    }
}

JOB_STORE_CONFIG = {
    'server': {},
    'maximum_records': 3
}

Create a master instance and start:

rq_master = RQMasterExtend()
rq_master.start()

Implement some jobs/functions:

def test_job(language=None):
    print 'my favorite language is {language}'.format(language=language)

Create a worker instace, specify worker's name and listening keys. Submit job and start worker.

# worker will only receive job from listen_keys that have been provided here
rq_worker = RQWorker(name='test', listen_keys=['job1', ])
# submit job to master
rq_worker.submit_job(test_job, 'job1', kwargs={'language': 'python'})
# start worker, then worker will receive and execute job from master by listening job queue on listen keys you provided
rq_worker.start()

Running the example code

This example illustrates how to submit different type of jobs to master.

Step 1. Setup environment in settings.py

Step 2. Start master

cd example
python test_master.py

Step 3. Start master

python test_worker.py

For more information

Documentation described in this blog post.