Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Simple loop job runner
Ruby Shell
Branch: master
Failed to load latest commit information.
bin Release 0.2.0
lib Release 0.2.6
.gitignore README + License
Gemfile First alpha version
Gemfile.lock Pushed 0.2.4 release
README.md Updated README
lobster.gemspec Github as homepage

README.md

Lobster

Simple job service that was originally written to run hadoop jobs, but can be used for any batch-processing scripts that you want to run.

Lobster runs as a service (daemon) and regularly checks a schedule in $LOBSTER_DIR/config/schedule.rb. The goal is to run the scheduled jobs in parallel but a job can't run twice at the same time (no overlap). The only configuration needed is the delay between 2 job runs.

Get Started

Install

gem install lobster

Create a config/schedule.rb file

Run lobsterize in the directory where you need to setup and run your jobs (defined by the LOBSTER_DIR variable)

lobsterize

Then add a job in config/schedule.rb

# config/schedule.rb
job "my-job" do
  cmd "runthis && runthat >> log/here.log"
  delay 5 # minutes
end

Additional settings:

  • directory this is set to LOBSTER_DIR by default
  • user the Unix user that will run the job, lobster must have full passwordless sudo access to this user
  • max_duration for monitoring, will log an error if the job has not been successful for max_duration + delay minutes

Run Lobster

Two environment variables are used (with defaults):

  • LOBSTER_DIR is the directory used for all job commands, and where the configuration/schedule files are (default: current directory)
  • LOBSTER_ENV is a variable used to handle different environments (default: development)

  • lobster run will run in the console

  • or lobster start as a deamon

Configuration

The lobster configuration file should be located in $LOBSTER_DIR/config/lobster.yml and has the following layout

# default values

# monitor the daemon and restart it if it stops
monitor: true
# log directory, absolute or relative to the LOBSTER_DIR
log_dir: log
# pids directory, absolute or relative to the LOBSTER_DIR
pid_dir: /var/run/lobster/
# schedule file, absolute or relative to the LOBSTER_DIR
schedule_file: config/schedule.rb

# environment overrides, the env variable LOBSTER_ENV has 
# to be set (default: development)
my_test_env:
  monitor: false
  log_dir: test_los
my_prod_env:
  pid_dir: /var/run/

Any log in stderr/stdout will be written in the log directory as lobster.output, the actual lobster log is in lobster.log

Capistrano Deployment

It is very easy to have a lobster project with a proper config/lobster.yml to handle different environments such as development, testing and production. You can simply have a schedule.rbfile per environment.

Then you can use capistrano to deploy the new jobs and schedule to your servers (the schedule is automatically reloaded).

Something went wrong with that request. Please try again.