Managing Cron Jobs with Moonshine

technicalpickles edited this page Jun 13, 2012 · 8 revisions

Managing Cron Jobs with Moonshine

The cron Method's Arguments

The cron method has a pretty simple api:

cron('name of the job', options = {})

The name of the job is required. The name will appear as a comment on the line above the job on the server.

The available options are:

  • :command (required) - a string that contains the command that cron should execute
  • :user (required) - the user account that the cron job will run under
  • :ensure - defaults to :present. Set to :absent to ensure that a job is removed from the server.
  • times to run the job (values are set to '*' by default so not setting the :minute will default to running the job every minute):
    • :minute
    • :hour
    • :weekday
    • :month
    • :monthday

Examples

Simple Example

Let's take a look at the example that is documented inline from Moonshine's default application_manifest:

some_rake_task = "/usr/bin/rake -f #{configuration[:deploy_to]}/current/Rakefile custom:task RAILS_ENV=#{ENV['RAILS_ENV']}"
cron 'custom:task', :command => some_rake_task, :user => configuration[:user], :minute => 0, :hour => 0

On the sever, the rails user would now have the following if you looked at the crontab:

# custom:task
0 0 * * * /usr/bin/rake -f /path/to//current/Rakefile custom:task RAILS_ENV=#{ENV['RAILS_ENV']}

This will run the custom:task with rake at midnight every night passing the appropriate Rails env.

Logging

Logging is good to add to any cron task so that you can debug it later. Let's add logging to the previous example and have it run every 8 hours.

some_rake_task = "(date && /usr/bin/rake -f #{configuration[:deploy_to]}/current/Rakefile custom:task RAILS_ENV=#{ENV['RAILS_ENV']}) >> #{configuration[:deploy_to]}/current/log/cron.custom_task.log 2>&1"
cron 'custom:task', :command => some_rake_task, :user => configuration[:user], :minute => 0, :hour => '*/8'

Now we have task running every 8 hours with a log that will contain the timestamp of the run plus any output from that task will be captured in the log. Things have gotten a little hard to read, set's clean that up a little bit.

# set a few variables for reuse on other cron jobs and/or to make the command more readable
current_path = "#{configuration[:deploy_to]}/current"
run_with_rake = "/usr/bin/rake -f #{current_path}/Rakefile"

some_rake_task =  "(date && #{run_with_rake} custom:task RAILS_ENV=#{rails_env}) >> #{current_path}/log/cron.custom_task.log 2>&1"
cron 'custom:task', :command => some_rake_task, :user => configuration[:user], :minute => 0, :hour => '*/8'

rails_env is a method provided in Moonshine that returns ENV["RAILS_ENV"] || 'production'.

Long running jobs

If you have a job that takes a while to run and may not actually finish before cron runs it again, you should use flock. Flock will create a temporary file and place a lock on it to prevent other scripts that check that lock from running at the same time.

current_path = "#{configuration[:deploy_to]}/current"
run_with_rake = "/usr/bin/rake -f #{current_path}/Rakefile"

# `flock -w 0` tells flock to exit in 0 seconds if a lock cannot be obtained (ie another script is using it)
some_rake_task = "flock -w 0 /tmp/custom_task.lock #{run_with_rake} custom:task RAILS_ENV=#{ENV['RAILS_ENV']}"
cron 'custom:task', :command => some_rake_task, :user => configuration[:user], :minute => 0, :hour => 0

Lengthy command arguments

Sometimes your cron job command argument can be pretty lengthy with all the file paths and arguments. Here is one way to clean it up a bit for easier reading/debugging.

current_path = "#{configuration[:deploy_to]}/current"
run_with_rake = "/usr/bin/rake -f #{current_path}/Rakefile"

some_rake_task = [
  "(date &&",
  "flock -w 0 /tmp/custom_task.lock",
  "#{run_with_rake} custom:task RAILS_ENV=#{ENV['RAILS_ENV']} &&",
  "touch #{current_path}/tmp/restart.txt)",
  ">> #{current_path}/log/cron.custom_task.log 2>&1"
].join(' ')
cron 'custom:task', :command => some_rake_task, :user => configuration[:user], :minute => 0, :hour => 0

Removing/Disabling Jobs

If you remove a job from your manifest, Moonshine will not remove it from the server by default. Moonshine only checks for existence of the jobs you've defined. To remove a cron job from the server you can set :ensure => :absent for that job and Moonshine will remove it on the next deploy. You can then remove the job from the manifest entirely if desired or keep it in the manifest in case you want to add that job back later. To remove our previous example you would do the following:

cron 'custom:task', :command => some_rake_task, :user => configuration[:user], :minute => 0, :hour => 0, :ensure => :absent

Troubleshooting

Duplicate Cron Jobs

This has been observed particularly with moonshine_scout's cron job:

# Puppet Name: cleanup_lynx_tempfiles 
0 0 * * * find /tmp/ -name 'lynx*' -type d -delete 
# Puppet Name: cleanup_lynx_tempfiles 
0 0 * * * find /tmp/ -name 'lynx*' -type d -delete 
# Puppet Name: cleanup_lynx_tempfiles 
0 0 * * * find /tmp/ -name 'lynx*' -type d -delete 
# Puppet Name: cleanup_lynx_tempfiles 
0 0 * * * find /tmp/ -name 'lynx*' -type d -delete 
# Puppet Name: cleanup_lynx_tempfiles 
0 0 * * * find /tmp/ -name 'lynx*' -type d -delete 
# Puppet Name: cleanup_lynx_tempfiles 
0 0 * * * find /tmp/ -name 'lynx*' -type d -delete

This has been tracked back to a bug in puppet's cron resource used by shadow_puppet < 0.5.0. The best way to fix is to:

  • manually use crontab -e (or sudo crontab -e for root) to remove the duplicates
  • update moonshine's dependencies:
    • cap <stage> ruby:install_moonshine_deps