Skip to content

Background Jobs

Esty Thomas edited this page Apr 16, 2019 · 28 revisions

Heliotrope uses Resque to process background jobs, such as characterizing files with FITS and creating derivatives (thumbnails, web-friendly video, etc.). In production we recommend resque-pool for managing resque workers. In development, you can use the relevant rake tasks. The job queue is stored in Redis.

How to install Redis

If you are using a Mac, you can use Homebrew to install redis:

brew install redis

If you're using a Vagrant box, redis server is probably already installed and running.
You can check with the following command: redis-cli ping, the server should respond with "pong".

Running background jobs

To start the redis server:

redis-server /usr/local/etc/redis.conf
# (or on ubuntu)
sudo redis-server /etc/redis/redis.conf
# (or on a Mac with Homebrew, to auto-start on login)
brew services start redis

To start workers(s) to run the jobs (one worker per terminal window:

QUEUE=* VERBOSE=1 bundle exec rake resque:work

OR you can use Resque pool to bring up a pool of workers in a daemonized way. The queue and worker count are pulled from config/resque-pool.yml

RUN_AT_EXIT_HOOKS=true TERM_CHILD=1 VERBOSE=1 bundle exec resque-pool --daemon --environment development start

The first two added variables help with clean up when forked workers are killed, the third adds verbosity which can help you find problems in dev.

Viewing background jobs

Resque-web is mounted onto the rails app, but is only available to platform-level admin users (superadmins). To see the status of recent jobs in the resque-web console, start up the server, log in as a platform admin, then navigate your browser to:

http://localhost:3000/resque

Checking Resque Pool Status on U-M deployment

On nectar.umdl.umichi.edu there is a script to check if heliotrope's resque pool is running at:

/hydra-dev/heliotrope-testing/bin/check_resque_pool

Installing FITS

If you haven't yet, you'll need to install FITS or some resque jobs like Characterize will fail. And if one job fails, the whole thing fails.

Download FITS and put it where Heliotrope expects it, /usr/local/bin/fits.sh.

  1. Download fits and unzip it somewhere (I did ~/bin/fits-0.10.1/)
  2. sudo ln -s /path/to/your/fits-0.10.1/fits.sh /usr/local/bin/fits.sh

For new versions of fits (I've tried 1.0.7) you are done. If running /usr/local/bin/fits.sh gives an error, thn you can try changing the line in fits.sh from this:

    . "$(dirname $BASH_SOURCE)/fits-env.sh"

To this:

    . "/path/to/your/fits-env.sh"
  1. Homebrew on Mac is an alternative to the preceding
You can’t perform that action at this time.