Skip to content
This repository has been archived by the owner on Mar 14, 2019. It is now read-only.

Set Up

Jump to bottom
David Copeland edited this page Jun 19, 2017 · 7 revisions

Getting Resque Brain up and running requires three steps (and a few optional ones):

  1. Deploy it somewhere
  2. Tell it how to find the Resque instances
  3. Configure HTTP Auth
  4. (optional) Adjust how long a job runs before it's been running "too long"
  5. (optional) set up monitoring tasks

Deploy Resque Brain

Resque Brain is a vanilla Rails app that requires no database. It deploys as-is to Heroku, so it should be friendly to your environment. As there are many possible configurations, I can't help with your specific set up. The only requirements are access to UNIX-style environment variables (and access to the Redis' powering your Resques).

The current repo has a Unicorn configuration, but Unicorn is not required.

Tell Resque Brain How To Find Resque Instances

Current, Best Method

  1. Set the environment variable RESQUE_BRIAN_INSTANCES to DERIVE. This instructs resque-brain to examine the environment and figure out what Resque instances it should monitor.

  2. Set however many environment variables ending in RESQUE_REDIS_URL you'd like to monitor. These should be URLs to a redis the same as you'd give to your application's Resque instance.

    For example, if you set these env vars:

    FOO_BAR_RESQUE_REDIS_URL
BLAH_RESQUE_REDIS_URL
CRUD_RESQUE_REDIS_URL

    Resque-brain will derive that you want to monitor these three resque instances, and name them "foo-bar", "blah", and "crud".

Old Explicit Way

  1. Set the environment variable RESQUE_BRAIN_INSTANCES to a comma-delimited list of Resque names. These names can be anything, but it's recommended that they not have spaces. These names will show up in the UI, so make sure they make sense.

  2. For each Resque you listed, set an environment variable named RESQUENAME_RESQUE_REDIS_URL to the URL to connect to that Resque's Redis. The name from RESQUE_BRAIN_INSTANCES will be upper-cased and dashes will be replaced with underscores, so admin-tool becomes ADMIN_TOOL

    For example:

    Suppose you have two Resques, one for your public website and one for your back-end admin console. Suppose that both Redis instances are on the same machine (10.0.1.1), with the public website's on port 6379 and the admin console's on port 6345. Let's also suppose that the admin console is using database 2 instead of the default of database 0.

    RESQUE_BRAIN_INSTANCES=admin-tool,www
ADMIN_TOOL_RESQUE_REDIS_URL=redis://10.0.1.1:6379
WWW_RESQUE_REDIS_URL=redis://10.0.1.1:6345/2

    You'll need to restart the app if you change these, as they are read on startup only.

Using Heroku's Add-on Sharing

If you are on Heroku, a way to manage this is to use the ability to share add-ons between apps.

Suppose you have two apps on heroku, public-site and admin, each with their own RedisCloud add-on:

> heroku addons -a public-site
=== Resources for public-site
Plan                         Name                   Price
---------------------------  ---------------------  -------------
rediscloud:2500              looking-quickly-1234   $252.00/month

=== Attachments for spectre-production
Name                       Add-on                 Billing App
-------------------------  ---------------------  ----------------------------
REDISCLOUD                 looking-quickly-1234   public-site
> heroku addons -a admin
=== Resources for admin
Plan                         Name                   Price
---------------------------  ---------------------  -------------
rediscloud:2500              seeing-slowly-2345     $252.00/month

=== Attachments for spectre-production
Name                       Add-on                 Billing App
-------------------------  ---------------------  ----------------------------
REDISCLOUD                 seeing-slowly-2345     admin

Suppose that your Resque Brain is deployed as the app my-resque-brain. You can then attach your two RedisCloud addons from public-site and admin to my-resque-brain like so:

> heroku addons:attach --as PUBLIC_SITE_RESQUE_REDIS looking-quickly-1234 --app my-resque-brain
> heroku addons:attach --as ADMIN_RESQUE_REDIS seeing-slowly-2345 --app my-resque-brain

This will attach them in Heroku's infrastructure and also set PUBLIC_SITE_RESQUE_REDIS_URL and ADMIN_RESQUE_REDIS_URL in my-resque-brain's environment (note that the arguments to --as do not have the _URL suffix—Heroku adds that in the environment).

You can now configure your Resque Brain like so:

> heroku config:set RESQUE_BRAIN_INSTANCES=public-site,admin --app my-resque-brain

Configure HTTP Auth

Currently, Resque Brain uses HTTP Auth to restrict access. Since Resque Brain exposes destructive actions (and possibly proprietary data via your job payloads), you should restrict access to it.

Out of the box, it's configured to use HTTP Auth. The username and password are configured via two environment variables:

  • HTTP_AUTH_USERNAME - username
  • HTTP_AUTH_PASSWORD - password

I realize this kinda sucks, but was the simplest thing to do and should at least work for anyone.

Adjust how long a job runs before it's been running "too long"

By default, a job running for more than an hour is considered to be running for "too long". While Resque Brain might someday allow configuring this per resque or even per job, right now it's global to all Resques.

You can change this by setting the environment variable RESQUE_BRAIN_STALE_WORKER_SECONDS:

RESQUE_BRAIN_STALE_WORKER_SECONDS=7200 # 2 hours

Caching the UI

If you are monitoring many Resque instances or you tend to have a of jobs in flight, the UI will start to get slow. Resque Brain includes a basic caching setup using Memcached. By default, it's not used, but to use it:

  1. Configure and deploy a Memcached instance.
  2. Set MEMCACHED_SERVERS to the url to your memcached. If you have a user/pass, set those in MEMCACHED_USERNAME and MEMCACHED_PASSWORD, respectively
  3. Set RESQUE_BRAIN_CACHE_RESQUE_CALLS to the value true in your deployment environment

Set Up Monitoring Tasks

Currently, Resque Brain can be monitored by running rake tasks that log information that can be consumed by Librato. You could certainly consume these messages yourself, or write custom notifiers (see Monitoring and Alerting).

The tasks are:

  • rake monitor:failed - Check the number of failed jobs and stat the results to the log in a way Librato can understand
  • rake monitor:queue_sizes - Stat the sizes of all queues to the log in a way Librato can understand
  • rake monitor:stale_workers - Check the number of stale workers and stat the results to the log in a way Librato can understand

Again, arranging for these tasks to run depends on your environment. I would be careful running them as frequently as once per minute as it's possible they might take longer than that to run, especially if you have an active Resque. We run these every 10 minutes and that works well.

Clone this wiki locally