An auto-scaling engine for Heroku web dynos using pluggable API connections. The first API is Google Analytics Live, which uses the number of active users on the site to decide how many dynos to run. There is also a New Relic plugin that uses the requests per minute.
Each plugin generates a 'value' at some set interval (e.g. polls the GA API
every 10s) and provides a conversion from the value ('active users on the site')
to estimated dynos required. (e.g. 100 active users per dyno). By default this
has some built in hysteresis, i.e. the plugins are conservative about
decreasing their estimate and report the highest estimate of the past N
seconds.
Dynosaur takes the maximum of all plugin estimates, applies a global min/max constraint and scales your Heroku app to match.
For example, see this plot of a dummy 'Sine wave' plugin (purple line) and the resulting estimated dynos (orange line).
The plugin estimates 'units' per dyno, and Dynosaur is configured with a minimum of 2 dynos. You can see that the estimated dynos rapidly tracks the upward slopes, but trails the downward slope by about 30s.
See below about how to deploy to Heroku.
$ gem install dynosaur
Create a YAML config file (see config.sample.yaml
and info below) and run with:
$ dynosaur config.yaml
The 'scaler' section of the config file configures the main parameters of the autoscaler.
heroku_app_name
(string): The name of the heroku app you want to autoscaleheroku_api_key
(string): Heroku API key can be retrieved from the Heroku account settings page.dry_run
(boolean): If enabled, the scaler does not actually change anything in Heroku, just simulates the values it would choose.interval
(int): The autoscaler sleeps for this many seconds before checking for activity. Note that each plugin is configured with an API polling interval too, so this does not increase the frequency of API polling.
Dynosaur will run indefinitely, with info output to stdout at intervals.
If multiple plugins are configured, the scaler will use the maximum of all plugins results (i.e. if your New Relic plugin returns 3 dynos, and your GA plugin returns 5, you should scale to 5 dynos.
Dynosaur can optionally use report statistics either to the command line or to Librato.
The following stats are sent every interval seconds.
- combined estimate of dynos required (includes min/max constraints)
- Actual number we are using (includes hysteresis)
For each plugin we send
- value (e.g. 'active users')
- plugin dyno estimate
Dynosaur can scale different layers of your app, each layer is documented separately:
- Web dynos (based on Google Analytics Live API and/or NewRelic RPM data)
- RedisCloud addon (based on number of connections and/or memory usage)
- Papertrail addon (based on amount of log data generated per month)
All input plugins have the following config values
name
: unique identifier for the plugin instance. Freeform.type
: the name of the plugin class e.g.Dynosaur::Inputs::GoogleAnalyticsPlugin
interval
(default 60s) : how often to poll the respective API. (i.e. the retrieved value is cached for 'interval' seconds.)hysteresis_period
(default 300s) : the current estimate is based on the maximum value observed within this interval. i.e. If the maximum value of active users observed in the last 5 minutes is 127, we will base our estimate on 127 active users.
You can sign up for a free account
type: Dynosaur::Stats::Librato
api_email
(string): Email of Librato account user.api_key
(string): API key.
type: Dynosaur::Stats::Console
We've added pluggable error handling, with three implementations available so far
Logs errors to the console. No config.
Send exceptions to Bugsnag. Configured like this in config.yaml:
error_handlers:
-
type: Dynosaur::ErrorHandler::Bugsnag
api_key: <%= ENV['BUGSNAG_API_KEY'] %>
Configured like this in config.yaml
error_handlers:
-
type: Dynosaur::ErrorHandler::Ses
from: you@example.com
to: you@example.com
aws_access_key_id: <%= ENV['AWS_ACCESS_KEY_ID'] %>
aws_secret_access_key: <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
Warning: this has the tendency to send a lot of email due to minor blips in API reliability.
We have included a pinger for our "dead man's switch" server Cronut. It will ping every minute or so, and we get an alert when Cronut hasn't received a ping for 10 minutes.
At Harry's, we run Dynosaur as a Heroku app. Here's the steps to get it deployed:
- Create heroku app.
- Clone heroku git repository.
- Create a Gemfile and add dynosaur as a gem (see
doc/Gemfile
). - Write your config file (see
doc/config.sample.yaml
) - We use environment variables for secrets, so set them locally and in Heroku
- Check that it works
bundle install
bundle exec dynosaur config.yaml
- Copy the Procfile (from
doc/Procfile
) to the root of your project. - Commit it all and deploy to heroku
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request
You'll need to implement the following methods:
retrieve()
: connect to an API (or wherever) and retrieve a new value. This is
wrapped in a caching layer by the plugin base class.
value_to_dynos()
: calculate the estimated number of dynos based on the recent
maximum value returned by retrieve()
(e.g. we use users_per_dyno
in the GA
plugin). For more fine-grained control, you can override estimate_dynos()
instead.
initialize(config)
: You can pull any configuration you require from the config hash passed in.
See the Google Analytics plugin or the toy Random plugin for an example.
- New Relic plugin by Jan Lindblom