Auto scaling

Sneakers provides auto-scaling capabilities that consists of dynamically adding or removing processes (unix processes) from the process supervisor, similar to how it's done with Unicorn.

this feature may be removed if underused to keep codebase simple

Connection Pools

Within our hybrid many-process, many-threads model, we have to be extra sensitive about thread pools.

To maximize performance, Sneakers lets you use several knobs:

Processes,
  Worker threads
     Queue prefetch
        Database/Resource library connectionpools (e.g.
ActiveRecord)
          Database/Resource configured hardlimit - connections

So for example, it wouldn't make sense to set up 8 processes, each with 25 threads (for this, queue-prefetch of 25 is very effective), when the ActiveRecord pool configuration is limited at the default of 5.

Even if we lift the ActiveRecord limit to 25 (this is per-process so we're good). It still wouldn't make sense if your database is configured for a hardlimit of 50 connections.

This is, because at worst case we will have 4(processes) x 25(threads) connections open just for workers.

How to tune resources

So based on the work you're doing and the scale you want to have, let's have a simple imaginative case.

We know that in average each job is I/O intensive, and it requires 1 second to complete.

We want 100 request/sec to happen, we have a 4-core machine. This means we want 4 processes, each having 25 threads. At best, we will have 4x25 threads working at the same time.

This means that per-process we want a 25-worth connectionpool allocated to each process. This will happen automatically with Sneakers.

So all you need to do is configure :threads => 25 within your worker, and pool: 25 within your database.yaml (in this case, Rails + postgres) configuration.

Worker Scaling

Scale up

Update sneakers.conf.rb configuration. For example, if you had 4 workers configured, update to 6:

workers 6

Now let sneakers know about it with the reload signal (SIGUSR2).

$ kill -SIGUSR2 `cat sneakers.pid`

Scale down

Scaling down is composed of configuration reload and graceful restart of workers. After restart, extra workers will simply quit.

Lets scale down from 6 back to 4:

workers 4

Signal reload

$ kill -SIGUSR2 `cat sneakers.pid`

and now signal graceful restart (SIGUSR1)

$ kill -SIGUSR1 `cat sneakers.pid`