Skip to content
Autoscale Kubernetes controllers or Docker Swarm services based on application metrics and more
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin
exe
lib
spec
.dockerignore
.gitignore
.rspec
.travis.yml
Dockerfile
Gemfile
LICENSE.txt
README.md
Rakefile
scaltainer.gemspec

README.md

Build Status Coverage Status Gem Version

Scaltainer

A Ruby gem to monitor Docker Swarm mode services and Kubernetes resources and auto-scale them based on user configuration. It can be used to monitor web services and worker services. The web services type has metrics like response time using New Relic. The worker services type metrics are basically the queue size for each. This gem is inspired by HireFire and was indeed motivated by the migration from Heroku to Docker.

Installation

Add this line to your application's Gemfile:

Install using rubygems:

$ gem install scaltainer

Usage

For Docker swarm:

scaltainer -o swarm

Or simply:

scaltainer

For Kubernetes:

scaltainer -o kubernetes

This will do a one-time check on the running docker service replicas or Kubernetes replication controllers, replica sets, or deployments. Then it sends scaling out/in commands to the cluster as appropriate. Configuration is read from scaltainer.yml by default. If you want to read from another file add -f yourconfig.yml:

scaltainer -f yourconfig.yml

Note that after each run a new file is created (yourconfig.yml.state) which stores the state of the previous run. This is because there are some configuration parameters (like sensitivity) need to remember previous runs. If you want to specify a different location for the state file, add the --state-file parameter. Example:

scaltainer -f /path/to/configuration/file.yml --state-file /path/to/different/state/file.yml

Typically one would want to repeatedly call scaltainer every minute or so. To do this specify the wait time between repetitions using the -w parameter in seconds:

scaltainer -w 60

This will repeatedly call scaltainer every 60 seconds, sleeping in-between.

Configuration

Environment variables

Docker swarm options

  • DOCKER_URL: Should point to the docker engine URL. If not set, it defaults to local unix socket.

Kubernetes options

  • KUBECONFIG: set to Kubernetes config (default: $HOME/.kube/config) if you want to connect to the current configured cluster.

  • KUBERNETES_API_SERVER: overrides option in KUBECONFIG and defaults to https://kubernetes.default:443.

  • KUBERNETES_SKIP_SSL_VERIFY: KUBECONFIG option overrides this, set to any value to skip SSL verification.

  • KUBERNETES_API_ENDPOINT: defaults to /api.

  • KUBERNETES_API_VERSION: overrides option in KUBECONFIG and defaults to v1.

  • KUBERNETES_CONTROLLER_KIND: controller kind to scale, allowed values: deployment (default), replication_controller, or replica_set.

Make sure the KUBERNETES_CONTROLLER_KIND you specify is part of the api specified using KUBERNETES_API_ENDPOINT and KUBERNETES_API_VERSION.

General options

  • HIREFIRE_TOKEN: If your application is configured the hirefire way, you need to set HIREFIRE_TOKEN environment variable before invoking scaltainer. This is used when probing your application endpoint (see below) to get the number of jobs per queue for each worker.

  • NEW_RELIC_API_KEY: New Relic API key. Currently New Relic is used to retrieve average response time metric for web services. More monitoring services can be added in the future.

  • RESPONSE_TIME_WINDOW: Time window in minutes to measure average response time till the moment. For example 3 means measure average response time in the past 3 minutes. Default value is 5.

  • LOG_LEVEL: Accepted values here are: DEBUG, INFO (default), WARN, ERROR, FATAL. Log output goes to stdout.

  • DOCKER_SECRETS_PATH_GLOB: Path glob containing environment files to load. This is useful if running from a docker swarm mode environment where one or more of the above environment variables are set using docker config or docker secret. These files should be in the form VARIABLE=value. A typical value of this variable would be: {/run/secrets/*,/config1,/config2}

Configuration file

The configuration file (determined by -f FILE command line parameter) should be in the following form:

# to get worker metrics
endpoint: https://your-app.com/hirefire/$HIREFIRE_TOKEN/info
# optional docker swarm stack name or kubernetes namespace
namespace: mynamespace
# list of web services to monitor
web_services:
  # each service name should match docker service name
  web:
    # New Relic application id (required)
    newrelic_app_id: <app_id>
    # minimum replicas to maintain (default: 0)
    min: 1
    # maximum replicas to maintain (default: unlimited)
    max: 5
    # maximum response time above which to scale up (required)
    max_response_time: 300
    # minimum response time below which to scale down (required)
    min_response_time: 100
    # replica quantitiy to scale up at a time (default: 1)
    upscale_quantity: 2
    # replica quantitiy to scale down at a time (default: 1)
    downscale_quantity: 1
    # number of breaches to wait for before scaling up (default: 1)
    upscale_sensitivity: 1
    # number of breaches to wait for before scaling down (default: 1)
    downscale_sensitivity: 1
  webapi:
    ...
worker_services:
  worker1:
    min: 1
    max: 10
    # number of jobs each worker replica should process (required)
    # the bigger the ratio, the less number of workers scaled out
    ratio: 3
    upscale_sensitivity: 1
    downscale_sensitivity: 1
  worker2:
    ...

More details about configuration parameters can be found in HireFire docs.

Docker Swarm usage

A service definition for scaltainer is typically something like this:

version: '3.3'
services:
  scaltainer:
    image: rayyanqcri/scaltainer:latest
    command: -f /scaltainer.yml --state-file /tmp/scaltainer-state.yml -w 60
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DOCKER_URL=unix:///var/run/docker.sock
      - DOCKER_SECRETS_PATH_GLOB={/run/secrets/*}
      - RESPONSE_TIME_WINDOW=3
    configs:
      - source: scaltainer
        target: /scaltainer.yml
    secrets:
      - scaltainer
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
configs:
  scaltainer:
    file: scaltainer.yml
secrets:
  scaltainer:
    file: scaltainer.env

Where scaltainer.env is a file containing HireFire and NewRelic secrets:

HIREFIRE_TOKEN=
NEW_RELIC_API_KEY=

And scaltainer.yml is the scaltainer configuration file.

Kubernetes usage

Create a ConfigMap

kubectl create configmap scaltainer --from-file=scaltainer.yaml=/path/to/your/scaltainer.yml

Where /path/to/your/scaltainer.yml is the scaltainer configuration file.

Create a Secret

kubectl create secret generic scaltainer --from-env-file=/path/to/scaltainer.env

Where /path/to/scaltainer.env is a file containing HireFire and NewRelic secrets:

HIREFIRE_TOKEN=
NEW_RELIC_API_KEY=

Create a Deployment:

kubectl apply -f scaltainer-kube.yaml

Where scaltainer-kube.yaml has the following content:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: scaltainer
  name: scaltainer
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: scaltainer
    spec:
      containers:
      - image: rayyanqcri/scaltainer:latest
        name: scaltainer
        args:
          - -o
          - kubernetes
          - -f
          - /etc/config/scaltainer.yaml
          - --state-file
          - /tmp/scaltainer-state.yaml
          - -w
          - "60"
        env:
        - name: KUBERNETES_SKIP_SSL_VERIFY
          value: "yes"
        - name: KUBERNETES_API_ENDPOINT
          value: /apis/extensions
        - name: KUBERNETES_API_VERSION
          value: v1beta1
        - name: KUBERNETES_CONTROLLER_KIND
          value: deployment
        envFrom:
        - secretRef:
            name: scaltainer
        volumeMounts:
        - name: scaltainer-config
          mountPath: "/etc/config"
      volumes:
      - name: scaltainer-config
        configMap:
          name: scaltainer

Development

After checking out the repo, run bin/setup to install dependencies. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/hammady/scaltainer.

Testing

rake

License

The gem is available as open source under the terms of the MIT License.

You can’t perform that action at this time.