Skip to content

Latest commit

 

History

History
310 lines (243 loc) · 13.5 KB

README.md

File metadata and controls

310 lines (243 loc) · 13.5 KB
Raw

docker-gen

latest v0.4.1 Build Status License MIT

docker-gen is a file generator that renders templates using docker container meta-data.

It can be used to generate various kinds of files for:

  • Centralized logging - fluentd, logstash or other centralized logging tools that tail the containers JSON log file or files within the container.
  • Log Rotation - logrotate files to rotate container JSON log files
  • Reverse Proxy Configs - nginx, haproxy, etc. reverse proxy configs to route requests from the host to containers
  • Service Discovery - Scripts (python, bash, etc..) to register containers within etcd, hipache, etc..

===

Installation

There are three common ways to run docker-gen:

  • on the host
  • bundled in a container with another application
  • separate standalone containers

Host Install

Linux/OSX binaries for release 0.4.1

Download the version you need, untar, and install to your PATH.

$ wget https://github.com/jwilder/docker-gen/releases/download/0.4.1/docker-gen-linux-amd64-0.4.1.tar.gz
$ tar xvzf docker-gen-linux-amd64-0.4.1.tar.gz
$ ./docker-gen

Bundled Container Install

Docker-gen can be bundled inside of a container along-side applications.

jwilder/nginx-proxy trusted build is an example of running docker-gen within a container along-side nginx. jwilder/docker-register is an example of running docker-gen within a container to do service registration with etcd.

Separate Container Install

It can also be run as two separate containers using the jwilder/docker-gen image virtually any other image.

This is how you could run the official nginx image and have dockgen-gen generate a reverse proxy config in the same way that nginx-proxy works. You may want to do this to prevent having the docker socket bound to an publicly exposed container service.

Start nginx with a shared volume:

$ docker run -d -p 80:80 --name nginx -v /tmp/nginx:/etc/nginx/conf.d -t nginx

Fetch the template and start the docker-gen container with the shared volume:

$ mkdir -p /tmp/templates && cd /tmp/templates
$ curl -o nginx.tmpl https://raw.githubusercontent.com/jwilder/docker-gen/master/templates/nginx.tmpl
$ docker run -d --name nginx-gen --volumes-from nginx \
   -v /var/run/docker.sock:/tmp/docker.sock:ro \
   -v /tmp/templates:/etc/docker-gen/templates \
   -t jwilder/docker-gen -notify-sighup nginx -watch -only-exposed /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf

===

Usage

$ docker-gen
Usage: docker-gen [-config file] [-watch=false] [-notify="restart xyz"] [-notify-sighup="nginx-proxy"] [-interval=0] [-endpoint tcp|unix://..] [-tlsverify] [-tlscert file] [-tlskey file] [-tlscacert file] [-keep-blank-lines] <template> [<dest>]

Options:

  -config="": Use the specified config file instead of command-line options.  Multiple templates can be defined and
              they will be executed in the order that they appear in the config file.
  -endpoint="": docker api endpoint [tcp|unix://..]. This can also be set w/ a `DOCKER_HOST` environment.
  -interval=0:run notify command interval (s). Useful for service registration use cases.
  -notify="": run command after template is regenerated ["restart xyz"]. Useful for restarting nginx,
              reloading haproxy, etc..
  -notify-sighup="": send HUP signal to container.  Equivalent to `docker kill -s HUP container-ID`
  -only-exposed=false: only include containers with exposed ports
  -only-published=false: only include containers with published ports (implies -only-exposed)
  -keep-blank-lines=false: keep blank lines in the output file
  -tlscacert="": path to TLS CA certificate file
  -tlscert="": path to TLS client certificate file
  -tlskey="": path to TLS client key file
  -tlsverify=false: verify docker daemon's TLS certicate
  -version=false: show version
  -watch=false: run continuously and monitors docker container events.  When containers are started
                or stopped, the template is regenerated.

If no <dest> file is specified, the output is sent to stdout. Mainly useful for debugging.

===

Templating

The templates used by docker-gen are written using the Go text/template language. In addition to the built-in functions supplied by Go, docker-gen provides a number of additional functions to make it simpler (or possible) to generate your desired output.

Emit Structure

Within the templates, the object emitted by docker-gen will be a structure consisting of following Go structs:

type RuntimeContainer struct {
    ID        string
    Addresses []Address
    Gateway   string
    Name      string
    Hostname  string
    Image     DockerImage
    Env       map[string]string
    Volumes   map[string]Volume
    Node      SwarmNode
    Labels    map[string]string
    IP        string
}

type Address struct {
    IP       string
    Port     string
    HostPort string
    Proto    string
    HostIP   string
}

type DockerImage struct {
    Registry   string
    Repository string
    Tag        string
}

type Volume struct {
    Path      string
    HostPath  string
    ReadWrite bool
}

type SwarmNode struct {
    ID      string
    Name    string
    Address Address
}

For example, this is a JSON version of an emitted RuntimeContainer struct:

{
   "ID":"71e9768075836eb38557adcfc71a207386a0c597dbeda240cf905df79b18cebf",
   "Addresses":[
      {
         "IP":"172.17.0.4",
         "Port":"22",
         "Proto":"tcp",
         "HostIP":"192.168.10.24",
         "HostPort":"2222"
      }
   ],
   "Gateway":"172.17.42.1",
   "Node": {
       "ID":"I2VY:P7PF:TZD5:PGWB:QTI7:QDSP:C5UD:DYKR:XKKK:TRG2:M2BL:DFUN",
       "Name":"docker-test",
       "Address": {
           "IP":"192.168.10.24"
       }
   },
   "Labels": {
       "operatingsystem":"Ubuntu 14.04.2 LTS",
       "storagedriver":"devicemapper",
       "anything_foo":"something_bar"
   },
   "IP":"172.17.0.4",
   "Name":"docker_register",
   "Hostname":"71e976807583",
   "Image":{
      "Registry":"jwilder",
      "Repository":"docker-register"
   },
   "Env":{
      "ETCD_HOST":"172.17.42.1:4001",
      "PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "DOCKER_HOST":"unix:///var/run/docker.sock",
      "HOST_IP":"172.17.42.1"
   },
   "Volumes":{
      "/mnt":{
         "Path":"/mnt",
         "HostPath":"/Users/joebob/tmp",
         "ReadWrite":true
      }
   }
}

Functions

  • closest $array $value: Returns the longest matching substring in $array that matches $value
  • coalesce ...: Returns the first non-nil argument.
  • contains $map $key: Returns true if $map contains $key. Takes maps from string to string.
  • dict $key $value ...: Creates a map from a list of pairs. Each $key value must be a string, but the $value can be any type (or nil). Useful for passing more than one value as a pipeline context to subtemplates.
  • dir $path: Returns an array of filenames in the specified $path.
  • exists $path: Returns true if $path refers to an existing file or directory. Takes a string.
  • first $array: Returns the first value of an array or nil if the arry is nil or empty.
  • groupBy $containers $fieldPath: Groups an array of RuntimeContainer instances based on the values of a field path expression $fieldPath. A field path expression is a dot-delimited list of map keys or struct member names specifying the path from container to a nested value, which must be a string. Returns a map from the value of the field path expression to an array of containers having that value. Containers that do not have a value for the field path in question are omitted.
  • groupByKeys $containers $fieldPath: Returns the same as groupBy but only returns the keys of the map.
  • groupByMulti $containers $fieldPath $sep: Like groupBy, but the string value specified by $fieldPath is first split by $sep into a list of strings. A container whose $fieldPath value contains a list of strings will show up in the map output under each of those strings.
  • hasPrefix $prefix $string: Returns whether $prefix is a prefix of $string.
  • hasSuffix $suffix $string: Returns whether $suffix is a suffix of $string.
  • intersect $slice1 $slice2: Returns the strings that exist in both string slices.
  • json $value: Returns the JSON representation of $value as a string.
  • keys $map: Returns the keys from $map. If $map is nil, a nil is returned. If $map is not a map, an error will be thrown.
  • last $array: Returns the last value of an array.
  • parseBool $string: parseBool returns the boolean value represented by the string. It accepts 1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False. Any other value returns an error. Alias for strconv.ParseBool
  • replace $string $old $new $count: Replaces up to $count occurences of $old with $new in $string. Alias for strings.Replace
  • sha1 $string: Returns the hexadecimal representation of the SHA1 hash of $string.
  • split $string $sep: Splits $string into a slice of substrings delimited by $sep. Alias for strings.Split
  • trimPrefix $prefix $string: If $prefix is a prefix of $string, return $string with $prefix trimmed from the beginning. Otherwise, return $string unchanged.
  • trimSuffix $suffix $string: If $suffix is a suffix of $string, return $string with $suffix trimmed from the end. Otherwise, return $string unchanged.
  • trim $string: Removes whitespace from both sides of $string.
  • where $items $fieldPath $value: Filters an array or slice based on the values of a field path expression $fieldPath. A field path expression is a dot-delimited list of map keys or struct member names specifying the path from container to a nested value. Returns an array of items having that value.
  • whereExist $items $fieldPath: Like where, but returns only items where $fieldPath exists (is not nil).
  • whereNotExist $items $fieldPath: Like where, but returns only items where $fieldPath does not exist (is nil).
  • whereAny $items $fieldPath $sep $values: Like where, but the string value specified by $fieldPath is first split by $sep into a list of strings. The comparison value is a string slice with possible matches. Returns items which OR intersect these values.
  • whereAll $items $fieldPath $sep $values: Like whereAny, except all $values must exist in the $fieldPath.
  • when $condition $trueValue $falseValue: Returns the $trueValue when the $condition is true and the $falseValue otherwise

===

Examples

NGINX Reverse Proxy Config

jwilder/nginx-proxy trusted build.

Start nginx-proxy:

$ docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock -t jwilder/nginx-proxy

Then start containers with a VIRTUAL_HOST env variable:

$ docker run -e VIRTUAL_HOST=foo.bar.com -t ...

If you wanted to run docker-gen directly on the host, you could do it with:

$ docker-gen -only-published -watch -notify "/etc/init.d/nginx reload" templates/nginx.tmpl /etc/nginx/sites-enabled/default

Fluentd Log Management

This template generate a fluentd.conf file used by fluentd. It would then ships log files off the host.

$ docker-gen -watch -notify "restart fluentd" templates/fluentd.tmpl /etc/fluent/fluent.conf

Service Discovery in Etcd

This template is an example of generating a script that is then executed. This tempalte generates a python script that is then executed which register containers in Etcd using it's HTTP API.

$ docker-gen -notify "/bin/bash /tmp/etcd.sh" -interval 10 templates/etcd.tmpl /tmp/etcd.sh

Development

This project uses glock for managing 3rd party dependencies. You'll need to install glock into your workspace before hacking on docker-gen.

$ git clone <your fork>
$ cd <your fork>
$ make get-deps
$ make

TODO

  • Add event status for handling start and stop events differently

License

MIT