Skip to content
Tool that fetches consul KV's and vault secrets providing a simple export list that can be sourced
Go Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
dev
.gitignore
.travis.yml
Gopkg.lock
Gopkg.toml
LICENSE
README.md
common.go
common_test.go
consul.go
consul_test.go
devrun.sh
devsetup.sh
docker-compose.yml
flags.go
flags_test.go
gollum.gif
main.go
minimesosFile
secrets.go
secrets_test.go

README.md

Build Status

Gollum

My Precious!

Gollum collects consul key values and vault secrets from their respective paths and turns them into a set of exports that can be easily sourced.

This is meant to be used as a temporary stop gap to doing this the right way which is adding support for both of these things inside your application.

Download

You can find releases on the releases page

If I wanted to download this on a Mac I would do the following:

wget https://github.com/OrigamiLogic/gollum/releases/download/0.0.8/gollum.darwin.amd64
mv gollum.darwin.amd64 gollum
chmod 755 gollum

Alternatively if you are grabbing this from a script and always want the latest version you could do the following:

LATEST_VERSION=$(curl -s https://api.github.com/repos/OrigamiLogic/gollum/releases/latest | grep browser_download_url | grep 'linux.amd64' | cut -d '"' -f 4)
wget $LATEST_VERSION
mv gollum.linux.amd64 gollum
chmod 755 gollum

Usage

  -consulurl string
      URL for consul cluster
  -debug
      Print debug output
  -gatekeeperurl string
      URL for vault gatekeeper
  -keypath string
      Path to consul key values for configuration
  -outfile string
      Export vault values to outfile
  -vaultkeypath string
      Vault path to key with secrets
  -vaulturl string
      URL for vault
  -envoverrides
      Do not load the value if the key already exists prefixed with GOLLUM_
  -version
      prints current app version

You would typically place this inside your docker container, but in this case we will just walk through a few scenarios

Collecting Key Values from Consul

The following will connect to https://consul.domain.com and retrieve all the key values in the settings/myapp/production/ path and convert them into a source export file with environment variables, for example assume you have the following in consul

settings/myapp/production/APPLICATION_NAME=foo
settings/myapp/production/FAVORITE_DRINK=absinthe
settings/myapp/production/ANSWER_TO_THE_UNIVERSE=42

Running this command:

./gollum -consulurl=https://consul.domain.com -keypath=settings/myapp/production/ -outfile=/tmp/config

Writes the following output to /tmp/config:

export APPLICATION_NAME=foo
export FAVORITE_DRINK=absinthe
export ANSWER_TO_THE_UNIVERSE=42

Collecting secrets from vault

Gollum assumes you're using vault-gatekeeper-mesos to handle the management of temporary tokens, AKA vault response wrapping.

Assuming your vault-gatekeeper-mesos instance is running at https://vault-gatekeeper.domain.com, your vault service is available at https://vault.domain.com:8200 and you have the following in vault at this path secret/myapp/production/secrets

secret/myapp/production/secrets
     MYSQL_PASSWORD=IloveTheSmellofSQLInTheMorning
     MONGODB_PASSWORD=CauseWhoWantsRelationships
     lowercase=shouting_is_fun

Running this command:

./gollum -gatekeeperurl=https://vault-gatekeeper.domain.com -vaulturl=https://vault.domain.com:8200 -vaultkeypath=secret/myapp/production/secrets -outfile=/tmp/config

Writes the following output to /tmp/config:

export MYSQL_PASSWORD=IloveTheSmellofSQLInTheMorning
export MONGODB_PASSWORD=CauseWhoWantsRelationships
export lowercase=shouting_is_fun

All the things

We can produce output for both consul and vault at the same time, given the assumptions of the examples above:

./gollum -consulurl=https://consul.domain.com -keypath=settings/myapp/production/ -gatekeeperurl=https://vault-gatekeeper.domain.com -vaulturl=https://vault.domain.com:8200 -vaultkeypath=secret/myapp/production/secrets -outfile=/tmp/config

Writes the following output to /tmp/config:

export APPLICATION_NAME=foo
export FAVORITE_DRINK=absinthe
export ANSWER_TO_THE_UNIVERSE=42
export MYSQL_PASSWORD=IloveTheSmellofSQLInTheMorning
export MONGODB_PASSWORD=CauseWhoWantsRelationships
export lowercase=shouting_is_fun

In the real world we would probably run gollum in an entrypoint.sh file within a docker container as follows:

./gollum -consulurl=https://consul.domain.com \
  -keypath=settings/myapp/production/ \
  -gatekeeperurl=https://vault-gatekeeper.domain.com \
  -vaulturl=https://vault.domain.com:8200
  -vaultkeypath=secret/myapp/production/secrets \
  -outfile=/tmp/config

source /tmp/config

This will make the values within vault and consul available as environment variables within your container.

Debugging

In the case where gollum may not be behaving correctly running it with -debug will provide debug level output, more specifically it will output the temp vault token and actual vault token received from vault-gatekeeper, in many cases, the token you're receiving may not have the correct policy applied and troubleshooting becomes significantly simpler when you have this information available. Please do not run gollum in production using the -debug flag.

Development

Development workflow

  1. go get https://github.com/OrigamiLogic/gollum
  2. cd $GOPATH/src/github.com/OrigamiLogic/gollum
  3. dep ensure
  4. go test

Dev Environment Requirements

Currently this has only been tested on macOS and Ubuntu Linux (16.04 LTS xenial), it may require some effort to make it work elsewhere

  1. Docker version 18.02 or higher
  2. minimesos
  3. jq

Running the Dev Environment

Make sure you have met the system level requirements

  1. Run the devsetup.sh script to start minimesos, the docker-compose stack, and create required data If you are running this in Linux, run export DOCKER_HOST_IP=172.17.0.1 (your actual docker host ip) before this script.
./devsetup.sh up
starting minimesos
Detected Mac Environment X, running with '--mapPortsToHost' so master and marathon ports are mapped to localhost
Detected Mac Environment X so running with --mapPortsToHost so master and marathon ports are mapped to localhost.
Minimesos cluster is running: 4022571272
Mesos version: 1.0.0
export MINIMESOS_NETWORK_GATEWAY=172.19.0.1
export MINIMESOS_AGENT=http://172.19.0.6:5051; export MINIMESOS_AGENT_IP=172.19.0.6
export MINIMESOS_ZOOKEEPER=zk://172.19.0.3:2181/mesos; export MINIMESOS_ZOOKEEPER_IP=172.19.0.3
export MINIMESOS_MARATHON=http://172.19.0.5:8080; export MINIMESOS_MARATHON_IP=172.19.0.5
export MINIMESOS_MASTER=http://172.19.0.4:5050; export MINIMESOS_MASTER_IP=172.19.0.4
You are running Docker on Mac so use localhost instead of container IPs for Master, Marathon, Zookeepr and Consul
running docker-compose
Creating network "gollum_gollum" with the default driver
Creating gollum_consul_1 ... done
Creating gollum_vault_1  ... done
Creating gollum_vault-gatekeeper-mesos_1 ... done
populating consul KV store
true
true
true
adding vault policy for gollumtestapp
adding vault gatekeeper policy for gollumtestapp
refreshing vault-gatekeeper-mesos policies
{"status":"Unsealed","ok":true}
adding secrets to vault
run gollumtest app in marathon
{"id":"/gollumtestapp","cmd":null,"args":null,"user":null,"env":{},"instances":1,"cpus":1,"mem":128,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"
backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"nginx","network":"BRIDGE","portMappings":null,"privileged":false,"parameters":[],"forcePullImag
e":false}},"healthChecks":[],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":1,"maximumOverCapacity":1},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version"
:"2018-06-12T23:53:11.951Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[0],"portDefinitions":[{"port":0,"protocol":"tcp","labels":{}}],"requirePorts":false,"tasksStaged":0,"tas
ksRunning":0,"tasksHealthy":0,"tasksUnhealthy":0,"deployments":[{"id":"03f6457c-bf12-451d-917a-c2e328ac0ea1"}],"tasks":[]}

2.) To automatically compile and integration test your app run ./devrun.sh. You can run this script over and over to iteratively test changes that you make to your app. If running golang from docker, you can override consul/vault's base url with the base_url variable, i.e. export base_url=http://172.17.0.1.

./devrun.sh
restarting the gollumtestapp so we can get a new mesos task id
{"version":"2018-06-12T23:59:48.601Z","deploymentId":"b092c67b-946a-40aa-b49b-aac0a80fb43b"}
waiting 15 seconds for task to restart
INFO[0000] Consul settings activated
DEBU[0000] export testkey1=testvalue1 written to /tmp/gollumtestapp
DEBU[0000] export testkey2=testvalue2 written to /tmp/gollumtestapp
DEBU[0000] export testkey3=testvalue3 written to /tmp/gollumtestapp
INFO[0000] Wrote 3 consul entries to /tmp/gollumtestapp
INFO[0000] Vault settings activated
DEBU[0000] Received unwrapped token from vault: ac91b7a1-732a-a876-f8e8-ce7aa5093b03
INFO[0000] Succesfully unwrapped token from vault
DEBU[0000] export SECRETKEY1=secretvalue1 written to /tmp/gollumtestapp
DEBU[0000] export SECRETKEY2=secretvalue2 written to /tmp/gollumtestapp
INFO[0000] Wrote 2 vault entries to /tmp/gollumtestapp
Contents of /tmp/gollumtestapp
export testkey1=testvalue1
export testkey2=testvalue2
export testkey3=testvalue3
export SECRETKEY1=secretvalue1
export SECRETKEY2=secretvalue2

3.) To destroy the development environment

./devsetup.sh down

Contributing

  1. Fork it ( https://github.com/OrigamiLogic/gollum )
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request
You can’t perform that action at this time.