Merge branch 'release/0.9.3'

CHANGELOG.md

@@ -2,6 +2,11 @@
 
 ## 0.9.x 
 
+* 0.9.3 (07/Sep/2016)
+  * New Hashicorp [Vault](https://www.vaultproject.io/) plugin, contributed by [liquid-sky](https://github.com/liquid-sky). Thanks so much for this _awesome_ PR!
+  * HTTP, Vault and Consul plugins just log an informational message if no configuration block exists for that environment. This lets you only enable these plugins in some environments.
+  * Cleaned up some duplicate information log messages
+
 * 0.9.2 (19/Aug/2016) : No changes, just corrected date in gemspec.
 
 * 0.9.1 (17/Aug/2016) : Added `deep_merge` flag to optionally merge hash values, instead of replacing them with values from a higher priority plugin ([issue #38](https://github.com/markround/tiller/issues/38)). Thanks to [pgleyzer](https://github.com/pgleyzer) for raising this issue.

Gemfile

@@ -8,9 +8,9 @@ group :development do
   gem 'httpclient'
   gem 'oj'
   gem 'zk'
+  gem 'vault', '~> 0.1'
   gem 'crack'
   gem 'rubyzip'
   gem 'diplomat' , '~> 0.18.0'
   gem 'tiller', :path => '.'
 end
-

README.md

@@ -104,6 +104,7 @@ In addition to specifying values in YAML environment files, there are other plug
  * [Random data](docs/plugins/random.md) : Simple wrapper to provide random values to your templates.
  * [XML files](docs/plugins/xml_file.md) : Load and parse XML data for use in your templates.
  * [Zookeeper plugins](docs/plugins/zookeeper.md) : These plugins allow you to store your templates and values in a ZooKeeper cluster.
+ * [Hashicorp Vault](docs/plugins/vault.md) : These plugins allow you to to store and retrieve your templates and values from the Hashicorp [Vault](https://www.vaultproject.io/) secrets store.
 
 ### Helper modules
 You can also make use of custom utility functions in Ruby that can be called from within templates. For more information on this, see the [developers documentation](docs/developers.md#helper-modules).
@@ -145,7 +146,7 @@ data_sources:
   - environment
 ```
 
-So, to summarise: *A template value will take priority over a global value, and any value from a plugin loaded later will take priority over any previously loaded plugins.*
+So, to summarise: A template value will take priority over a global value, and a value from a plugin loaded later will take priority over any previously loaded plugins.
 
 ## Arguments
 Tiller understands the following *optional* command-line arguments (mostly used for debugging purposes) :

bin/tiller

@@ -35,7 +35,7 @@ module Tiller
 
   puts "tiller v#{VERSION} (https://github.com/markround/tiller) <github@markround.com>"
 
-  class << self;
+  class << self
     attr_accessor :config, :log, :templates, :tiller
   end
 

docs/plugins/vault.md

@@ -0,0 +1,156 @@
+# Vault plugins
+
+Tiller includes plugins to retrieve templates and values [Vault](https://www.vaultproject.io) cluster. These plugins rely on the `vault` gem to be present, so before proceeding ensure you have run `gem install vault` in your environment. This is not listed as a hard dependency of Tiller, as this would force the gem to be installed even on systems that would never use these plugins.
+
+# Enabling the plugins
+Add the `vault` plugins in your `common.yaml`, e.g.
+
+```yaml
+data_sources: [ "vault" ]
+template_sources: [ "vault" ]
+```
+
+If you're fetching all your values and templates from Vault, those should be the only plugins you need.
+
+However, you do not need to enable both plugins; for example you may just want to retrieve values for your templates from Vault, but continue to use files to store your actual template content. For example :
+
+```yaml
+data_sources: [ "vault", "environment" ]
+template_sources: [ "file" ]
+```
+
+The above example would use templates from files, and retrieve values from Vault first, then try the `environment` plugin.
+
+
+# Configuring
+Configuration for this plugin is placed inside a "vault" block. This should be in the the top-level of `common.yaml` file, or in a per-environment block. See the main [README.md](https://github.com/markround/tiller/blob/master/README.md#common-configuration) for more information on this.
+
+A sample configuration (showing the defaults for most parameters) is as follows :
+
+```yaml
+vault:
+  url: 'https://localhost:8200'
+  token: '8313ef8c-0c6f-ca24-2783-ab92f10d7717'
+  ssl_verify: false
+
+  templates: '/secret/tiller/templates'
+  values:
+   global: '/secret/tiller/globals/all'
+   per_env: 'secret/tiller/globals/%e'
+   template: '/secret/tiller/values/%e/%t'
+   target: '/secret/tiller/target_values/%t/%e'
+```
+
+At a bare minimum, you need to specify a URL for the plugins to connect to. This is the HTTP port of your Vault server, e.g. `http://localhost:8200`. If you would like to verify the validity of certificate, set `ssl_verify` to `true` and provide path to certificate with `ssl_pem_file`. URL should also be switched to `https`. If you're happy to accept the rest of the defaults, your configuration can therefore be as simple as this :
+
+```yaml
+data_sources: [ "vault" ]
+template_sources: [ "vault" ]
+vault:
+  url: 'http://localhost:8200'
+```
+
+## Authentication
+
+Vault requires a token in order to connect to it. If you omit the token parameter, Tiller will look for it in your `~/.vault-token` file (which is created automatically when vault is started in dev mode). 
+
+
+```yaml
+vault:
+  url: 'http://localhost:8200'
+  token: '8313ef8c-0c6f-ca24-2783-ab92f10d7717'
+```
+
+# Paths
+You can use any K/V hierarchy inside Vault, but the default is expected to look like the following:
+
+Since Vault stores documents in JSON with a body like: 
+
+```json
+{
+  "content": "bar"
+}
+```
+
+by default Tiller will assume that the key name is `content`, however it is configurable with `json_key_name` parameter. If you want it to be stored, for example, as 
+
+```json
+{
+  "value": "bar"
+}
+```
+
+you can configure Tiller as follows: 
+
+```yaml
+vault:
+  url: 'https://localhost:8200'
+  json_key_name: :value
+```
+	/secret
+	  ├──tiller
+		 ├── globals
+	 	 │   ├── all
+	 	 │   │   └── some_key_for_all_environments -> { "content" : "some_configuration_value" }
+	 	 │   │
+	 	 │   ├── production
+	 	 │   │   └── some_key_only_for_production_environment -> { "content" : "some_configuration_value" }
+	 	 │   │
+	 	 │   ... more environments here...
+	 	 │
+	 	 ├── templates (each key contains the ERB template as its value)
+	 	 │   ├── template1.erb -> { "content" : "template contents..."}
+	 	 │   ├── template2.erb -> { "content" : "template contents..."}
+	 	 │   ... more templates here ...
+	 	 │
+	 	 ├── values
+	 	 │   ├── production (keys and values for the 'production' environment)
+	 	 │   │       ├ template1.erb
+	 	 │   │       │     ├── some_key 
+	 	 │   │       │     ├── some_other_key
+     	 │   │       ├ template2.erb
+	 	 │   │       │     ├── some_key
+	 	 │   │       │     ├── some_other_key
+     	 │   │       ...more templates and keys...
+ 	 	 │   │
+ 	 	 │   └── development (keys and values for the 'development' environment)
+	 	 │           ├ template1.erb
+	 	 │           │     ├── some_key
+	 	 │           │     ├── some_other_key
+     	 │           ├ template2.erb
+	 	 │           │     ├── some_key
+	 	 │           │     ├── some_other_key
+     	 │           ...more templates and keys...
+ 	 	 │
+ 	 	 │
+    	 └── target_values (controls which templates get installed and where)
+	     	├── template1.erb
+	     	│   ├── production
+	     	│   │       └── target (where to install the template in production)
+	     	│   └── development
+	     	│           └── target (where to install the template in development)
+	     	│
+	     	└── template1.erb (don't install template2.erb in development)
+	       	  └── production
+	                  └── target (where to install the template in production)
+
+
+
+You can change this to any structure you like by altering the `templates` and `values` parameters. The paths specified for any of these parameters listed above may include the following placeholders :
+
+* `%e` : This will be replaced with the value of the current environment
+* `%t` : This will be replaced with the value of the current template
+
+There is a benefit to keeping this default layout though: if you're using a shared Vault service, it makes it easy to define [ACLs](https://www.vaultproject.io/intro/getting-started/acl.html) so that you can, for example, deny access to the `/values/production` or `/globals/production` paths for non-production services.
+
+# Accessing data from templates
+
+## K/V store
+Vault keys and their values will be exposed to templates as regular variables. So, using the example structure above, you could just reference a vault key for your environment/template within your template like so :
+
+```erb
+This is a value for template1 : <%= some_key %>
+This is a global value : <%= some_key_for_all_environments %>
+This should only be present in production : <%= some_key_only_for_production_environment %>
+```
+

features/consul.feature

@@ -57,3 +57,30 @@ Feature: Consul plugin
     And the file "template1.txt" should contain "Nodes : {"
     And the file "template1.txt" should contain "ServicePort=8300"
 
+  Scenario: Test environment without Consul block
+    Given a file named "common.yaml" with:
+    """
+    ---
+    exec: ["true"]
+    data_sources: [ "consul" , "file" ]
+    template_sources: [ "consul" , "file" ]
+
+    environments:
+      development:
+        test.erb:
+          target: test.txt
+          config:
+            test_var: "This is a template var from the development env"
+    """
+    And a directory named "templates"
+    And a file named "templates/test.erb" with:
+    """
+    test_var: <%= test_var %>
+    """
+    When I successfully run `tiller -b . -v -n -e development`
+    Then a file named "test.txt" should exist
+    And the file "test.txt" should contain:
+    """
+    test_var: This is a template var from the development env
+    """
+    And the output should contain "No Consul configuration block for this environment"

features/fixtures/vault/common.yaml

@@ -0,0 +1,7 @@
+---
+exec: ['date']
+data_sources: ['vault']
+template_sources: [ 'vault' ]
+
+vault:
+  url: 'http://127.0.0.1:8200'

features/http.feature

@@ -15,4 +15,32 @@ Some globals, now.
 
  * This is a value from HTTP 
  * Another global value 
-"""
+"""
+
+  Scenario: Test environment without HTTP block
+    Given a file named "common.yaml" with:
+    """
+    ---
+    exec: ["true"]
+    data_sources: [ "http" , "file" ]
+    template_sources: [ "http" , "file" ]
+
+    environments:
+      development:
+        test.erb:
+          target: test.txt
+          config:
+            test_var: "This is a template var from the development env"
+    """
+    And a directory named "templates"
+    And a file named "templates/test.erb" with:
+    """
+    test_var: <%= test_var %>
+    """
+    When I successfully run `tiller -b . -v -n -e development`
+    Then a file named "test.txt" should exist
+    And the file "test.txt" should contain:
+    """
+    test_var: This is a template var from the development env
+    """
+    And the output should contain "No HTTP configuration block for this environment"

features/step_definitions/vault.rb

@@ -0,0 +1,46 @@
+require 'open-uri'
+require 'vault'
+require 'pp'
+
+# Vault configuration
+VAULT_TOKEN_FILE = "#{Dir.home}/.vault-token"
+# So that Cucumber does not complain that the file does not exist
+File.open(VAULT_TOKEN_FILE, "w+"){|file| file.write(".")} if !File.exists? VAULT_TOKEN_FILE
+
+Vault.configure do |config|
+  config.address = "http://127.0.0.1:8200"
+  config.token = File.read(VAULT_TOKEN_FILE)
+end
+
+When(/^I have downloaded vault "(.+)" to "(.+)"$/) do |version, path|
+  if RUBY_PLATFORM =~ /darwin/
+    uri = "https://releases.hashicorp.com/vault/#{version}/vault_#{version}_darwin_amd64.zip"
+  elsif RUBY_PLATFORM =~ /linux/
+    uri = "https://releases.hashicorp.com/vault/#{version}/vault_#{version}_linux_amd64.zip"
+  else
+    fail!("Unsupported platform for vault")
+  end
+  puts "Downloading #{uri}"
+
+  download = open(uri)
+  IO.copy_stream(download, path)
+end
+
+And (/^a token should be created$/) do
+  test = File.exists? VAULT_TOKEN_FILE
+  expect(test).to be_truthy
+end
+
+
+Given(/^I have populated vault with test data$/) do
+  Vault.configure do |config|
+    config.address = "http://127.0.0.1:8200"
+    config.token = File.read(VAULT_TOKEN_FILE)
+  end
+  populate_vault_test_data
+end
+
+Then (/^the vault key "(.+)" should exist$/) do |key|
+  test = Vault.logical.read(key)
+  expect(test.data).to be_truthy
+end

features/support/vault_test_data.rb

@@ -0,0 +1,49 @@
+#!/usr/bin/env ruby
+
+require 'vault'
+require 'pp'
+
+def populate_vault_test_data(url='http://127.0.0.1:8200')
+  Vault.configure do |config|
+    config.address = url
+  end
+
+  # Template contents
+  template1 = %{This is template1.
+This is a value from Vault : <%= vault_value %>
+This is a global value from Vault : <%= vault_global %>
+This is a per-environment global : <%= vault_per_env %>}
+
+  template2 = %{This is template2.
+This is a value from Vault : <%= vault_value %>
+This is a global value from Vault : <%= vault_global %>
+This is a per-environment global : <%= vault_per_env %>}
+
+
+  # Populate globals
+  Vault.logical.write('/secret/tiller/globals/all/vault_global', content: 'Vault global value')
+  # Populate per-environment globals
+  Vault.logical.write('/secret/tiller/globals/development/vault_per_env', content: 'per-env global for development enviroment')
+  Vault.logical.write('/secret/tiller/globals/production/vault_per_env', content: 'per-env global for production enviroment')
+  # Populate template values for development environment
+  Vault.logical.write('/secret/tiller/values/development/template1.erb/vault_value', content: 'development value from Vault for template1.erb')
+  Vault.logical.write('/secret/tiller/values/development/template1.erb/vault_per_env', content: 'This is over-written for template1 in development')
+  Vault.logical.write('/secret/tiller/values/development/template2.erb/vault_value', content: 'development value from Vault for template2.erb')
+  # Populate template values for production environment
+  Vault.logical.write('/secret/tiller/values/production/template1.erb/vault_value', content: 'production value from Vault for template1.erb')
+  Vault.logical.write('/secret/tiller/values/production/template2.erb/vault_value', content: 'production value from Vault for template2.erb')
+  # Populate target_values for environments
+  Vault.logical.write('/secret/tiller/target_values/template1.erb/development/target', content: 'template1.txt')
+  Vault.logical.write('/secret/tiller/target_values/template2.erb/development/target', content: 'template2.txt')
+  # No template 2 for production
+  Vault.logical.write('/secret/tiller/target_values/template1.erb/production/target', content: 'template1.txt')
+  # Populate templates content
+  Vault.logical.write('/secret/tiller/templates/template1.erb', content: template1)
+  Vault.logical.write('/secret/tiller/templates/template2.erb', content: template2)
+end
+
+if ! defined?(Cucumber)
+  url = ARGV[0] ? ARGV[0] : "http://127.0.0.1:8200"
+  puts "Populating Vault at #{url} with test data"
+  populate_vault_test_data(url)
+end