New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds Vault data source #40

Merged
merged 1 commit into from Sep 7, 2016
Jump to file or symbol
Failed to load files and symbols.
+478 −4
Diff settings

Always

Just for now

View
@@ -8,9 +8,9 @@ group :development do
gem 'httpclient'
gem 'oj'
gem 'zk'
gem 'vault', '~> 0.1'
gem 'crack'
gem 'rubyzip'
gem 'diplomat' , '~> 0.18.0'
gem 'tiller', :path => '.'
end
View
@@ -0,0 +1,156 @@
# Vault plugins
Tiller includes plugins to retrieve templates and values [Vault](https://www.vaultproject.io) cluster. These plugins rely on the `vault` gem to be present, so before proceeding ensure you have run `gem install vault` in your environment. This is not listed as a hard dependency of Tiller, as this would force the gem to be installed even on systems that would never use these plugins.
# Enabling the plugins
Add the `vault` plugins in your `common.yaml`, e.g.
```yaml
data_sources: [ "vault" ]
template_sources: [ "vault" ]
```
If you're fetching all your values and templates from Vault, those should be the only plugins you need.
However, you do not need to enable both plugins; for example you may just want to retrieve values for your templates from Vault, but continue to use files to store your actual template content. For example :
```yaml
data_sources: [ "vault", "environment" ]
template_sources: [ "file" ]
```
The above example would use templates from files, and retrieve values from Vault first, then try the `environment` plugin.
# Configuring
Configuration for this plugin is placed inside a "vault" block. This should be in the the top-level of `common.yaml` file, or in a per-environment block. See the main [README.md](https://github.com/markround/tiller/blob/master/README.md#common-configuration) for more information on this.
A sample configuration (showing the defaults for most parameters) is as follows :
```yaml
vault:
url: 'https://localhost:8200'
token: '8313ef8c-0c6f-ca24-2783-ab92f10d7717'
ssl_verify: false
templates: '/secret/tiller/templates'
values:
global: '/secret/tiller/globals/all'
per_env: 'secret/tiller/globals/%e'
template: '/secret/tiller/values/%e/%t'
target: '/secret/tiller/target_values/%t/%e'
```
At a bare minimum, you need to specify a URL for the plugins to connect to. This is the HTTP port of your Vault server, e.g. `http://localhost:8200`. If you would like to verify the validity of certificate, set `ssl_verify` to `true` and provide path to certificate with `ssl_pem_file`. URL should also be switched to `https`. If you're happy to accept the rest of the defaults, your configuration can therefore be as simple as this :
```yaml
data_sources: [ "vault" ]
template_sources: [ "vault" ]
vault:
url: 'http://localhost:8200'
```
## Authentication
Vault requires a token in order to connect to it. If you omit the token parameter, Tiller will look for it in your `~/.vault-token` file (which is created automatically when vault is started in dev mode).
```yaml
vault:
url: 'http://localhost:8200'
token: '8313ef8c-0c6f-ca24-2783-ab92f10d7717'
```
# Paths
You can use any K/V hierarchy inside Vault, but the default is expected to look like the following:
Since Vault stores documents in JSON with a body like:
```json
{
"content": "bar"
}
```
by default Tiller will assume that the key name is `content`, however it is configurable with `json_key_name` parameter. If you want it to be stored, for example, as
```json
{
"value": "bar"
}
```
you can configure Tiller as follows:
```yaml
vault:
url: 'https://localhost:8200'
json_key_name: :value
```
/secret
├──tiller
├── globals
│ ├── all
│ │ └── some_key_for_all_environments -> { "content" : "some_configuration_value" }
│ │
│ ├── production
│ │ └── some_key_only_for_production_environment -> { "content" : "some_configuration_value" }
│ │
│ ... more environments here...
├── templates (each key contains the ERB template as its value)
│ ├── template1.erb -> { "content" : "template contents..."}
│ ├── template2.erb -> { "content" : "template contents..."}
│ ... more templates here ...
├── values
│ ├── production (keys and values for the 'production' environment)
│ │ ├ template1.erb
│ │ │ ├── some_key
│ │ │ ├── some_other_key
│ │ ├ template2.erb
│ │ │ ├── some_key
│ │ │ ├── some_other_key
│ │ ...more templates and keys...
│ │
│ └── development (keys and values for the 'development' environment)
│ ├ template1.erb
│ │ ├── some_key
│ │ ├── some_other_key
│ ├ template2.erb
│ │ ├── some_key
│ │ ├── some_other_key
│ ...more templates and keys...
└── target_values (controls which templates get installed and where)
├── template1.erb
│ ├── production
│ │ └── target (where to install the template in production)
│ └── development
│ └── target (where to install the template in development)
└── template1.erb (don't install template2.erb in development)
└── production
└── target (where to install the template in production)
You can change this to any structure you like by altering the `templates` and `values` parameters. The paths specified for any of these parameters listed above may include the following placeholders :
* `%e` : This will be replaced with the value of the current environment
* `%t` : This will be replaced with the value of the current template
There is a benefit to keeping this default layout though: if you're using a shared Vault service, it makes it easy to define [ACLs](https://www.vaultproject.io/intro/getting-started/acl.html) so that you can, for example, deny access to the `/values/production` or `/globals/production` paths for non-production services.
# Accessing data from templates
## K/V store
Vault keys and their values will be exposed to templates as regular variables. So, using the example structure above, you could just reference a vault key for your environment/template within your template like so :
```erb
This is a value for template1 : <%= some_key %>
This is a global value : <%= some_key_for_all_environments %>
This should only be present in production : <%= some_key_only_for_production_environment %>
```
@@ -0,0 +1,7 @@
---
exec: ['date']
data_sources: ['vault']
template_sources: [ 'vault' ]
vault:
url: 'http://127.0.0.1:8200'
@@ -0,0 +1,46 @@
require 'open-uri'
require 'vault'
require 'pp'
# Vault configuration
VAULT_TOKEN_FILE = "#{Dir.home}/.vault-token"
# So that Cucumber does not complain that the file does not exist
File.open(VAULT_TOKEN_FILE, "w+"){|file| file.write(".")} if !File.exists? VAULT_TOKEN_FILE
Vault.configure do |config|
config.address = "http://127.0.0.1:8200"
config.token = File.read(VAULT_TOKEN_FILE)
end
When(/^I have downloaded vault "(.+)" to "(.+)"$/) do |version, path|
if RUBY_PLATFORM =~ /darwin/
uri = "https://releases.hashicorp.com/vault/#{version}/vault_#{version}_darwin_amd64.zip"
elsif RUBY_PLATFORM =~ /linux/
uri = "https://releases.hashicorp.com/vault/#{version}/vault_#{version}_linux_amd64.zip"
else
fail!("Unsupported platform for vault")
end
puts "Downloading #{uri}"
download = open(uri)
IO.copy_stream(download, path)
end
And (/^a token should be created$/) do
test = File.exists? VAULT_TOKEN_FILE
expect(test).to be_truthy
end
Given(/^I have populated vault with test data$/) do
Vault.configure do |config|
config.address = "http://127.0.0.1:8200"
config.token = File.read(VAULT_TOKEN_FILE)
end
populate_vault_test_data
end
Then (/^the vault key "(.+)" should exist$/) do |key|
test = Vault.logical.read(key)
expect(test.data).to be_truthy
end
@@ -0,0 +1,49 @@
#!/usr/bin/env ruby
require 'vault'
require 'pp'
def populate_vault_test_data(url='http://127.0.0.1:8200')
Vault.configure do |config|
config.address = url
end
# Template contents
template1 = %{This is template1.
This is a value from Vault : <%= vault_value %>
This is a global value from Vault : <%= vault_global %>
This is a per-environment global : <%= vault_per_env %>}
template2 = %{This is template2.
This is a value from Vault : <%= vault_value %>
This is a global value from Vault : <%= vault_global %>
This is a per-environment global : <%= vault_per_env %>}
# Populate globals
Vault.logical.write('/secret/tiller/globals/all/vault_global', content: 'Vault global value')
# Populate per-environment globals
Vault.logical.write('/secret/tiller/globals/development/vault_per_env', content: 'per-env global for development enviroment')
Vault.logical.write('/secret/tiller/globals/production/vault_per_env', content: 'per-env global for production enviroment')
# Populate template values for development environment
Vault.logical.write('/secret/tiller/values/development/template1.erb/vault_value', content: 'development value from Vault for template1.erb')
Vault.logical.write('/secret/tiller/values/development/template1.erb/vault_per_env', content: 'This is over-written for template1 in development')
Vault.logical.write('/secret/tiller/values/development/template2.erb/vault_value', content: 'development value from Vault for template2.erb')
# Populate template values for production environment
Vault.logical.write('/secret/tiller/values/production/template1.erb/vault_value', content: 'production value from Vault for template1.erb')
Vault.logical.write('/secret/tiller/values/production/template2.erb/vault_value', content: 'production value from Vault for template2.erb')
# Populate target_values for environments
Vault.logical.write('/secret/tiller/target_values/template1.erb/development/target', content: 'template1.txt')
Vault.logical.write('/secret/tiller/target_values/template2.erb/development/target', content: 'template2.txt')
# No template 2 for production
Vault.logical.write('/secret/tiller/target_values/template1.erb/production/target', content: 'template1.txt')
# Populate templates content
Vault.logical.write('/secret/tiller/templates/template1.erb', content: template1)
Vault.logical.write('/secret/tiller/templates/template2.erb', content: template2)
end
if ! defined?(Cucumber)
url = ARGV[0] ? ARGV[0] : "http://127.0.0.1:8200"
puts "Populating Vault at #{url} with test data"
populate_vault_test_data(url)
end
View
@@ -0,0 +1,50 @@
Feature: Vault plugin
Scenario: Download Vault
When I have downloaded vault "0.6.1" to "/tmp/vault.zip"
And I have unzipped the archive "/tmp/vault.zip"
And I have made the file "/tmp/vault" executable"
Then an absolute file named "/tmp/vault" should exist
Scenario: Start vault daemon in development mode
When I start my daemon with "/tmp/vault server -dev"
Then a daemon called "vault" should be running
And a token should be created
Scenario: Populate vault with test data
Given I have populated vault with test data
Then the vault key "secret/tiller/globals/all/vault_global" should exist
Scenario: Test dev environment template generation with Vault
Given I use a fixture named "vault"
When I successfully run `tiller -b . -v -n`
Then a file named "template1.txt" should exist
And the file "template1.txt" should contain:
"""
This is template1.
This is a value from Vault : development value from Vault for template1.erb
This is a global value from Vault : Vault global value
This is a per-environment global : This is over-written for template1 in development
"""
And a file named "template2.txt" should exist
And the file "template2.txt" should contain:
"""
This is template2.
This is a value from Vault : development value from Vault for template2.erb
This is a global value from Vault : Vault global value
This is a per-environment global : per-env global for development enviroment
"""
Scenario: Test prod environment template generation with Vault
Given I use a fixture named "vault"
When I successfully run `tiller -b . -v -n -e production`
Then a file named "template1.txt" should exist
And the file "template1.txt" should contain:
"""
This is template1.
This is a value from Vault : production value from Vault for template1.erb
This is a global value from Vault : Vault global value
This is a per-environment global : per-env global for production enviroment
"""
And a file named "template2.txt" should not exist
View
@@ -0,0 +1,59 @@
require 'yaml'
require 'vault'
require 'tiller/datasource'
require 'tiller/vault.rb'
class VaultDataSource < Tiller::DataSource
include Tiller::VaultCommon
def global_values
path = interpolate("#{@vault_config['values']['global']}")
Tiller::log.debug("#{self} : Fetching globals from #{path}")
globals = get_values(path)
# Do we have per-env globals ? If so, merge them
path = interpolate("#{@vault_config['values']['per_env']}")
Tiller::log.debug("#{self} : Fetching per-environment globals from #{path}")
globals.deep_merge!(get_values(path))
end
def values(template_name)
path = interpolate("#{@vault_config['values']['template']}", template_name)
Tiller::log.debug("#{self} : Fetching template values from #{path}")
get_values(path)
end
def target_values(template_name)
path = interpolate("#{@vault_config['values']['target']}", template_name)
Tiller::log.debug("#{self} : Fetching template target values from #{path}")
get_values(path)
end
# Helper method, not used by DataSource API
def get_values(path)
keys = nil
Vault.with_retries(Vault::HTTPConnectionError, Vault::HTTPError) do |attempt, e|
Tiller::log.warn("#{self} : Received exception #{e} from Vault") if e
keys = Vault.logical.list(path)
end
values = {}
if keys.is_a? Array
keys.each do |k|
Tiller::log.debug("#{self} : Fetching value at #{path}/#{k}")
Vault.with_retries(Vault::HTTPConnectionError, Vault::HTTPError) do |attempt, e|
Tiller::log.warn("#{self} : Received exception #{e} from Vault") if e
values[k] = Vault.logical.read(File.absolute_path(k,path)).data[@vault_config['json_key_name']]
end
end
values
else
{}
end
end
end
Oops, something went wrong.
ProTip! Use n and p to navigate between commits in a pull request.