Merge pull request #6 from webframp/develop

Release 1.3 for new pseudo parameter behavior

README.md

@@ -1,8 +1,11 @@
-# SparkleFormation Vault Read Callback
+# SparkleFormation Vault Callback
 
 Provides a mechanism to read dynamic credentials for use with AWS Orchestration
 APIs from a [Vault](https://www.vaultproject.io/intro/getting-started/dynamic-secrets.html) secret backend.
 
+Also provides a method to set and store template parameters in a configured
+Vault instance. This is implemented by handling a custom parameter 'type'.
+
 **This is early alpha quality code**
 
 Currently supported cloud providers:
@@ -43,7 +46,7 @@ end
 #### Configuration
 
 The default read path is `aws/creds/deploy` and will be used without
-configuration but it is customizable. 
+configuration but it is customizable.
 
 Vault read configuration is controlled within the `.sfn` file:
 
@@ -89,6 +92,82 @@ Configuration.new
 end
 ~~~
 
+### Vault Pseudo Parameters
+Cloudformation parameters can have
+optional
+[AWS-Specific Parameter Types](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html?shortFooter=true#aws-specific-parameter-types).
+In a similar way this callback looks for a special parameter type named
+`Vault::Generic::Secret` and will dynamically get or set a key value from Vault.
+The key will be named to match the parameters name. Then the parameter type in the
+template is changed to 'String' which can be understood by AWS.
+
+Generally these should be set as `NoEcho` parameters and a dsl helper method is
+provided to generate this type of named parameter.
+
+Example usage in template:
+~~~ruby
+vault_parameter!(:secret_value)
+~~~
+
+Will result in a template with the following parameter defined:
+~~~json
+"Parameters": {
+  "SecretValue": {
+    "NoEcho": true,
+    "Description": "Generated secret automatically stored in Vault",
+    "Type": "String"
+  }
+}
+~~~
+
+And the value of this parameter will be stored and retrieved from a stack
+specific Vault key by default named like:
+
+~~~
+cubbyhole/SparkleFormation/template/SecretValue
+~~~
+
+In this example the template is named 'template', but this will be replaced with
+the stack name during create/update operations.
+
+The value will be stored in vault and retrieved dynamically at stack creation
+time. If the `sfn` command is running in a CI environment, where the `CI`
+environment variable is set, then the callback will attempt to use the default
+generic secret backed path of `secret` in a stack specific location.
+
+For local development needs or if this environment variable is undetected the
+vault cubbyhole is used.
+
+The base path is also configurable by using the `:pseudo_parameter_path` in the
+sfn config. This replaces either `cubbyhole` or `secret` in the vault key used
+to store the parameter values. If configured this base path will override any CI
+environment detection and always be honored.
+
+~~~ruby
+...
+vault do
+  pseudo_parameter_path "/secret/aws_secrets"
+end
+~~~
+
+By default 15 character base64 strings are generated using SecureRandom. The
+length can be adjusted by setting `:pseudo_parameter_length` in the config to
+any integer value.
+
+# Known Issues
+
+If the iam_delay for the vault read behavior is not long enough the generated
+credentials will not be available for use and api requests will fail. As noted
+in
+the
+[Vault documentation](https://www.vaultproject.io/docs/secrets/aws/index.html#dynamic-iam-users) this
+is a limitation due to the eventually consistent behavior of IAM credentials. If
+you need the credentials to be immediately available, it is suggested to use
+the
+[STS Callback](http://www.sparkleformation.io/docs/sfn/callbacks.html#aws-assume-role) and
+set thestatus to `disabled` in the vault config section. This will disable the
+vault read behavior but still handles pseudo parameters.
+
 # Info
 
 * Repository: https://github.com/webframp/sfn-vault

lib/sfn-vault.rb

@@ -5,8 +5,9 @@ module SfnVault
   autoload :Platform, 'sfn-vault/platform'
   autoload :Windows, 'sfn-vault/windows'
   autoload :CertificateStore, 'sfn-vault/certificate_store'
+  autoload :Utils, 'sfn-vault/utils'
 end
 
 require 'sfn-vault/version'
 require 'sfn-vault/callback'
-
+require 'sfn-vault/inject'

lib/sfn-vault/callback.rb

@@ -1,10 +1,13 @@
 require 'sfn-parameters'
+require 'securerandom'
+require 'vault'
 
 # Modeled after the Assume Role callback
 module Sfn
   class Callback
     class VaultRead < Callback
 
+
       include SfnVault::Utils
 
       # Cache credentials for re-use to prevent re-generation of temporary
@@ -16,10 +19,95 @@ class VaultRead < Callback
         :aws_secret_access_key
       ]
 
-      # Inject credentials read from vault path
-      # into API provider configuration
+      def template(*args)
+        # search for all parameters of type 'Vault::Generic::Secret'
+        # 1. use the sparkleformation instance to get at the parameter hash,
+        config[:parameters] ||= Smash.new
+        stack = args.first[:sparkle_stack]
+        # 2. find names for things you want,
+        client = vault_client
+        pseudo_parameters(stack).each do |param|
+          param_path = vault_path_name(args, param)
+          ui.debug "Using #{param_path} for saved parameter"
+          # check if already saved in vault
+          # Save the secret unless one already exists at the defined path
+          unless client.logical.read(param_path)
+            ui.info "Vault: No pre-existing value for parameter #{param} saving new secret"
+            client.logical.write(param_path, value: random_secret)
+          end
+          # Read saved secret back from Vault and update parameters config
+          # 3. set param into config
+          config[:parameters][param] = client.logical.read(param_path).data[:value]
+          # 4. update type in template and that should do it
+          stack.compile.parameters.set!(param).type 'String'
+        end
+      end
+
+      # Use SecureRandom to generate a suitable password
+      # Length is configurable by setting `pseudo_parameter_length` in the vault
+      # section of the sfn config
+      #
+      # @return [String] The generated string
+      def random_secret
+        SecureRandom.base64(config.fetch(:vault, :pseudo_parameter_length, 15))
+      end
+
+      # Build the path where generated secrets can be saved in Vault
+      # This will use the value of `:pseudo_parameter_path` from the config if set. If
+      # unset it will attempt to build a type of standardized path based on the
+      # combined value any stack 'Project' tag and Stack name.
+      # Project will fallback to 'SparkleFormation' if unset
+      #
+      # @param args [Array] Array of args passed to the sfn instance
+      # @param parameter [String] Template parameter to save value for in vault
+      # @return [String] String value or stack name if available or default to template name
+      def vault_path_name(args, parameter)
+        pref = config.get(:vault, :pseudo_parameter_path)
+        # If we have a stack name use it, otherwise try to get from env and fallback to just template name
+        stack = args.first[:sparkle_stack]
+        stack_name = args.first[:stack_name].nil? ? ENV.fetch('STACK_NAME', stack.name).to_s : args.first[:stack_name]
+        project = config[:options][:tags].fetch('Project', 'SparkleFormation')
+
+        # When running in a detectable CI environment assume that we have rights to save a generic secret
+        # but honor user preference value if set
+        vault_path = if ci_environment?
+                       # write to vault at generic path
+                       base = pref.nil? ?  "secret" : pref
+                       File.join(base, project, stack_name, parameter)
+                     else
+                       base = pref.nil? ?  "cubbyhole" : pref
+                       # or for local dev use cubbyhole
+                       File.join(base, project, stack_name, parameter)
+                     end
+        vault_path
+      end
+
+      # Lookup all pseudo parameters in the template
+      #
+      # @param stack [SparkleFormation] An instance of the stack template
+      # @param parameter [String] The string value of the pseudo type to lookup
+      # @return [Array] Array of parameter names matching the pseudo type
+      def pseudo_parameters(stack, parameter: 'Vault::Generic::Secret')
+        stack.dump.fetch('Parameters', {}).map{|k,v| k if v['Type'] == parameter}.compact
+      end
+
+
+      # Check if we are running in any detectable CI type environments
+      #
+      # @return [TrueClass, FalseClass]
+      def ci_environment?
+        # check for any ci system env variables
+        return true if ENV['GO_PIPELINE_NAME']
+        return true if ENV['CI']
+        false
+      end
+
       def after_config(*_)
+        # Inject credentials read from configured vault path
+        # into API provider configuration
         # if credentials block contains vault_read_path
+        # TODO: this could be done earlier if at all possible so credentials
+        # struct does not need the aws config
         if(enabled? && config.fetch(:credentials, :vault_read_path))
           load_stored_session
         end
@@ -48,7 +136,7 @@ def enabled?
         config.fetch(:vault, :status, 'enabled').to_s == 'enabled'
       end
 
-      # @return String path
+      # @return [String ]path
       def cache_file
         config.fetch(:vault, :cache_file, '.sfn-vault')
       end

lib/sfn-vault/inject.rb

@@ -0,0 +1,18 @@
+class SparkleFormation
+  module SparkleAttribute
+    module Aws
+
+      # A small helper method for adding the specific named
+      # parameter struct with the custom type
+      def _vault_parameter(vp_name)
+        __t_stringish(vp_name)
+        parameters.set!(vp_name) do
+          no_echo true
+          description "Generated secret automatically stored in Vault"
+          type 'Vault::Generic::Secret'
+        end
+      end
+      alias_method :vault_parameter!, :_vault_parameter
+    end
+  end
+end

lib/sfn-vault/version.rb

@@ -1,3 +1,3 @@
 module SfnVault
-  VERSION = Gem::Version.new('0.1.2')
+  VERSION = Gem::Version.new('0.1.3')
 end

sfn-vault.gemspec

@@ -11,7 +11,7 @@ Gem::Specification.new do |s|
   s.license = 'Apache-2.0'
   s.require_path = 'lib'
   s.add_dependency 'sfn', '>= 3.0', '< 4.0'
-  s.add_dependency 'vault', '~> 0.7.3'
+  s.add_dependency 'vault', '~> 0.9.0'
   s.add_dependency 'ffi', '~> 1.9.14'
   s.add_development_dependency 'pry', '~> 0.10.4'
   s.add_development_dependency 'pry-byebug', '~> 3.4', '>= 3.4.2'