kubectl -f-style support for deploy task

CHANGELOG.md

@@ -1,11 +1,14 @@
 ## next
-
 *Enhancements*
+- (alpha) Introduce a new `-f` flag for `kubernetes-deploy`. Allows passing in of multiple directories and/or filenames. Currently only usable by `kubernetes-deploy`, not `kubernetes-render`. [#514](https://github.com/Shopify/kubernetes-deploy/pull/514)
 
 - **[Breaking change]** Added ServiceAccount, PodTemplate, ReplicaSet, Role, and RoleBinding to the prune whitelist.
   * To see what resources may be affected, run `kubectl get $RESOURCE -o jsonpath='{ range .items[*] }{.metadata.namespace}{ "\t" }{.metadata.name}{ "\t" }{.metadata.annotations}{ "\n" }{ end }' --all-namespaces | grep "last-applied"`
   * To exclude a resource from kubernetes-deploy (and kubectl apply) management, remove the last-applied annotation `kubectl annotate $RESOURCE $SECRET_NAME kubectl.kubernetes.io/last-applied-configuration-`.
 
+*Other*
+- `EjsonSecretProvisioner#new` signature has changed. `EjsonSecretProvisioner` objects no longer have access to `kubectl`. Rather, the `ejson-keys` secret used for decryption is now passed in via the calling task. Note that we only consider the `new` and `run(!)` methods of tasks (render, deploy, etc) to have inviolable APIs, so we do not consider this change breaking. [#514](https://github.com/Shopify/kubernetes-deploy/pull/514)
+
 ## 0.26.7
 
 *Other*

CONTRIBUTING.md

@@ -47,7 +47,7 @@ We handle Kubernetes secrets, so it is critical that changes do not cause the co
 
 **Project architecture**
 
-The main interface of this project is our four tasks: `DeployTask`, `RestartTask`, `RunnerTask`, and `RenderTask`. The code in these classes should be high-level abstractions, with implementation details encapsulated in other classes. The public interface of these tasks is a `run` method (and a `run!` equivalent), the body of which should read like a set of phases and steps.
+The main interface of this project is our four tasks: `DeployTask`, `RestartTask`, `RunnerTask`, and `RenderTask`. The code in these classes should be high-level abstractions, with implementation details encapsulated in other classes. The public interface of these tasks is a `run` method (and a `run!` equivalent), the body of which should read like a set of phases and steps. Note that non-task classes are considered internal and we reserve the right to change their API at any time.
 
 An important design principle of the tasks is that they should try to fail fast before touching the cluster if they will not succeed overall. Part of how we achieve this is by separating each task into phases, where the first phase simply gathers information and runs validations to determine what needs to be done and whether that will be able to succeed. In practice, this is the “Initializing <task>” phase for all tasks, plus the “Checking initial resource statuses” phase for DeployTask. Our users should be able to assume that these initial phases never modify their clusters.
 

README.md

@@ -109,6 +109,7 @@ official compatibility chart below.
 Refer to `kubernetes-deploy --help` for the authoritative set of options.
 
 - `--template-dir=DIR`: Used to set the deploy directory. Set `$ENVIRONMENT` instead to use `config/deploy/$ENVIRONMENT`. This flag also supports reading from STDIN. You can do this by using `--template-dir=-`. Example: `cat templates_from_stdin/*.yml | kubernetes-deploy ns ctx --template-dir=-`.
+- (alpha feature) `-f [PATHS]`: Accepts a comma-separated list of directories and/or filenames to specify the set of directories/files that will be deployed (use `-` to read from STDIN). Can be invoked multiple times. Cannot be combined with `--template-dir`. Example: `cat templates_from_stdin/*.yml | kubernetes-deploy ns ctx -f -,path/to/dir,path/to/file.yml`
 - `--bindings=BINDINGS`: Makes additional variables available to your ERB templates. For example, `kubernetes-deploy my-app cluster1 --bindings=color=blue,size=large` will expose `color` and `size`.
 - `--no-prune`: Skips pruning of resources that are no longer in your Kubernetes template set. Not recommended, as it allows your namespace to accumulate cruft that is not reflected in your deploy directory.
 - `--max-watch-seconds=seconds`: Raise a timeout error if it takes longer than _seconds_ for any

exe/kubernetes-deploy

@@ -9,7 +9,8 @@ require 'kubernetes-deploy/label_selector'
 require 'optparse'
 
 skip_wait = false
-template_dir = nil
+template_dir = nil # deprecated
+template_paths = []
 allow_protected_ns = false
 prune = true
 bindings = {}
@@ -26,9 +27,13 @@ ARGV.options do |opts|
   prot_ns = KubernetesDeploy::DeployTask::PROTECTED_NAMESPACES.join(', ')
   opts.on("--allow-protected-ns", "Enable deploys to #{prot_ns}; requires --no-prune") { allow_protected_ns = true }
   opts.on("--no-prune", "Disable deletion of resources that do not appear in the template dir") { prune = false }
-  opts.on("--template-dir=DIR", "Set the template dir (default: config/deploy/$ENVIRONMENT).") do |d|
-    template_dir = d
+  opts.on("--template-dir=DIR", "Set the template dir (default: config/deploy/$ENVIRONMENT).") do |dir|
+    template_dir = dir
   end
+  opts.on("-f [PATHS]", Array, "comma separated list of template directories and/or filenames") do |paths|
+    template_paths += paths
+  end
+
   opts.on("--verbose-log-prefix", "Add [context][namespace] to the log prefix") { verbose_log_prefix = true }
   opts.on("--max-watch-seconds=seconds",
     "Timeout error is raised if it takes longer than the specified number of seconds") do |t|
@@ -55,13 +60,20 @@ namespace = ARGV[0]
 context = ARGV[1]
 logger = KubernetesDeploy::FormattedLogger.build(namespace, context, verbose_prefix: verbose_log_prefix)
 
+# Deprecation path: this can be removed when --template-dir is fully replaced by -f
+if template_dir && !template_paths.empty?
+  logger.error("Error: --template-dir and -f flags cannot be combined")
+  exit(1)
+end
+template_paths = [template_dir] if template_paths.empty? && template_dir
+
 begin
-  KubernetesDeploy::OptionsHelper.with_validated_template_dir(template_dir) do |dir|
+  KubernetesDeploy::OptionsHelper.with_processed_template_paths(template_paths) do |paths|
     runner = KubernetesDeploy::DeployTask.new(
       namespace: namespace,
       context: context,
       current_sha: ENV["REVISION"],
-      template_dir: dir,
+      template_paths: paths,
       bindings: bindings,
       logger: logger,
       max_watch_seconds: max_watch_seconds,

exe/kubernetes-render

@@ -7,27 +7,28 @@ require 'kubernetes-deploy/bindings_parser'
 
 require 'optparse'
 
-template_dir = nil
+template_dir = []
 bindings = {}
 
 ARGV.options do |opts|
   parser = KubernetesDeploy::BindingsParser.new
   opts.on("--bindings=BINDINGS", "Expose additional variables to ERB templates " \
     "(format: k1=v1,k2=v2, JSON string or file (JSON or YAML) path prefixed by '@')") { |b| parser.add(b) }
   opts.on("--template-dir=DIR", "Set the template dir (default: config/deploy/$ENVIRONMENT).") do |d|
-    template_dir = d
+    template_dir = [d]
   end
   opts.parse!
   bindings = parser.parse
 end
 
 templates = ARGV
+logger = KubernetesDeploy::FormattedLogger.build(verbose_prefix: false)
 
 begin
-  KubernetesDeploy::OptionsHelper.with_validated_template_dir(template_dir) do |dir|
+  KubernetesDeploy::OptionsHelper.with_processed_template_paths(template_dir) do |dir|
     runner = KubernetesDeploy::RenderTask.new(
       current_sha: ENV["REVISION"],
-      template_dir: dir,
+      template_dir: dir.first,
       bindings: bindings,
     )
 

lib/kubernetes-deploy/deploy_task.rb

@@ -42,7 +42,7 @@
 require 'kubernetes-deploy/ejson_secret_provisioner'
 require 'kubernetes-deploy/renderer'
 require 'kubernetes-deploy/cluster_resource_discovery'
-require 'kubernetes-deploy/template_discovery'
+require 'kubernetes-deploy/template_sets'
 
 module KubernetesDeploy
   class DeployTask
@@ -107,23 +107,21 @@ def server_version
       kubectl.server_version
     end
 
-    def initialize(namespace:, context:, current_sha:, template_dir:, logger: nil, kubectl_instance: nil, bindings: {},
-      max_watch_seconds: nil, selector: nil)
+    def initialize(namespace:, context:, current_sha:, logger: nil, kubectl_instance: nil, bindings: {},
+      max_watch_seconds: nil, selector: nil, template_paths: [], template_dir: nil)
+      template_dir = File.expand_path(template_dir) if template_dir
+      template_paths = (template_paths.map { |path| File.expand_path(path) } << template_dir).compact
+
       @logger = logger || KubernetesDeploy::FormattedLogger.build(namespace, context)
+      @template_sets = TemplateSets.from_dirs_and_files(paths: template_paths, logger: @logger)
       @task_config = KubernetesDeploy::TaskConfig.new(context, namespace, @logger)
+      @bindings = bindings
       @namespace = namespace
       @namespace_tags = []
       @context = context
       @current_sha = current_sha
-      @template_dir = File.expand_path(template_dir)
       @kubectl = kubectl_instance
       @max_watch_seconds = max_watch_seconds
-      @renderer = KubernetesDeploy::Renderer.new(
-        current_sha: @current_sha,
-        template_dir: @template_dir,
-        logger: @logger,
-        bindings: bindings,
-      )
       @selector = selector
     end
 
@@ -210,15 +208,18 @@ def cluster_resource_discoverer
       )
     end
 
-    def ejson_provisioner
-      @ejson_provisioner ||= EjsonSecretProvisioner.new(
-        namespace: @namespace,
-        context: @context,
-        template_dir: @template_dir,
-        logger: @logger,
-        statsd_tags: @namespace_tags,
-        selector: @selector,
-      )
+    def ejson_provisioners
+      @ejson_provisoners ||= @template_sets.ejson_secrets_files.map do |ejson_secret_file|
+        EjsonSecretProvisioner.new(
+          namespace: @namespace,
+          context: @context,
+          ejson_keys_secret: ejson_keys_secret,
+          ejson_file: ejson_secret_file,
+          logger: @logger,
+          statsd_tags: @namespace_tags,
+          selector: @selector,
+        )
+      end
     end
 
     def deploy_has_priority_resources?(resources)
@@ -273,54 +274,37 @@ def check_initial_status(resources)
     measure_method(:check_initial_status, "initial_status.duration")
 
     def secrets_from_ejson
-      ejson_provisioner.resources
+      ejson_provisioners.flat_map(&:resources)
     end
 
     def discover_resources
+      @logger.info("Discovering resources:")
       resources = []
-      crds = cluster_resource_discoverer.crds.group_by(&:kind)
-      @logger.info("Discovering templates:")
-
-      TemplateDiscovery.new(@template_dir).templates.each do |filename|
-        split_templates(filename) do |r_def|
-          crd = crds[r_def["kind"]]&.first
-          r = KubernetesResource.build(namespace: @namespace, context: @context, logger: @logger, definition: r_def,
-            statsd_tags: @namespace_tags, crd: crd)
-          resources << r
-          @logger.info("  - #{r.id}")
-        end
+      crds_by_kind = cluster_resource_discoverer.crds.group_by(&:kind)
+      @template_sets.with_resource_definitions(render_erb: true,
+          current_sha: @current_sha, bindings: @bindings) do |r_def|
+        crd = crds_by_kind[r_def["kind"]]&.first
+        r = KubernetesResource.build(namespace: @namespace, context: @context, logger: @logger, definition: r_def,
+          statsd_tags: @namespace_tags, crd: crd)
+        resources << r
+        @logger.info("  - #{r.id}")
       end
+
       secrets_from_ejson.each do |secret|
         resources << secret
         @logger.info("  - #{secret.id} (from ejson)")
       end
+
       if (global = resources.select(&:global?).presence)
         @logger.warn("Detected non-namespaced #{'resource'.pluralize(global.count)} which will never be pruned:")
         global.each { |r| @logger.warn("  - #{r.id}") }
       end
       resources.sort
-    end
-    measure_method(:discover_resources)
-
-    def split_templates(filename)
-      file_content = File.read(File.join(@template_dir, filename))
-      rendered_content = @renderer.render_template(filename, file_content)
-      YAML.load_stream(rendered_content, "<rendered> #{filename}") do |doc|
-        next if doc.blank?
-        unless doc.is_a?(Hash)
-          raise InvalidTemplateError.new("Template is not a valid Kubernetes manifest",
-            filename: filename, content: doc)
-        end
-        yield doc
-      end
     rescue InvalidTemplateError => e
-      e.filename ||= filename
       record_invalid_template(err: e.message, filename: e.filename, content: e.content)
       raise FatalDeploymentError, "Failed to render and parse template"
-    rescue Psych::SyntaxError => e
-      record_invalid_template(err: e.message, filename: filename, content: rendered_content)
-      raise FatalDeploymentError, "Failed to render and parse template"
     end
+    measure_method(:discover_resources)
 
     def record_invalid_template(err:, filename:, content: nil)
       debug_msg = ColorizedString.new("Invalid template: #{filename}\n").red
@@ -338,12 +322,7 @@ def record_invalid_template(err:, filename:, content: nil)
     def validate_configuration(allow_protected_ns:, prune:)
       errors = []
       errors += kubeclient_builder.validate_config_files
-
-      if !File.directory?(@template_dir)
-        errors << "Template directory `#{@template_dir}` doesn't exist"
-      elsif Dir.entries(@template_dir).none? { |file| file =~ /(\.ya?ml(\.erb)?)$|(secrets\.ejson)$/ }
-        errors << "`#{@template_dir}` doesn't contain valid templates (secrets.ejson or postfix .yml, .yml.erb)"
-      end
+      errors += @template_sets.validate
 
       if @namespace.blank?
         errors << "Namespace must be specified"
@@ -572,8 +551,7 @@ def namespace_definition
 
     # make sure to never prune the ejson-keys secret
     def confirm_ejson_keys_not_prunable
-      secret = ejson_provisioner.ejson_keys_secret
-      return unless secret.dig("metadata", "annotations", KubernetesResource::LAST_APPLIED_ANNOTATION)
+      return unless ejson_keys_secret.dig("metadata", "annotations", KubernetesResource::LAST_APPLIED_ANNOTATION)
 
       @logger.error("Deploy cannot proceed because protected resource " \
         "Secret/#{EjsonSecretProvisioner::EJSON_KEYS_SECRET} would be pruned.")
@@ -592,6 +570,17 @@ def kubectl
       @kubectl ||= Kubectl.new(namespace: @namespace, context: @context, logger: @logger, log_failure_by_default: true)
     end
 
+    def ejson_keys_secret
+      @ejson_keys_secret ||= begin
+        out, err, st = kubectl.run("get", "secret", EjsonSecretProvisioner::EJSON_KEYS_SECRET, output: "json",
+          raise_if_not_found: true, attempts: 3, output_is_sensitive: true, log_failure: true)
+        unless st.success?
+          raise EjsonSecretError, "Error retrieving Secret/#{EjsonSecretProvisioner::EJSON_KEYS_SECRET}: #{err}"
+        end
+        JSON.parse(out)
+      end
+    end
+
     def statsd_tags
       %W(namespace:#{@namespace} sha:#{@current_sha} context:#{@context}) | @namespace_tags
     end

lib/kubernetes-deploy/ejson_secret_provisioner.rb

@@ -17,10 +17,11 @@ class EjsonSecretProvisioner
     EJSON_SECRETS_FILE = "secrets.ejson"
     EJSON_KEYS_SECRET = "ejson-keys"
 
-    def initialize(namespace:, context:, template_dir:, logger:, statsd_tags:, selector: nil)
+    def initialize(namespace:, context:, ejson_keys_secret:, ejson_file:, logger:, statsd_tags:, selector: nil)
       @namespace = namespace
       @context = context
-      @ejson_file = "#{template_dir}/#{EJSON_SECRETS_FILE}"
+      @ejson_keys_secret = ejson_keys_secret
+      @ejson_file = ejson_file
       @logger = logger
       @statsd_tags = statsd_tags
       @selector = selector
@@ -37,20 +38,12 @@ def resources
       @resources ||= build_secrets
     end
 
-    def ejson_keys_secret
-      @ejson_keys_secret ||= begin
-        out, err, st = @kubectl.run("get", "secret", EJSON_KEYS_SECRET, output: "json",
-          raise_if_not_found: true, attempts: 3, output_is_sensitive: true, log_failure: true)
-        unless st.success?
-          raise EjsonSecretError, "Error retrieving Secret/#{EJSON_KEYS_SECRET}: #{err}"
-        end
-        JSON.parse(out)
-      end
-    end
-
     private
 
     def build_secrets
+      unless @ejson_keys_secret
+        raise EjsonSecretError, "Secret #{EJSON_KEYS_SECRET} not provided, cannot decrypt secrets"
+      end
       return [] unless File.exist?(@ejson_file)
       with_decrypted_ejson do |decrypted|
         secrets = decrypted[EJSON_SECRET_KEY]
@@ -153,7 +146,7 @@ def decrypt_ejson(key_dir)
     end
 
     def fetch_private_key_from_secret
-      encoded_private_key = ejson_keys_secret["data"][public_key]
+      encoded_private_key = @ejson_keys_secret["data"][public_key]
       unless encoded_private_key
         raise EjsonSecretError, "Private key for #{public_key} not found in #{EJSON_KEYS_SECRET} secret"
       end

lib/kubernetes-deploy/options_helper.rb

@@ -6,30 +6,43 @@ class OptionsError < StandardError; end
 
     STDIN_TEMP_FILE = "from_stdin.yml.erb"
     class << self
-      def with_validated_template_dir(template_dir)
-        if template_dir == '-'
+      def with_processed_template_paths(template_paths)
+        validated_paths = []
+        if template_paths.empty?
+          validated_paths << default_template_dir
+        else
+          template_paths.uniq!
+          template_paths.each do |template_path|
+            next if template_path == '-'
+            validated_paths << template_path
+          end
+        end
+
+        if template_paths.include?("-")
           Dir.mktmpdir("kubernetes-deploy") do |dir|
             template_dir_from_stdin(temp_dir: dir)
-            yield dir
+            validated_paths << dir
+            yield validated_paths
           end
-        elsif template_dir
-          yield template_dir
         else
-          yield default_template_dir(template_dir)
+          yield validated_paths
         end
       end
 
       private
 
-      def default_template_dir(template_dir)
-        if ENV.key?("ENVIRONMENT")
-          template_dir = File.join("config", "deploy", ENV['ENVIRONMENT'])
+      def default_template_dir
+        template_dir = if ENV.key?("ENVIRONMENT")
+          File.join("config", "deploy", ENV['ENVIRONMENT'])
         end
 
-        if !template_dir || template_dir.empty?
+        unless template_dir
           raise OptionsError, "Template directory is unknown. " \
-            "Either specify --template-dir argument or set $ENVIRONMENT to use config/deploy/$ENVIRONMENT " \
-            "as a default path."
+          "Either specify --template-dir argument or set $ENVIRONMENT to use config/deploy/$ENVIRONMENT " \
+          "as a default path."
+        end
+        unless Dir.exist?(template_dir)
+          raise OptionsError, "Template directory #{template_dir} does not exist."
         end
 
         template_dir