Krane global deploy #574

dturn · 2019-09-27T22:07:16Z

What are you trying to accomplish with this PR?

Part of the krane 1.0 work is refusing to allow krane deploy to deploy global resources. See #522 for more of an explanation.

To allow our users to have a better experience than just kubectl apply we're adding a new command krane global-deploy. This command will not support erb rendering, for that you can pipe the results of krane render. This version does not support pruning or --label-selector thought we would accept the feature in future versions.

How is this accomplished?
Add a krane global-deploy command and a krane/global_deploy_task. The global_deploy_task is a thin wrapper around `kubernetes_deploy/deploy_task.

What could go wrong?

This assumes the approach we want is to share deploy_task with the namespaced version and only override the parts we want.

We could chop up the deploy task and then make the deploy_task and global_deploy_task be more fully featured.

lib/krane/global_deploy_task.rb

dturn · 2019-09-27T22:08:29Z

lib/kubernetes-deploy/cluster_resource_discovery.rb

-      @logger = logger
+    delegate :namespace, :context, :logger, to: :@task_config
+
+    def initialize(task_config:, namespace_tags: [])


Took a pass at using task_config this can be reverted if too much is going on in the pr

What do you think about doing this the other way around: moving all of the stuff in this class into TaskConfig? Everything in here is effectively collected global context about the task's execution environment.

I think this is a reasonable change for a follow-up PR. My original concern was that TaskConfig should actually be the holder of static information, but I don't actually think there would be issues with it having dynamic abilities.

KnVerey · 2019-09-27T23:05:36Z

This version does not support pruning or --label-selector thought we would accept the feature in future versions.

Neither needs to be in the initial PR, but we should chat more about whether they're required for the release. I was actually thinking that label selector might be mandatory for a sane global command, given that you're pretty much guaranteed to not be the only actor modifying the global namespace. As for pruning, I'm not sure whether it should be the default, but our value prop definitely goes down without it, having it off by default would be inconsistent, and making it the default would be a horrifying breaking change if it isn't that way to start. One thing I'm sure about is that if we do have pruning, using a label selector with it should be mandatory.

This assumes the approach we want is to share deploy_task with the namespaced version and only override the parts we want.

Won't that be impossible once we 🔥 the kubernetes-deploy version, since the krane version should not be capable at all of deploying globals?

dturn · 2019-09-27T23:29:41Z

Neither needs to be in the initial PR, but we should chat more about whether they're required for the release. I was actually thinking that label selector might be mandatory for a sane global command, given that you're pretty much guaranteed to not be the only actor modifying the global namespace. As for pruning, I'm not sure whether it should be the default, but our value prop definitely goes down without it, having it off by default would be inconsistent, and making it the default would be a horrifying breaking change if it isn't that way to start. One thing I'm sure about is that if we do have pruning, using a label selector with it should be mandatory.

selector I believe is trivial to add back, so I'll do that here. Pruning is not, so that's going to need to be a second PR either way. I totally agree that pruning requires selector. I think requiring the selector is reasonable. I'll keep thinking about pruning default.

Won't that be impossible once we 🔥 the kubernetes-deploy version, since the krane version should not be capable at all of deploying globals?

I did not explain what I was thinking very well at all:

kubernetes_deploy/deploy_task -> krane/private_deploy_task or some better name that's clear its not to be used directly and possibly always raises an error if used directly. And then have deploy_task and global_deploy_task inherit from the private version and among other things override the bit that raises. I just can't decide if this could work, or is going to be a mess as each grows.

lib/krane/global_deploy_task.rb

KnVerey · 2019-09-30T18:35:41Z

kubernetes_deploy/deploy_task -> krane/private_deploy_task or some better name that's clear its not to be used directly and possibly always raises an error if used directly. And then have deploy_task and global_deploy_task inherit from the private version and among other things override the bit that raises. I just can't decide if this could work, or is going to be a mess as each grows.

I think it could work if you make initialize and or run raise (so that it is an abstract superclass), but I'm not sure that it will be reasonably easy to reason about, as you say. Have you tried the other option yet? My design inkling is that we'll end up with better code overall if we separate them and encapsulate the bits they need to share.

dturn · 2019-10-08T21:13:31Z

lib/krane/cli/global_deploy_command.rb

+
+          deploy.run!(
+            verify_result: options["verify-result"],
+            prune: false,


This will get flipped to true in a follow-up PR #589

dturn · 2019-10-10T22:06:21Z

lib/kubernetes-deploy/deploy_task.rb

      max_watch_seconds: nil, selector: nil, template_paths: [], template_dir: nil, protected_namespaces: nil,
      render_erb: true, allow_globals: false)
      template_dir = File.expand_path(template_dir) if template_dir
      template_paths = (template_paths.map { |path| File.expand_path(path) } << template_dir).compact

      @logger = logger || KubernetesDeploy::FormattedLogger.build(namespace, context)
      @template_sets = TemplateSets.from_dirs_and_files(paths: template_paths, logger: @logger)
-      @task_config = KubernetesDeploy::TaskConfig.new(context, namespace, @logger)
+      @task_config = KubernetesDeploy::TaskConfig.new(context, namespace, @logger, allow_globals && namespace.blank?)


allow_globals && namespace.blank? is the result of having to support the legacy behavior of kubernetes-deploy.

should we also make https://github.com/Shopify/kubernetes-deploy/pull/574/files#diff-4c33bbb177fdf882c41d892f3bde9570R144 be allow_globals && namespace.blank?

I'm not very sure about the @allow_globals and @global_mode in TaskConfig.

But, this PR looks good to me 🙂

@allow_globals at this point is only used to determine if we need to display a warning when you use exe/kubernetes-deploy with a global resource. We should be able to kill it when we delete exe/kubernetes-deploy.

👍 Thanks for the explanation.

dirceu

Makes sense to me 👍

douglas · 2019-10-16T18:48:29Z

Can we wait for #585 before shipping this one? I will help with rebase when the former is merged 🙏

KnVerey

I still feel strongly that DeployTask should not be used as though it is an abstract superclass.

Do you have an issue for actually supporting (via proper resource classes) some of the most common global resource types as part of the launch of this new feature? If we don't have any, the verify result option isn't really doing its work.

KnVerey · 2019-10-16T19:27:23Z

lib/kubernetes-deploy/deploy_task.rb

@@ -122,15 +122,15 @@ def server_version
    # @param protected_namespaces [Array<String>] Array of protected Kubernetes namespaces (defaults
    #   to KubernetesDeploy::DeployTask::PROTECTED_NAMESPACES)
    # @param render_erb [Boolean] Enable ERB rendering
-    def initialize(namespace:, context:, current_sha:, logger: nil, kubectl_instance: nil, bindings: {},
+    def initialize(namespace: nil, context:, current_sha:, logger: nil, kubectl_instance: nil, bindings: {},


It's a big smell to me that a bunch of these arguments (bindings, namespace, protected_namespaces) aren't/shouldn't be supported for GlobalDeployTask, and the allow_globals is always true for one subclass and always false for the other (once we do the rename). It's true that this PR achieves adding the new task with a minimal amount of code. But by doing this way, you're exchanging diff minimization for cognitive overhead for future users (who need to know these oddities about the arguments) and maintainers (who need to simultaneously consider the two types of possible work being executed when changing this task). In your earlier comments you mentioned you planned to create an abstract superclass for the deploy tasks. Did that not work out?

KnVerey · 2019-10-16T19:33:20Z

lib/kubernetes-deploy/deploy_task.rb

@@ -161,14 +161,17 @@ def run(*args)
    # @param prune [Boolean] Enable deletion of resources that do not appear in the template dir
    #
    # @return [nil]
-    def run!(verify_result: true, allow_protected_ns: false, prune: true)
+    def run!(verify_result: true, allow_protected_ns: false, prune: true,
+      task_config_validator: DeployTaskConfigValidator)


The validator selection is an internal detail of the class and should not be configurable by users. This is another symptom of not implementing the abstract superclass (or two separate tasks that leverage common components, or some other option that gives us separation in the code).

KnVerey · 2019-10-16T19:37:21Z

lib/kubernetes-deploy/deploy_task_config_validator.rb

+      end
+      global_names = FormattedLogger.indent_four(global_names.join("\n"))
+
+      if allow_globals


Especially since the big rename PR looks so imminent, I'd suggest waiting to merge the globals command into a version that doesn't need to support the legacy behaviour. If you're using a version where we're giving you a deploy command, you should absolutely not be using the regular deploy command to ship globals.

KnVerey · 2019-10-16T19:38:34Z

lib/kubernetes-deploy/global_deploy_task_config_validator.rb

+    def initialize(protected_namespaces, allow_protected_ns, prune, *arguments)
+      super(*arguments, skip: [:validate_namespace_exists])
+      @protected_namespaces = protected_namespaces
+      @allow_protected_ns = allow_protected_ns


These two vars don't make any sense for the global command

KnVerey · 2019-10-16T19:39:47Z

lib/kubernetes-deploy/global_deploy_task_config_validator.rb

+# frozen_string_literal: true
+module KubernetesDeploy
+  class GlobalDeployTaskConfigValidator < TaskConfigValidator
+    def initialize(protected_namespaces, allow_protected_ns, prune, *arguments)


Should we also validate that the selector is present, or do we get that for free? Specifically, what happens if you pass in an empty string selector, either via CLI or via ruby? It wasn't required on the regular command, so we might not have cared to check.

KnVerey · 2019-10-16T19:55:47Z

lib/kubernetes-deploy/kubernetes_resource.rb

@@ -317,7 +317,7 @@ def debug_message(cause = nil, info_hash = {})
    def fetch_events(kubectl)
      return {} unless exists?
      out, _err, st = kubectl.run("get", "events", "--output=go-template=#{Event.go_template_for(type, name)}",
-        log_failure: false)
+        log_failure: false, use_namespace: !global?)


🤔 It surprises me, but is true, that events can be both namespaced and not. Are we actually formulating a query that works here? Just removing the namespace arg doesn't seem to make kubectl fetch the non-namespaced version:

kubectl get events -v=9 GET https://kubernetes.docker.internal:6443/api/v1/namespaces/default/events?limit=500 200 OK in 19 milliseconds

KnVerey · 2019-10-16T19:59:40Z

lib/kubernetes-deploy/resource_cache.rb

@@ -53,7 +53,7 @@ def fetch_by_kind(kind)
      resource_class = KubernetesResource.class_for_kind(kind)
      output_is_sensitive = resource_class.nil? ? false : resource_class::SENSITIVE_TEMPLATE_CONTENT
      raw_json, _, st = @kubectl.run("get", kind, "--chunk-size=0", attempts: 5, output: "json",
-         output_is_sensitive: output_is_sensitive)
+         output_is_sensitive: output_is_sensitive, use_namespace: !global_mode)


This isn't necessarily an on/off mode for the cache. It needs to be on a per-resource basis. To give a concrete example, Dale's PR needs to fetch nodes for the purposes of validating daemonSet deploys.

KnVerey · 2019-10-16T20:00:35Z

lib/kubernetes-deploy/task_config.rb


-    def initialize(context, namespace, logger = nil)
+    def initialize(context, namespace, logger = nil, global_mode = false)


Is "global mode" anything beyond the absence of the namespace?

KnVerey · 2019-10-16T20:10:33Z

lib/kubernetes-deploy/task_config_validator.rb

      @task_config = task_config
      @kubectl = kubectl
      @kubeclient_builder = kubeclient_builder
      @errors = nil
-      @validations = only || DEFAULT_VALIDATIONS
+      @validations = (only || DEFAULT_VALIDATIONS) - skip


I'm kinda surprised to see that in addition to subclassing, we also need the ability to modify the defaults with both only and skip options. Am I reading correctly that the only option is just used by tests? Could this work:

class TaskConfigValidator def validations DEFAULT_VALIDATIONS end end class SomeValidator < TaskConfigValidator def validations DEFAULT_VALIDATIONS - [:validation_to_skip] # or + something, or just a specific list end end

Then the superclass isn't messing around with constructing the list, when it should in practice be a static property of each subclass.

KnVerey · 2019-10-16T20:13:35Z

lib/kubernetes-deploy/task_config_validator.rb


-      unless st.success?
+      if st.success?
+        @task_config.namespace_definition = JSON.parse(definition, symbolize_names: true)


It isn't intuitive that validate_namespace_exists would have the side-effect of setting a property on an entirely different object. Could we make task_config capable of caching/returning the namespace object, and raising errors we can catch here?

dturn · 2019-10-23T17:36:12Z

Closing for #602

dturn commented Sep 27, 2019

View reviewed changes

lib/krane/global_deploy_task.rb Outdated Show resolved Hide resolved

dturn commented Sep 27, 2019

View reviewed changes

dturn force-pushed the krane-global-deploy branch from addc809 to c36ebcc Compare September 27, 2019 22:10

KnVerey reviewed Sep 30, 2019

View reviewed changes

lib/krane/global_deploy_task.rb Outdated Show resolved Hide resolved

dturn force-pushed the global-deploy branch 3 times, most recently from 1bbc1cb to bd32085 Compare September 30, 2019 22:18

dturn force-pushed the global-deploy branch from 1b3dab0 to d7ba225 Compare October 8, 2019 15:44

dturn force-pushed the krane-global-deploy branch from de4c4a1 to 474fdea Compare October 8, 2019 15:50

dturn force-pushed the global-deploy branch from d7ba225 to 81f4b69 Compare October 8, 2019 15:58

dturn mentioned this pull request Oct 8, 2019

Task config kubectl #584

Merged

dturn changed the base branch from global-deploy to master October 8, 2019 19:18

dturn force-pushed the krane-global-deploy branch 2 times, most recently from aab7866 to ca6b65f Compare October 8, 2019 19:25

dturn changed the base branch from master to task-config-kubectl October 8, 2019 19:25

dturn commented Oct 8, 2019

View reviewed changes

douglas mentioned this pull request Oct 9, 2019

Rename kubernetes-deploy to krane #585

Merged

5 tasks

dturn changed the base branch from task-config-kubectl to master October 10, 2019 18:23

dturn added 2 commits October 10, 2019 11:28

Actually add krane global-deploy

c5bd38c

Don't implicitly use namespace

3f90cde

dturn force-pushed the krane-global-deploy branch 2 times, most recently from 8545bcc to ed87492 Compare October 10, 2019 20:43

Add Deploy and GlobalDeploy TaskConfigValidators

d6d1706

dturn force-pushed the krane-global-deploy branch 2 times, most recently from e47e556 to 6f4e178 Compare October 10, 2019 21:13

move validations around

1e501bf

dturn force-pushed the krane-global-deploy branch from 6f4e178 to 1e501bf Compare October 10, 2019 21:30

dturn commented Oct 10, 2019

View reviewed changes

dturn marked this pull request as ready for review October 10, 2019 22:09

dturn requested review from KnVerey and a team October 10, 2019 22:09

dirceu approved these changes Oct 16, 2019

View reviewed changes

jessie-JNing approved these changes Oct 16, 2019

View reviewed changes

KnVerey suggested changes Oct 16, 2019

View reviewed changes

dturn mentioned this pull request Oct 22, 2019

Global deploy 2 #602

Merged

2 tasks

dturn closed this Oct 23, 2019

douglas deleted the krane-global-deploy branch November 15, 2019 20:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Krane global deploy #574

Krane global deploy #574

dturn commented Sep 27, 2019

dturn Sep 27, 2019

KnVerey Sep 30, 2019

dturn Oct 10, 2019

KnVerey commented Sep 27, 2019

dturn commented Sep 27, 2019 •

edited

Loading

KnVerey commented Sep 30, 2019

dturn Oct 8, 2019 •

edited

Loading

dturn Oct 10, 2019

jessie-JNing Oct 16, 2019 •

edited

Loading

dturn Oct 16, 2019

jessie-JNing Oct 16, 2019

dirceu left a comment

douglas commented Oct 16, 2019

KnVerey left a comment

KnVerey Oct 16, 2019

KnVerey Oct 16, 2019

KnVerey Oct 16, 2019

KnVerey Oct 16, 2019

KnVerey Oct 16, 2019

KnVerey Oct 16, 2019

KnVerey Oct 16, 2019

KnVerey Oct 16, 2019

KnVerey Oct 16, 2019

KnVerey Oct 16, 2019

dturn commented Oct 23, 2019


		def initialize(context, namespace, logger = nil)
		def initialize(context, namespace, logger = nil, global_mode = false)

Krane global deploy #574

Krane global deploy #574

Conversation

dturn commented Sep 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KnVerey commented Sep 27, 2019

dturn commented Sep 27, 2019 • edited Loading

KnVerey commented Sep 30, 2019

dturn Oct 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jessie-JNing Oct 16, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dirceu left a comment

Choose a reason for hiding this comment

douglas commented Oct 16, 2019

KnVerey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dturn commented Oct 23, 2019

dturn commented Sep 27, 2019 •

edited

Loading

dturn Oct 8, 2019 •

edited

Loading

jessie-JNing Oct 16, 2019 •

edited

Loading