diff --git a/CHANGELOG.md b/CHANGELOG.md index 2cb6f1639..698c1bc83 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ ## next +*Features* +- Add support for specifying pass/fail conditions of Custom Resources ([#376](https://github.com/Shopify/kubernetes-deploy/pull/376)). +- Add support for custom timeouts for Custom Resources([#376](https://github.com/Shopify/kubernetes-deploy/pull/376)) + *Enhancements* - Officially support Kubernetes 1.13 ([#409](https://github.com/Shopify/kubernetes-deploy/pull/409)) diff --git a/README.md b/README.md index 697717ead..5b267d2e3 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,7 @@ This repo also includes related tools for [running tasks](#kubernetes-run) and [ * [Customizing behaviour with annotations](#customizing-behaviour-with-annotations) * [Running tasks at the beginning of a deploy](#running-tasks-at-the-beginning-of-a-deploy) * [Deploying Kubernetes secrets (from EJSON)](#deploying-kubernetes-secrets-from-ejson) + * [Deploying custom resources](#deploying-custom-resources) **KUBERNETES-RESTART** * [Usage](#usage-1) @@ -312,7 +313,95 @@ Since their data is only base64 encoded, Kubernetes secrets should not be commit } ``` +### Deploying custom resources +By default, kubernetes-deploy does not check the status of custom resources; it simply assumes that they deployed successfully. In order to meaningfully monitor the rollout of custom resources, kubernetes-deploy supports configuring pass/fail conditions using annotations on CustomResourceDefinitions (CRDs). + +>Note: +This feature is only available on clusters running Kubernetes 1.11+ since it relies on the `metadata.generation` field being updated when custom resource specs are changed. + +*Requirements:* + +* The custom resource must expose a `status` subresource with an `observedGeneration` field. +* The `kubernetes-deploy.shopify.io/instance-rollout-conditions` annotation must be present on the CRD that defines the custom resource. +* (optional) The `kubernetes-deploy.shopify.io/instance-timeout` annotation can be added to the CRD that defines the custom resource to override the global default timeout for all instances of that resource. This annotation can use ISO8601 format or unprefixed ISO8601 time components (e.g. '1H', '60S'). + +#### Specifying pass/fail conditions + +The presence of a valid `kubernetes-deploy.shopify.io/instance-rollout-conditions` annotation on a CRD will cause kubernetes-deploy to monitor the rollout of all instances of that custom resource. Its value can either be `"true"` (giving you the defaults described in the next section) or a valid JSON string with the following format: +``` +'{ + "success_conditions": [ + { "path": , "value": } + ... more success conditions + ], + "failure_conditions": [ + { "path": , "value": } + ... more failure conditions + ] +}' +``` + +For all conditions, `path` must be a valid JsonPath expression that points to a field in the custom resource's status. `value` is the value that must be present at `path` in order to fulfill a condition. For a deployment to be successful, _all_ `success_conditions` must be fulfilled. Conversely, the deploy will be marked as failed if _any one of_ `failure_conditions` is fulfilled. `success_conditions` are mandatory, but `failure_conditions` can be omitted (the resource will simply time out if it never reaches a successful state). + +In addition to `path` and `value`, a failure condition can also contain `error_msg_path` or `custom_error_msg`. `error_msg_path` is a JsonPath expression that points to a field you want to surface when a failure condition is fulfilled. For example, a status condition may expose a `message` field that contains a description of the problem it encountered. `custom_error_msg` is a string that can be used if your custom resource doesn't contain sufficient information to warrant using `error_msg_path`. Note that `custom_error_msg` has higher precedence than `error_msg_path` so it will be used in favor of `error_msg_path` when both fields are present. + +**Warning:** + +You **must** ensure that your custom resource controller sets `.status.observedGeneration` to match the observed `.metadata.generation` of the monitored resource once its sync is complete. If this does not happen, kubernetes-deploy will not check success or failure conditions and the deploy will time out. + +#### Example + +As an example, the following is the default configuration that will be used if you set `kubernetes-deploy.shopify.io/instance-rollout-conditions: "true"` on the CRD that defines the custom resources you wish to monitor: + +``` +'{ + "success_conditions": [ + { + "path": "$.status.conditions[?(@.type == \"Ready\")].status", + "value": "True", + }, + ], + "failure_conditions": [ + { + "path": '$.status.conditions[?(@.type == \"Failed\")].status', + "value": "True", + "error_msg_path": '$.status.conditions[?(@.type == \"Failed\")].message', + }, + ], +}' +``` + +The paths defined here are based on the [typical status properties](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#typical-status-properties) as defined by the Kubernetes community. It expects the `status` subresource to contain a `conditions` array whose entries minimally specify `type`, `status`, and `message` fields. + +You can see how these conditions relate to the following resource: + +``` +apiVersion: stable.shopify.io/v1 +kind: Example +metadata: + generation: 2 + name: example + namespace: namespace +spec: + ... +status: + observedGeneration: 2 + conditions: + - type: "Ready" + status: "False" + reason: "exampleNotReady" + message: "resource is not ready" + - type: "Failed" + status: "True" + reason: "exampleFailed" + message: "resource is failed" +``` + +- `observedGeneration == metadata.generation`, so kubernetes-deploy will check this resource's success and failure conditions. +- Since `$.status.conditions[?(@.type == "Ready")].status == "False"`, the resource is not considered successful yet. +- `$.status.conditions[?(@.type == "Failed")].status == "True"` means that a failure condition has been fulfilled and the resource is considered failed. +- Since `error_msg_path` is specified, kubernetes-deploy will log the contents of `$.status.conditions[?(@.type == "Failed")].message`, which in this case is: `resource is failed`. # kubernetes-restart diff --git a/kubernetes-deploy.gemspec b/kubernetes-deploy.gemspec index 29ab970bd..8fbcb1860 100644 --- a/kubernetes-deploy.gemspec +++ b/kubernetes-deploy.gemspec @@ -31,6 +31,7 @@ Gem::Specification.new do |spec| spec.add_dependency("statsd-instrument", '~> 2.3', '>= 2.3.2') spec.add_dependency("oj", "~> 3.7") spec.add_dependency("concurrent-ruby", "~> 1.1") + spec.add_dependency("jsonpath", "~> 0.9.6") spec.add_development_dependency("bundler") spec.add_development_dependency("rake", "~> 10.0") diff --git a/lib/kubernetes-deploy/deploy_task.rb b/lib/kubernetes-deploy/deploy_task.rb index 5ce014194..9fdc8b78e 100644 --- a/lib/kubernetes-deploy/deploy_task.rb +++ b/lib/kubernetes-deploy/deploy_task.rb @@ -6,6 +6,7 @@ require 'fileutils' require 'kubernetes-deploy/kubernetes_resource' %w( + custom_resource cloudsql config_map deployment @@ -24,7 +25,6 @@ elasticsearch statefulservice topic - bucket stateful_set cron_job job @@ -46,19 +46,6 @@ class DeployTask include KubeclientBuilder extend KubernetesDeploy::StatsD::MeasureMethods - PREDEPLOY_SEQUENCE = %w( - ResourceQuota - Cloudsql - Redis - Memcached - ConfigMap - PersistentVolumeClaim - ServiceAccount - Role - RoleBinding - Pod - ) - PROTECTED_NAMESPACES = %w( default kube-system @@ -73,6 +60,22 @@ class DeployTask # extensions/v1beta1/ReplicaSet -- managed by deployments # core/v1/Secret -- should not committed / managed by shipit + def predeploy_sequence + before_crs = %w( + ResourceQuota + ) + after_crs = %w( + ConfigMap + PersistentVolumeClaim + ServiceAccount + Role + RoleBinding + Pod + ) + + before_crs + cluster_resource_discoverer.crds.map(&:kind) + after_crs + end + def prune_whitelist wl = %w( core/v1/ConfigMap @@ -194,11 +197,11 @@ def cluster_resource_discoverer end def deploy_has_priority_resources?(resources) - resources.any? { |r| PREDEPLOY_SEQUENCE.include?(r.type) } + resources.any? { |r| predeploy_sequence.include?(r.type) } end def predeploy_priority_resources(resource_list) - PREDEPLOY_SEQUENCE.each do |resource_type| + predeploy_sequence.each do |resource_type| matching_resources = resource_list.select { |r| r.type == resource_type } next if matching_resources.empty? deploy_resources(matching_resources, verify: true, record_summary: false) @@ -254,12 +257,14 @@ def create_ejson_secrets(prune) def discover_resources resources = [] + crds = cluster_resource_discoverer.crds.group_by(&:kind) @logger.info("Discovering templates:") TemplateDiscovery.new(@template_dir).templates.each do |filename| split_templates(filename) do |r_def| - r = KubernetesResource.build(namespace: @namespace, context: @context, logger: @logger, - definition: r_def, statsd_tags: @namespace_tags) + crd = crds[r_def["kind"]]&.first + r = KubernetesResource.build(namespace: @namespace, context: @context, logger: @logger, definition: r_def, + statsd_tags: @namespace_tags, crd: crd) resources << r @logger.info(" - #{r.id}") end diff --git a/lib/kubernetes-deploy/kubernetes_resource.rb b/lib/kubernetes-deploy/kubernetes_resource.rb index 774ea3a79..3bb7dbead 100644 --- a/lib/kubernetes-deploy/kubernetes_resource.rb +++ b/lib/kubernetes-deploy/kubernetes_resource.rb @@ -31,13 +31,12 @@ class KubernetesResource TIMEOUT_OVERRIDE_ANNOTATION = "kubernetes-deploy.shopify.io/timeout-override" class << self - def build(namespace:, context:, definition:, logger:, statsd_tags:) + def build(namespace:, context:, definition:, logger:, statsd_tags:, crd: nil) opts = { namespace: namespace, context: context, definition: definition, logger: logger, statsd_tags: statsd_tags } if definition["kind"].blank? raise InvalidTemplateError.new("Template missing 'Kind'", content: definition.to_yaml) end - begin if KubernetesDeploy.const_defined?(definition["kind"]) klass = KubernetesDeploy.const_get(definition["kind"]) @@ -45,10 +44,13 @@ def build(namespace:, context:, definition:, logger:, statsd_tags:) end rescue NameError end - - inst = new(**opts) - inst.type = definition["kind"] - inst + if crd + CustomResource.new(crd: crd, **opts) + else + inst = new(**opts) + inst.type = definition["kind"] + inst + end end def timeout @@ -167,13 +169,13 @@ def exists? def current_generation return -1 unless exists? # must be different default than observed_generation - @instance_data["metadata"]["generation"] + @instance_data.dig("metadata", "generation") end def observed_generation return -2 unless exists? # populating this is a best practice, but not all controllers actually do it - @instance_data["status"]["observedGeneration"] + @instance_data.dig('status', 'observedGeneration') end def status diff --git a/lib/kubernetes-deploy/kubernetes_resource/bucket.rb b/lib/kubernetes-deploy/kubernetes_resource/bucket.rb deleted file mode 100644 index aebc57e38..000000000 --- a/lib/kubernetes-deploy/kubernetes_resource/bucket.rb +++ /dev/null @@ -1,22 +0,0 @@ -# frozen_string_literal: true -module KubernetesDeploy - class Bucket < KubernetesResource - def deploy_succeeded? - return false unless deploy_started? - - unless @success_assumption_warning_shown - @logger.warn("Don't know how to monitor resources of type #{type}. Assuming #{id} deployed successfully.") - @success_assumption_warning_shown = true - end - true - end - - def status - exists? ? "Available" : "Unknown" - end - - def deploy_failed? - false - end - end -end diff --git a/lib/kubernetes-deploy/kubernetes_resource/custom_resource.rb b/lib/kubernetes-deploy/kubernetes_resource/custom_resource.rb new file mode 100644 index 000000000..ad263509d --- /dev/null +++ b/lib/kubernetes-deploy/kubernetes_resource/custom_resource.rb @@ -0,0 +1,87 @@ +# frozen_string_literal: true +require 'jsonpath' + +module KubernetesDeploy + class CustomResource < KubernetesResource + TIMEOUT_MESSAGE_DIFFERENT_GENERATIONS = <<~MSG + This resource's status could not be used to determine rollout success because it is not up-to-date + (.metadata.generation != .status.observedGeneration). + MSG + + def initialize(namespace:, context:, definition:, logger:, statsd_tags: [], crd:) + super(namespace: namespace, context: context, definition: definition, + logger: logger, statsd_tags: statsd_tags) + @crd = crd + end + + def timeout + timeout_override || @crd.timeout_for_instance || TIMEOUT + end + + def deploy_succeeded? + return super unless rollout_conditions + return false unless observed_generation == current_generation + + rollout_conditions.rollout_successful?(@instance_data) + end + + def deploy_failed? + return super unless rollout_conditions + return false unless observed_generation == current_generation + + rollout_conditions.rollout_failed?(@instance_data) + end + + def failure_message + return super unless rollout_conditions + messages = rollout_conditions.failure_messages(@instance_data) + messages.join("\n") if messages.present? + end + + def timeout_message + if rollout_conditions && current_generation != observed_generation + TIMEOUT_MESSAGE_DIFFERENT_GENERATIONS + else + super + end + end + + def status + if !exists? || rollout_conditions.nil? + super + elsif deploy_succeeded? + "Healthy" + elsif deploy_failed? + "Unhealthy" + else + "Unknown" + end + end + + def type + kind + end + + def validate_definition(kubectl) + super + + @crd.validate_rollout_conditions + rescue RolloutConditionsError => e + @validation_errors << "The CRD that specifies this resource is using invalid rollout conditions. " \ + "Kubernetes-deploy will not be able to continue until those rollout conditions are fixed.\n" \ + "Rollout conditions can be found on the CRD that defines this resource (#{@crd.name}), " \ + "under the annotation #{CustomResourceDefinition::ROLLOUT_CONDITIONS_ANNOTATION}.\n" \ + "Validation failed with: #{e}" + end + + private + + def kind + @definition["kind"] + end + + def rollout_conditions + @crd.rollout_conditions + end + end +end diff --git a/lib/kubernetes-deploy/kubernetes_resource/custom_resource_definition.rb b/lib/kubernetes-deploy/kubernetes_resource/custom_resource_definition.rb index 017712602..62c5e6829 100644 --- a/lib/kubernetes-deploy/kubernetes_resource/custom_resource_definition.rb +++ b/lib/kubernetes-deploy/kubernetes_resource/custom_resource_definition.rb @@ -1,7 +1,11 @@ # frozen_string_literal: true +require 'kubernetes-deploy/rollout_conditions' + module KubernetesDeploy class CustomResourceDefinition < KubernetesResource TIMEOUT = 2.minutes + ROLLOUT_CONDITIONS_ANNOTATION = "kubernetes-deploy.shopify.io/instance-rollout-conditions" + TIMEOUT_FOR_INSTANCE_ANNOTATION = "kubernetes-deploy.shopify.io/instance-timeout" GLOBAL = true def deploy_succeeded? @@ -16,6 +20,13 @@ def timeout_message "The names this CRD is attempting to register were neither accepted nor rejected in time" end + def timeout_for_instance + timeout = @definition.dig("metadata", "annotations", TIMEOUT_FOR_INSTANCE_ANNOTATION) + DurationParser.new(timeout).parse!.to_i + rescue DurationParser::ParsingError + nil + end + def status if !exists? super @@ -36,11 +47,42 @@ def kind @definition.dig("spec", "names", "kind") end + def name + @definition.dig("metadata", "name") + end + def prunable? prunable = @definition.dig("metadata", "annotations", "kubernetes-deploy.shopify.io/prunable") prunable == "true" end + def rollout_conditions + return @rollout_conditions if defined?(@rollout_conditions) + + @rollout_conditions = if rollout_conditions_annotation + RolloutConditions.from_annotation(rollout_conditions_annotation) + end + rescue RolloutConditionsError + @rollout_conditions = nil + end + + def validate_definition(_) + super + + validate_rollout_conditions + rescue RolloutConditionsError => e + @validation_errors << "Annotation #{ROLLOUT_CONDITIONS_ANNOTATION} on #{name} is invalid: #{e}" + end + + def validate_rollout_conditions + if rollout_conditions_annotation && @rollout_conditions_validated.nil? + conditions = RolloutConditions.from_annotation(rollout_conditions_annotation) + conditions.validate! + end + + @rollout_conditions_validated = true + end + private def names_accepted_condition @@ -51,5 +93,9 @@ def names_accepted_condition def names_accepted_status names_accepted_condition["status"] end + + def rollout_conditions_annotation + @definition.dig("metadata", "annotations", ROLLOUT_CONDITIONS_ANNOTATION) + end end end diff --git a/lib/kubernetes-deploy/rollout_conditions.rb b/lib/kubernetes-deploy/rollout_conditions.rb new file mode 100644 index 000000000..c254f6081 --- /dev/null +++ b/lib/kubernetes-deploy/rollout_conditions.rb @@ -0,0 +1,103 @@ +# frozen_string_literal: true +module KubernetesDeploy + class RolloutConditionsError < StandardError + end + + class RolloutConditions + VALID_FAILURE_CONDITION_KEYS = [:path, :value, :error_msg_path, :custom_error_msg] + VALID_SUCCESS_CONDITION_KEYS = [:path, :value] + + class << self + def from_annotation(conditions_string) + return new(default_conditions) if conditions_string.downcase.strip == "true" + + conditions = JSON.parse(conditions_string).slice('success_conditions', 'failure_conditions') + conditions.deep_symbolize_keys! + + # Create JsonPath objects + conditions[:success_conditions]&.each do |query| + query.slice!(*VALID_SUCCESS_CONDITION_KEYS) + query[:path] = JsonPath.new(query[:path]) if query.key?(:path) + end + conditions[:failure_conditions]&.each do |query| + query.slice!(*VALID_FAILURE_CONDITION_KEYS) + query[:path] = JsonPath.new(query[:path]) if query.key?(:path) + query[:error_msg_path] = JsonPath.new(query[:error_msg_path]) if query.key?(:error_msg_path) + end + + new(conditions) + rescue JSON::ParserError => e + raise RolloutConditionsError, "Rollout conditions are not valid JSON: #{e}" + rescue StandardError => e + raise RolloutConditionsError, + "Error parsing rollout conditions. " \ + "This is most likely caused by an invalid JsonPath expression. Failed with: #{e}" + end + + def default_conditions + { + success_conditions: [ + { + path: JsonPath.new('$.status.conditions[?(@.type == "Ready")].status'), + value: "True", + }, + ], + failure_conditions: [ + { + path: JsonPath.new('$.status.conditions[?(@.type == "Failed")].status'), + value: "True", + error_msg_path: JsonPath.new('$.status.conditions[?(@.type == "Failed")].message'), + }, + ], + } + end + end + + def initialize(conditions) + @success_conditions = conditions.fetch(:success_conditions, []) + @failure_conditions = conditions.fetch(:failure_conditions, []) + end + + def rollout_successful?(instance_data) + @success_conditions.all? do |query| + query[:path].first(instance_data) == query[:value] + end + end + + def rollout_failed?(instance_data) + @failure_conditions.any? do |query| + query[:path].first(instance_data) == query[:value] + end + end + + def failure_messages(instance_data) + @failure_conditions.map do |query| + next unless query[:path].first(instance_data) == query[:value] + query[:custom_error_msg].presence || query[:error_msg_path]&.first(instance_data) + end.compact + end + + def validate! + errors = validate_conditions(@success_conditions, 'success_conditions') + errors += validate_conditions(@failure_conditions, 'failure_conditions', required: false) + raise RolloutConditionsError, errors.join(", ") unless errors.empty? + end + + private + + def validate_conditions(conditions, source_key, required: true) + return [] unless conditions.present? || required + errors = [] + errors << "#{source_key} should be Array but found #{conditions.class}" unless conditions.is_a?(Array) + return errors if errors.present? + errors << "#{source_key} must contain at least one entry" if conditions.empty? + return errors if errors.present? + + conditions.each do |query| + missing = [:path, :value].reject { |k| query.key?(k) } + errors << "Missing required key(s) for #{source_key.singularize}: #{missing}" if missing.present? + end + errors + end + end +end diff --git a/test/fixtures/crd/redis.yml b/test/fixtures/crd/redis.yml new file mode 100644 index 000000000..d7586e3b7 --- /dev/null +++ b/test/fixtures/crd/redis.yml @@ -0,0 +1,12 @@ +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: redises.stable.shopify.io +spec: + group: stable.shopify.io + names: + kind: Redis + listKind: RedisList + plural: redises + singular: redis + scope: Namespaced diff --git a/test/fixtures/crd/with_custom_conditions.yml b/test/fixtures/crd/with_custom_conditions.yml new file mode 100644 index 000000000..d31130250 --- /dev/null +++ b/test/fixtures/crd/with_custom_conditions.yml @@ -0,0 +1,38 @@ +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: customizeds.stable.example.io + annotations: + kubernetes-deploy.shopify.io/instance-rollout-conditions: '{ + "success_conditions": [ + { + "path": "$.status.condition", + "value":"success_value" + }, + { + "path":"$.status.test_field", + "value":"success_value" + } + ], + "failure_conditions": [ + { + "path":"$.status.condition", + "value":"failure_value", + "custom_error_msg":"test custom error message" + }, + { + "path":"$.status.test_field", + "value":"failure_value", + "error_msg_path":"$.status.error_msg" + } + ] + }' +spec: + group: stable.example.io + names: + kind: Customized + listKind: CustomizedList + plural: customizeds + singular: customized + scope: Namespaced + version: v1 diff --git a/test/fixtures/crd/with_custom_conditions_cr.yml b/test/fixtures/crd/with_custom_conditions_cr.yml new file mode 100644 index 000000000..22e4ab830 --- /dev/null +++ b/test/fixtures/crd/with_custom_conditions_cr.yml @@ -0,0 +1,5 @@ +--- +apiVersion: "stable.example.io/v1" +kind: Customized +metadata: + name: with-customized-params diff --git a/test/fixtures/crd/with_custom_conditions_cr2.yml b/test/fixtures/crd/with_custom_conditions_cr2.yml new file mode 100644 index 000000000..2a094521f --- /dev/null +++ b/test/fixtures/crd/with_custom_conditions_cr2.yml @@ -0,0 +1,5 @@ +--- +apiVersion: "stable.example.io/v1" +kind: Customized +metadata: + name: with-customized-params-2 diff --git a/test/fixtures/crd/with_default_conditions.yml b/test/fixtures/crd/with_default_conditions.yml new file mode 100644 index 000000000..3dad17882 --- /dev/null +++ b/test/fixtures/crd/with_default_conditions.yml @@ -0,0 +1,15 @@ +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: parameterizeds.stable.example.io + annotations: + kubernetes-deploy.shopify.io/instance-rollout-conditions: "true" +spec: + group: stable.example.io + names: + kind: Parameterized + listKind: ParameterizedList + plural: parameterizeds + singular: parameterized + scope: Namespaced + version: v1 diff --git a/test/fixtures/crd/with_default_conditions_cr.yml b/test/fixtures/crd/with_default_conditions_cr.yml new file mode 100644 index 000000000..2188985f6 --- /dev/null +++ b/test/fixtures/crd/with_default_conditions_cr.yml @@ -0,0 +1,5 @@ +--- +apiVersion: "stable.example.io/v1" +kind: Parameterized +metadata: + name: with-default-params diff --git a/test/fixtures/for_unit_tests/crd_test.yml b/test/fixtures/for_unit_tests/crd_test.yml new file mode 100644 index 000000000..f4b2248aa --- /dev/null +++ b/test/fixtures/for_unit_tests/crd_test.yml @@ -0,0 +1,13 @@ +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: unittests.stable.example.io +spec: + group: stable.example.io + names: + kind: UnitTest + listKind: UnitTestList + plural: unittests + singular: unittest + scope: Namespaced + version: v1 diff --git a/test/integration-serial/serial_deploy_test.rb b/test/integration-serial/serial_deploy_test.rb index d115a3eaf..74c602432 100644 --- a/test/integration-serial/serial_deploy_test.rb +++ b/test/integration-serial/serial_deploy_test.rb @@ -235,6 +235,153 @@ def test_all_expected_statsd_metrics_emitted_with_essential_tags end end + def test_cr_deploys_without_rollout_conditions_when_none_present + assert_deploy_success(deploy_fixtures("crd", subset: %w(widgets.yml))) + assert_deploy_success(deploy_fixtures("crd", subset: %w(widgets_cr.yml))) + assert_logs_match_all([ + "Don't know how to monitor resources of type Widget. Assuming Widget/my-first-widget deployed successfully.", + %r{Widget/my-first-widget\s+Exists}, + ]) + ensure + wait_for_all_crd_deletion + end + + def test_cr_success_with_default_rollout_conditions + assert_deploy_success(deploy_fixtures("crd", subset: ["with_default_conditions.yml"])) + success_conditions = { + "status" => { + "observedGeneration" => 1, + "conditions" => [ + { + "type" => "Ready", + "reason" => "test", + "message" => "test", + "status" => "True", + }, + ], + }, + } + + result = deploy_fixtures("crd", subset: ["with_default_conditions_cr.yml"]) do |resource| + cr = resource["with_default_conditions_cr.yml"]["Parameterized"].first + cr.merge!(success_conditions) + end + assert_deploy_success(result) + assert_logs_match_all([ + %r{Successfully deployed in .*: Parameterized\/with-default-params}, + %r{Parameterized/with-default-params\s+Healthy}, + ]) + ensure + wait_for_all_crd_deletion + end + + def test_cr_failure_with_default_rollout_conditions + assert_deploy_success(deploy_fixtures("crd", subset: ["with_default_conditions.yml"])) + failure_conditions = { + "status" => { + "observedGeneration" => 1, + "conditions" => [ + { + "type" => "Failed", + "reason" => "test", + "message" => "custom resource rollout failed", + "status" => "True", + }, + ], + }, + } + + result = deploy_fixtures("crd", subset: ["with_default_conditions_cr.yml"]) do |resource| + cr = resource["with_default_conditions_cr.yml"]["Parameterized"].first + cr.merge!(failure_conditions) + end + assert_deploy_failure(result) + + assert_logs_match_all([ + "Parameterized/with-default-params: FAILED", + "custom resource rollout failed", + "Final status: Unhealthy", + ], in_order: true) + ensure + wait_for_all_crd_deletion + end + + def test_cr_success_with_arbitrary_rollout_conditions + assert_deploy_success(deploy_fixtures("crd", subset: ["with_custom_conditions.yml"])) + + success_conditions = { + "spec" => {}, + "status" => { + "observedGeneration" => 1, + "test_field" => "success_value", + "condition" => "success_value", + }, + } + + result = deploy_fixtures("crd", subset: ["with_custom_conditions_cr.yml"]) do |resource| + cr = resource["with_custom_conditions_cr.yml"]["Customized"].first + cr.merge!(success_conditions) + end + assert_deploy_success(result) + assert_logs_match_all([ + %r{Successfully deployed in .*: Customized\/with-customized-params}, + ]) + ensure + wait_for_all_crd_deletion + end + + def test_cr_failure_with_arbitrary_rollout_conditions + assert_deploy_success(deploy_fixtures("crd", subset: ["with_custom_conditions.yml"])) + cr = load_fixtures("crd", ["with_custom_conditions_cr.yml"]) + failure_conditions = { + "spec" => {}, + "status" => { + "test_field" => "failure_value", + "error_msg" => "test error message jsonpath", + "observedGeneration" => 1, + "condition" => "failure_value", + }, + } + + result = deploy_fixtures("crd", subset: ["with_custom_conditions_cr.yml"]) do |resource| + cr = resource["with_custom_conditions_cr.yml"]["Customized"].first + cr.merge!(failure_conditions) + end + assert_deploy_failure(result) + assert_logs_match_all([ + "test error message jsonpath", + "test custom error message", + ]) + ensure + wait_for_all_crd_deletion + end + + def test_deploying_crs_with_invalid_crd_conditions_fails + # Since CRDs are not always deployed along with their CRs and kubernetes-deploy is not the only way CRDs are + # deployed, we need to model the case where poorly configured rollout_conditions are present before deploying a CR + KubernetesDeploy::DeployTask.any_instance.expects(:validate_resources).returns(:true) + crd_result = deploy_fixtures("crd", subset: ["with_custom_conditions.yml"]) do |resource| + crd = resource["with_custom_conditions.yml"]["CustomResourceDefinition"].first + crd["metadata"]["annotations"].merge!( + KubernetesDeploy::CustomResourceDefinition::ROLLOUT_CONDITIONS_ANNOTATION => "blah" + ) + end + + assert_deploy_success(crd_result) + KubernetesDeploy::DeployTask.any_instance.unstub(:validate_resources) + + cr_result = deploy_fixtures("crd", subset: ["with_custom_conditions_cr.yml", "with_custom_conditions_cr2.yml"]) + assert_deploy_failure(cr_result) + assert_logs_match_all([ + /Invalid template: Customized-with-customized-params/, + /Rollout conditions are not valid JSON/, + /Invalid template: Customized-with-customized-params/, + /Rollout conditions are not valid JSON/, + ], in_order: true) + ensure + wait_for_all_crd_deletion + end + private def wait_for_all_crd_deletion diff --git a/test/unit/kubernetes-deploy/kubernetes_resource/custom_resource_definition_test.rb b/test/unit/kubernetes-deploy/kubernetes_resource/custom_resource_definition_test.rb new file mode 100644 index 000000000..d823f6af0 --- /dev/null +++ b/test/unit/kubernetes-deploy/kubernetes_resource/custom_resource_definition_test.rb @@ -0,0 +1,232 @@ +# frozen_string_literal: true +require 'test_helper' + +class CustomResourceDefinitionTest < KubernetesDeploy::TestCase + def test_rollout_conditions_nil_when_none_present + crd = build_crd(crd_spec) + refute(crd.rollout_conditions) + end + + def test_arbitrary_rollout_conditions + rollout_conditions = { + success_conditions: [ + { + path: "$.test_path", + value: "test_value", + }, + ], + failure_conditions: [ + { + path: "$.test_path", + value: "test_value", + }, + ], + }.to_json + + crd = build_crd(merge_rollout_annotation(rollout_conditions)) + crd.validate_definition(kubectl) + refute(crd.validation_failed?, "Valid rollout conditions failed validation") + end + + def test_rollout_conditions_failure_conditions_optional + rollout_conditions = { + success_conditions: [ + { + path: "$.test_path", + value: "test_value", + }, + ], + }.to_json + + crd = build_crd(merge_rollout_annotation(rollout_conditions)) + crd.validate_definition(kubectl) + refute(crd.validation_failed?, "Valid rollout conditions failed validation") + end + + def test_rollout_conditions_invalid_when_path_or_value_missing + missing_keys = { + success_conditions: [{ path: "$.test" }], + failure_conditions: [{ value: "test" }], + } + + crd = build_crd(merge_rollout_annotation(missing_keys.to_json)) + crd.validate_definition(kubectl) + + assert(crd.validation_failed?, "Missing path/value keys should fail validation") + assert_equal(crd.validation_error_msg, + "Annotation #{KubernetesDeploy::CustomResourceDefinition::ROLLOUT_CONDITIONS_ANNOTATION} " \ + "on #{crd.name} is invalid: Missing required key(s) for success_condition: [:value], " \ + "Missing required key(s) for failure_condition: [:path]") + end + + def test_rollout_conditions_fails_validation_when_missing_condition_keys + missing_keys = { success_conditions: [] }.to_json + + crd = build_crd(merge_rollout_annotation(missing_keys)) + crd.validate_definition(kubectl) + + assert(crd.validation_failed?, "success_conditions requires at least one entry") + assert_equal(crd.validation_error_msg, + "Annotation #{KubernetesDeploy::CustomResourceDefinition::ROLLOUT_CONDITIONS_ANNOTATION} " \ + "on #{crd.name} is invalid: success_conditions must contain at least one entry") + end + + def test_rollout_conditions_fails_validation_with_invalid_json + crd = build_crd(merge_rollout_annotation('bad string')) + crd.validate_definition(kubectl) + assert(crd.validation_failed?, "Invalid rollout conditions were accepted") + assert(crd.validation_error_msg.match( + "Annotation #{KubernetesDeploy::CustomResourceDefinition::ROLLOUT_CONDITIONS_ANNOTATION} " \ + "on #{crd.name} is invalid: Rollout conditions are not valid JSON:" + )) + end + + def test_rollout_conditions_fails_validation_when_condition_is_wrong_type + crd = build_crd(merge_rollout_annotation({ + success_conditions: {}, + }.to_json)) + crd.validate_definition(kubectl) + assert(crd.validation_failed?, "Invalid rollout conditions were accepted") + assert(crd.validation_error_msg.match("success_conditions should be Array but found Hash")) + end + + def test_cr_instance_fails_validation_when_rollout_conditions_for_crd_invalid + crd = build_crd(merge_rollout_annotation('bad string')) + cr = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: @statsd_tags, crd: crd, + definition: { + "kind" => "UnitTest", + "metadata" => { "name" => "test" }, + }) + cr.validate_definition(kubectl) + assert_equal(cr.validation_error_msg, + "The CRD that specifies this resource is using invalid rollout conditions. Kubernetes-deploy will not be " \ + "able to continue until those rollout conditions are fixed.\nRollout conditions can be found on the CRD " \ + "that defines this resource (unittests.stable.example.io), under the annotation " \ + "kubernetes-deploy.shopify.io/instance-rollout-conditions.\nValidation failed with: " \ + "Rollout conditions are not valid JSON: Empty input () at line 1, column 1 [parse.c:963] in 'bad string") + end + + def test_cr_instance_valid_when_rollout_conditions_for_crd_valid + rollout_conditions = { + success_conditions: [ + { + path: "$.test_path", + value: "test_value", + }, + ], + failure_conditions: [ + { + path: "$.test_path", + value: "test_value", + }, + ], + }.to_json + + crd = build_crd(merge_rollout_annotation(rollout_conditions)) + cr = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: [], crd: crd, + definition: { + "kind" => "UnitTest", + "metadata" => { "name" => "test" }, + }) + cr.validate_definition(kubectl) + refute(cr.validation_failed?) + end + + def test_instance_timeout_annotation + crd = build_crd(crd_spec.merge( + "metadata" => { + "name" => "unittests.stable.example.io", + }, + )) + cr = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: [], crd: crd, + definition: { "kind" => "UnitTest", "metadata" => { "name" => "test" } }) + assert_equal(cr.timeout, KubernetesDeploy::CustomResource.timeout) + + crd = build_crd(crd_spec.merge( + "metadata" => { + "name" => "unittests.stable.example.io", + "annotations" => { + KubernetesDeploy::CustomResourceDefinition::TIMEOUT_FOR_INSTANCE_ANNOTATION => "60S", + }, + } + )) + cr = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: [], crd: crd, + definition: { "kind" => "UnitTest", "metadata" => { "name" => "test" } }) + assert_equal(cr.timeout, 60) + end + + def test_instance_timeout_messages_with_rollout_conditions + crd = build_crd(crd_spec.merge( + "metadata" => { + "name" => "unittests.stable.example.io", + "annotations" => { + KubernetesDeploy::CustomResourceDefinition::ROLLOUT_CONDITIONS_ANNOTATION => "true", + }, + }, + )) + cr = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: [], crd: crd, + definition: { + "kind" => "UnitTest", + "metadata" => { + "name" => "test", + }, + }) + + cr.expects(:current_generation).returns(1) + cr.expects(:observed_generation).returns(1) + assert_equal(cr.timeout_message, KubernetesDeploy::KubernetesResource::STANDARD_TIMEOUT_MESSAGE) + + cr.expects(:current_generation).returns(1) + cr.expects(:observed_generation).returns(2) + assert_equal(cr.timeout_message, KubernetesDeploy::CustomResource::TIMEOUT_MESSAGE_DIFFERENT_GENERATIONS) + end + + def test_instance_timeout_messages_without_rollout_conditions + crd = build_crd(crd_spec.merge( + "metadata" => { + "name" => "unittests.stable.example.io", + }, + )) + cr = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: [], crd: crd, + definition: { + "kind" => "UnitTest", + "metadata" => { + "name" => "test", + }, + }) + + assert_equal(cr.timeout_message, KubernetesDeploy::KubernetesResource::STANDARD_TIMEOUT_MESSAGE) + end + + private + + def kubectl + @kubectl ||= build_runless_kubectl + end + + def crd_spec + @crd_spec ||= YAML.load_file(File.join(fixture_path('for_unit_tests'), 'crd_test.yml')) + end + + def merge_rollout_annotation(rollout_conditions) + crd_spec.merge( + "metadata" => { + "name" => "unittests.stable.example.io", + "annotations" => { + KubernetesDeploy::CustomResourceDefinition::ROLLOUT_CONDITIONS_ANNOTATION => rollout_conditions, + }, + }, + ) + end + + def build_crd(spec) + KubernetesDeploy::CustomResourceDefinition.new(namespace: 'test', context: 'nope', + definition: spec, logger: @logger) + end +end diff --git a/test/unit/kubernetes-deploy/kubernetes_resource_test.rb b/test/unit/kubernetes-deploy/kubernetes_resource_test.rb index 5f75e283c..a3fdbf236 100644 --- a/test/unit/kubernetes-deploy/kubernetes_resource_test.rb +++ b/test/unit/kubernetes-deploy/kubernetes_resource_test.rb @@ -299,6 +299,42 @@ def test_lowercase_custom_resource_kind_does_not_raise ) end + def test_build_handles_hardcoded_and_core_and_dynamic_objects + # Hardcoded CRs + redis_crd = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: [], definition: build_crd(name: "redis")) + redis_cr = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: [], crd: redis_crd, + definition: { "kind" => "Redis", "metadata" => { "name" => "test" } }) + assert_equal(redis_cr.class, KubernetesDeploy::Redis) + + # Dynamic with no rollout config + no_config_crd = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: [], definition: build_crd(name: "noconfig")) + no_config_cr = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: [], crd: no_config_crd, + definition: { "kind" => "Noconfig", "metadata" => { "name" => "test" } }) + assert_equal(no_config_cr.class, KubernetesDeploy::CustomResource) + + # With rollout config + with_config_crd = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: [], definition: build_crd(name: "withconfig", with_config: true)) + with_config_cr = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", + logger: @logger, statsd_tags: [], crd: with_config_crd, + definition: { "kind" => "Withconfig", "metadata" => { "name" => "test" } }) + assert_equal(with_config_cr.class, KubernetesDeploy::CustomResource) + + # Hardcoded resource + svc = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", logger: @logger, + statsd_tags: [], definition: { "kind" => "Service", "metadata" => { "name" => "test" } }) + assert_equal(svc.class, KubernetesDeploy::Service) + + # Generic resource + resource = KubernetesDeploy::KubernetesResource.build(namespace: "test", context: "test", logger: @logger, + statsd_tags: [], definition: { "kind" => "Unkonwn", "metadata" => { "name" => "test" } }) + assert_equal(resource.class, KubernetesDeploy::KubernetesResource) + end + private def kubectl @@ -374,4 +410,23 @@ def build_event_jsonpath(dummy_events) jsonpaths << [e[:kind], e[:name], e[:count], e[:last_seen].to_s, e[:reason], e[:message]].join(field_separator) end.join(event_separator) end + + def build_crd(name:, with_config: false) + crd = { + "kind" => "CustomResourceDefinition", + "metadata" => { + "name" => "#{name}s.test.io", + "annotations" => {}, + }, + "spec" => { + "names" => { + "kind" => name.titleize, + }, + }, + } + if with_config + crd["metadata"]["annotations"][KubernetesDeploy::CustomResourceDefinition::ROLLOUT_CONDITIONS_ANNOTATION] = "true" + end + crd + end end