Fix kubernetes_job updates #769

pdecat · 2020-02-18T17:32:28Z

This is marked as WIP because it needs:

complete acceptance test run
documentation updates
~~Add ExpectedDiffChanges to assert expected diff change type in acceptance test steps terraform-plugin-sdk#329~~

This PR allows to update the following fields of a kubernetes_job resource without recreating it:

active_deadline_seconds
backoff_limit
parallelism

It also forces the recreation of a job resource when its pod template image field is modified.
Previously, it would detect a change was needed:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # kubernetes_job.terraform will be updated in-place
  ~ resource "kubernetes_job" "terraform" {
        id = "default/terraform"

        metadata {
            annotations      = {}
            generation       = 0
            labels           = {}
            name             = "terraform"
            namespace        = "default"
            resource_version = "10107"
            self_link        = "/apis/batch/v1/namespaces/default/jobs/terraform"
            uid              = "c1f56d3b-b9cb-40ad-abb3-0ca76a677329"
        }

      ~ spec {
            active_deadline_seconds = 0
            backoff_limit           = 1
            completions             = 1
            manual_selector         = false
            parallelism             = 1

          ~ selector {
              + match_labels = {}
            }

          ~ template {
                metadata {
                    annotations = {}
                    generation  = 0
                    labels      = {}
                }

              ~ spec {
                    active_deadline_seconds          = 0
                    automount_service_account_token  = true
                    dns_policy                       = "ClusterFirst"
                    host_ipc                         = false
                    host_network                     = false
                    host_pid                         = false
                    node_selector                    = {}
                    restart_policy                   = "Never"
                    service_account_name             = "terraform"
                    share_process_namespace          = false
                    termination_grace_period_seconds = 30

                  ~ container {
                        args                     = []
                        command                  = [
                            "sh",
                            "-c",
2020/02/18 16:44:31 [DEBUG] command: asking for input: "Do you want to perform these actions?"
                            "set && set -x && apk --no-cache add curl && mkdir -p ~/.terraform.d/plugins && curl https://storage.googleapis.com/pdecat-builds/terraform-provider-kubernet
es_v1.11.1-dev > ~/.terraform.d/plugins/terraform-provider-kubernetes_v1.11.1-dev && chmod +x ~/.terraform.d/plugins/* && mkdir /tf && cd /tf && cp /configuration/main.tf . && TF_LOG=TR
ACE terraform init && TF_LOG=TRACE terraform plan && TF_LOG=TRACE terraform apply -auto-approve ; sleep 60 && terraform destroy -auto-approve",
                        ]
                      ~ image                    = "hashicorp/terraform:0.12.20" -> "hashicorp/terraform:0.12.13"
                        image_pull_policy        = "IfNotPresent"
                        name                     = "terraform"
                        stdin                    = false
                        stdin_once               = false
                        termination_message_path = "/dev/termination-log"
                        tty                      = false

                        resources {
                        }

                        volume_mount {
                            mount_path        = "/configuration"
                            mount_propagation = "None"
                            name              = "configuration"
                            read_only         = false
                        }
                        volume_mount {
                            mount_path        = "/root"
                            mount_propagation = "None"
                            name              = "home"
                            read_only         = false
                        }
                    }

                    volume {
                        name = "configuration"

                        config_map {
                            default_mode = "0644"
                            name         = "terraform"
                        }
                    }
                    volume {
                        name = "home"

                        empty_dir {}
                    }
                }
            }
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

But submit an empty patch operation:

2020-02-18T16:46:07.508+0100 [DEBUG] plugin.terraform-provider-kubernetes_v1.11.1-dev: 2020/02/18 16:46:07 [DEBUG] Kubernetes API Request Details:
2020-02-18T16:46:07.508+0100 [DEBUG] plugin.terraform-provider-kubernetes_v1.11.1-dev: ---[ REQUEST ]---------------------------------------
2020-02-18T16:46:07.508+0100 [DEBUG] plugin.terraform-provider-kubernetes_v1.11.1-dev: PATCH /apis/batch/v1/namespaces/default/jobs/terraform HTTP/1.1
2020-02-18T16:46:07.508+0100 [DEBUG] plugin.terraform-provider-kubernetes_v1.11.1-dev: Host: 192.168.39.222:8443
2020-02-18T16:46:07.508+0100 [DEBUG] plugin.terraform-provider-kubernetes_v1.11.1-dev: User-Agent: HashiCorp/1.0 Terraform/0.12.20
2020-02-18T16:46:07.508+0100 [DEBUG] plugin.terraform-provider-kubernetes_v1.11.1-dev: Content-Length: 2
2020-02-18T16:46:07.508+0100 [DEBUG] plugin.terraform-provider-kubernetes_v1.11.1-dev: Accept: application/json, */*
2020-02-18T16:46:07.508+0100 [DEBUG] plugin.terraform-provider-kubernetes_v1.11.1-dev: Content-Type: application/json-patch+json
2020-02-18T16:46:07.508+0100 [DEBUG] plugin.terraform-provider-kubernetes_v1.11.1-dev: Accept-Encoding: gzip
2020-02-18T16:46:07.508+0100 [DEBUG] plugin.terraform-provider-kubernetes_v1.11.1-dev:
2020-02-18T16:46:07.508+0100 [DEBUG] plugin.terraform-provider-kubernetes_v1.11.1-dev: []

pdecat · 2020-02-18T17:33:20Z

kubernetes/schema_container.go

@@ -384,6 +384,7 @@ func containerFields(isUpdatable, isInitContainer bool) map[string]*schema.Schem
 		},
 		"image": {
 			Type:        schema.TypeString,
+			ForceNew:    !isUpdatable,


Only added ForceNew: !isUpdatable here as that's what made me notice the issue, but it should probably be generalized to other fields.

pdecat · 2020-02-18T17:34:49Z

kubernetes/schema_job_spec.go

@@ -36,14 +36,12 @@ func jobSpecFields() map[string]*schema.Schema {
 		"active_deadline_seconds": {
 			Type:         schema.TypeInt,
 			Optional:     true,
-			ForceNew:     true,


This used to force recreation of the resource so the previous acceptance passed anyway: https://github.com/terraform-providers/terraform-provider-kubernetes/pull/769/files#diff-de5bc44a1e838d53cd603bc71a631b48R74

pdecat · 2020-02-18T17:35:25Z

kubernetes/resource_kubernetes_job.go

@@ -82,7 +82,13 @@ func resourceKubernetesJobUpdate(d *schema.ResourceData, meta interface{}) error
 	}

 	ops := patchMetadata("metadata.0.", "/metadata/", d)
-
+	if d.HasChange("spec") {
+		specOps, err := patchJobSpec("/spec", "spec.0.", d)


patchJobSpec() was not invoked anywhere before.

pdecat · 2020-02-18T17:44:54Z

kubernetes/resource_kubernetes_job_test.go

 					resource.TestCheckResourceAttr("kubernetes_job.test", "spec.0.template.0.spec.0.container.0.name", "hello"),
 					resource.TestCheckResourceAttr("kubernetes_job.test", "spec.0.template.0.spec.0.container.0.image", "alpine"),
 				),
 			},
+			{
+				Config: testAccKubernetesJobConfig_recreated_image(name),


@alexsomesan do you know of a way to ensure a resource is updated or recreated during acceptance test steps checks?

In the context of kubernetes, this could probably be achieved comparing resource uid between steps, but it is not trivial, and it could most certainly be achieved more generically at the terraform TestStep level.

There isn't any canonical way. For Kubernetes specifically I would look at metadata.generation or metadata.creation_timestamp and track those between the different test steps. I'm not entirely sure how generation behaves between updates, but the creation timestamp should not change.

I'll try this. In the mean time, I found https://github.com/hashicorp/terraform-plugin-sdk/issues/77

I've submitted hashicorp/terraform-plugin-sdk#329, based on hashicorp/terraform-plugin-sdk#222

@pdecat you were right about looking at the UID, btw. I've been adding a bunch of checks to ensure resources aren't deleted and re-created. I built off of someone else's work who implemented the ForceNew checks in the provider tests. So that's how the UID comparison is done.

pdecat · 2020-02-19T17:22:54Z

Acceptance test results with the new TestStep.ExpectedDiffChanges() assertions:

# make testacc TEST=./kubernetes TESTARGS='-run=TestAccKubernetesJob -count=1'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./kubernetes -v -run=TestAccKubernetesJob -count=1 -timeout 120m
go: finding k8s.io/client-go v12.0.0+incompatible
go: finding k8s.io/client-go v12.0.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-google v2.17.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-aws v2.32.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-google v2.17.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-aws v2.32.0+incompatible
=== RUN   TestAccKubernetesJob_basic
--- PASS: TestAccKubernetesJob_basic (32.57s)
PASS
ok      github.com/terraform-providers/terraform-provider-kubernetes/kubernetes 32.616s

…reated/updated/recreate between acceptance test steps

pdecat

I believe this PR is still useful, so I'm going to remove all references to ExpectedDiffChanges that I was trying to introduce in hashicorp/terraform-plugin-sdk#329 PR which was declined because of the Terraform SDK v2 transition.

pdecat · 2020-10-14T06:59:26Z

kubernetes/resource_kubernetes_job_test.go

@@ -67,7 +67,8 @@ func TestAccKubernetesJob_basic(t *testing.T) {
 		CheckDestroy:  testAccCheckKubernetesJobDestroy,
 		Steps: []resource.TestStep{
 			{
-				Config: testAccKubernetesJobConfig_basic(name),
+				Config:              testAccKubernetesJobConfig_basic(name),
+				ExpectedDiffChanges: map[string]terraform.DiffChangeType{"kubernetes_job.test": terraform.DiffCreate},


The hashicorp/terraform-plugin-sdk#329 PR that was introducing ExpectedDiffChanges was declined because of the Terraform SDK v2 transition.

pdecat · 2020-10-14T06:59:32Z

kubernetes/resource_kubernetes_job_test.go

@@ -86,7 +87,29 @@ func TestAccKubernetesJob_basic(t *testing.T) {
 				),
 			},
 			{
-				Config: testAccKubernetesJobConfig_modified(name),
+				Config:              testAccKubernetesJobConfig_modified(name),
+				ExpectedDiffChanges: map[string]terraform.DiffChangeType{"kubernetes_job.test": terraform.DiffUpdate},


The hashicorp/terraform-plugin-sdk#329 PR that was introducing ExpectedDiffChanges was declined because of the Terraform SDK v2 transition.

pdecat · 2020-10-14T06:59:40Z

kubernetes/resource_kubernetes_job_test.go

+			},
+			{
+				Config:              testAccKubernetesJobConfig_recreated_selector(name),
+				ExpectedDiffChanges: map[string]terraform.DiffChangeType{"kubernetes_job.test": terraform.DiffDestroyCreate},


The hashicorp/terraform-plugin-sdk#329 PR that was introducing ExpectedDiffChanges was declined because of the Terraform SDK v2 transition.

pdecat · 2020-10-14T06:59:45Z

kubernetes/resource_kubernetes_job_test.go

 					resource.TestCheckResourceAttr("kubernetes_job.test", "spec.0.template.0.spec.0.container.0.name", "hello"),
 					resource.TestCheckResourceAttr("kubernetes_job.test", "spec.0.template.0.spec.0.container.0.image", "alpine"),
 					resource.TestCheckNoResourceAttr("kubernetes_job.test", "wait_for_completion"),
 				),
 			},
+			{
+				Config:              testAccKubernetesJobConfig_recreated_image(name),
+				ExpectedDiffChanges: map[string]terraform.DiffChangeType{"kubernetes_job.test": terraform.DiffDestroyCreate},


The hashicorp/terraform-plugin-sdk#329 PR that was introducing ExpectedDiffChanges was declined because of the Terraform SDK v2 transition.

…declined

pdecat · 2020-10-14T13:00:13Z

Some terrafmt errors are not picked up by the CI:

$ make tests-lint
go install github.com/bflad/tfproviderdocs
go install github.com/client9/misspell/cmd/misspell
go install github.com/katbyte/terrafmt
go mod tidy
go mod vendor
==> Checking acceptance test terraform blocks code with terrafmt...
ERRO[0000] block 1 @ kubernetes/resource_kubernetes_job_test.go:340 failed to find end of block 
ERRO[0000] block 2 @ kubernetes/resource_kubernetes_job_test.go:340 failed to find end of block 
The command "make tests-lint" exited with 0.

dak1n1 · 2020-11-20T17:06:26Z

Thanks for this PR! I just noticed it. I happen to have a bunch of related work in progress. I think I can fit your changes into my PR, since it's related to PodSpec. I'll make sure you get credit for the work too (co-author in github).

dak1n1 · 2020-12-02T23:06:52Z

I pulled your changes into this PR #1074, so I'll go ahead and close this old PR. Thanks for these contributions!

ghost · 2021-01-02T05:10:00Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

ghost added the size/M label Feb 18, 2020

pdecat commented Feb 18, 2020

View reviewed changes

pdecat requested a review from alexsomesan February 18, 2020 17:36

pdecat commented Feb 18, 2020

View reviewed changes

This was referenced Feb 19, 2020

helper/resource: Support Expected/Unexpected Diff Changes in TestStep hashicorp/terraform-plugin-sdk#222

Closed

Add ExpectedDiffChanges to assert expected diff change type in acceptance test steps hashicorp/terraform-plugin-sdk#329

Closed

ghost added size/XXL dependencies and removed size/M labels Feb 19, 2020

pdecat marked this pull request as ready for review March 4, 2020 09:59

pdecat force-pushed the fix-job-updates branch from 50e1b83 to 7c775e0 Compare March 4, 2020 10:03

ghost added size/L and removed size/XXL labels Mar 4, 2020

pdecat force-pushed the fix-job-updates branch from 7c775e0 to 6330de6 Compare April 1, 2020 07:59

pdecat force-pushed the fix-job-updates branch 2 times, most recently from ecb65a3 to b8defeb Compare April 22, 2020 05:34

ghost added size/XXL and removed size/L labels Apr 22, 2020

pdecat added 2 commits October 14, 2020 08:55

Fix kubernetes_job updates

f2025f5

Use TestStep.ExpectedDiffChanges() to ensure resources are properly c…

7f9e6d2

…reated/updated/recreate between acceptance test steps

pdecat force-pushed the fix-job-updates branch from b8defeb to 7f9e6d2 Compare October 14, 2020 06:56

ghost added size/M and removed size/XXL labels Oct 14, 2020

pdecat commented Oct 14, 2020

View reviewed changes

Remove ExpectedDiffChanges as hashicorp/terraform-plugin-sdk#329 was …

a4ee694

…declined

pdecat changed the title ~~[WIP] Fix kubernetes_job updates~~ Fix kubernetes_job updates Oct 14, 2020

dak1n1 removed the request for review from alexsomesan November 20, 2020 17:06

dak1n1 self-assigned this Nov 20, 2020

dak1n1 closed this Dec 2, 2020

pdecat deleted the fix-job-updates branch December 3, 2020 07:03

dak1n1 mentioned this pull request Dec 7, 2020

Fix ForceNew and non-empty plans caused by PodSpec defaults #1074

Merged

2 tasks

ghost locked as resolved and limited conversation to collaborators Jan 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix kubernetes_job updates #769

Fix kubernetes_job updates #769

pdecat commented Feb 18, 2020 •

edited

pdecat Feb 18, 2020

pdecat Feb 18, 2020

pdecat Feb 18, 2020

pdecat Feb 18, 2020

alexsomesan Feb 18, 2020

pdecat Feb 19, 2020

pdecat Feb 19, 2020

dak1n1 Nov 20, 2020

pdecat commented Feb 19, 2020

pdecat left a comment

pdecat Oct 14, 2020

pdecat Oct 14, 2020

pdecat Oct 14, 2020

pdecat Oct 14, 2020

pdecat commented Oct 14, 2020 •

edited

dak1n1 commented Nov 20, 2020

dak1n1 commented Dec 2, 2020

ghost commented Jan 2, 2021

Fix kubernetes_job updates #769

Fix kubernetes_job updates #769

Conversation

pdecat commented Feb 18, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdecat commented Feb 19, 2020

pdecat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdecat commented Oct 14, 2020 • edited

dak1n1 commented Nov 20, 2020

dak1n1 commented Dec 2, 2020

ghost commented Jan 2, 2021

pdecat commented Feb 18, 2020 •

edited

pdecat commented Oct 14, 2020 •

edited