Bug in role binding patch

[simonklb]

I noticed that during an in-place update of the subjects in a kubernetes_cluster_role_binding resource the resulting applied changes were incorrect. This seems to be happening when the update changes the number of subjects and the order they are added. The plan looks correct but when applied the resulting subjects in the Kubernetes resource are incorrect.

This is a pretty critical error as it could give the user a false sense of assurance that the correct access rules are in place while they are in fact not.

Terraform Version

Terraform v0.12.17

• provider.kubernetes v1.10.0

Affected Resource(s)

• kubernetes_cluster_role_binding
• kubernetes_role_binding (probably, not tested)

Terraform Configuration Files

resource "kubernetes_cluster_role_binding" "cluster_admin" {
  metadata {
    name = "custom-cluster-admin"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }

  dynamic "subject" {
    for_each = local.rbac_cluster_admin_users
    content {
      kind      = "User"
      name      = subject.value
      api_group = "rbac.authorization.k8s.io"
    }
  }

  dynamic "subject" {
    for_each = local.rbac_cluster_admin_groups
    content {
      kind      = "Group"
      name      = subject.value
      api_group = "rbac.authorization.k8s.io"
    }
  }
}

Expected Behavior

The correct subjects in the ClusterRoleBinding k8s resource.

Actual Behavior

Incorrect subjects in the ClusterRoleBinding k8s resource.

Steps to Reproduce

1. Update the kubernetes_cluster_role_binding resource subject configuration.
2. terraform apply
3. terraform apply again

See the failing test case I commited here for a specific example: #712

[pdecat]

Looks a lot like #650, isn't it?

[simonklb]

@pdecat Looks like that issue is in regards to a value not being changed, while in this case, things are actually changed but incorrectly. Might be related, but I don't think it's the same bug.

[pdecat]

Ok, my bad, that sounds much worse!

[pdecat]

Running with TF_LOG=debug, the PATCH operation looks ok (unlike what I reported in #650):

2019/12/17 17:54:33 [DEBUG] Kubernetes API Request Details:                                                                                                                              
---[ REQUEST ]---------------------------------------                                                                                                                                    
PATCH /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/tf-acc-test:gfudvbgvaz HTTP/1.1                                                                                             
Host: 192.168.39.31:8443                                                                                                                                                                 
User-Agent: HashiCorp/1.0 Terraform/0.12.7-sdk                                                                                                                                           
Content-Length: 322                                                                                                                                                                      
Accept: application/json, */*                                                                                                                                                            
Content-Type: application/json-patch+json                                                                                                                                                
Accept-Encoding: gzip                                                                                                                                                                    
                                                                                                                                                                                         
[                                                                                                                                                                                        
 {                                                                                                                                                                                       
  "path": "/subjects/0",                                                                                                                                                                 
  "value": {                                                                                                                                                                             
   "kind": "User",                                                                                                                                                                       
   "apiGroup": "rbac.authorization.k8s.io",                                                                                                                                              
   "name": "notauser2",                                                                                                                                                                  
   "namespace": "default"                                                                                                                                                                
  },                                                                                                                                                                                     
  "op": "replace"                                                                                                                                                                        
 },                                                                                                                                                                                      
 {                                                                                                                                                                                       
  "path": "/subjects/1",                                                                                                                                                                 
  "value": {                                                                                                                                                                             
   "kind": "User",                                                                                                                                                                       
   "apiGroup": "rbac.authorization.k8s.io",                                                                                                                                              
   "name": "notauser4",                                                                                                                                                                  
   "namespace": "default"                                                                                                                                                                
  },                                                                                                                                                                                     
  "op": "replace"                                                                                                                                                                        
 },                                                                                                                                                                                      
 {                                                                                                                                                                                       
  "path": "/subjects/1",                                                                                                                                                                 
  "op": "remove"                                                                                                                                                                         
 }                                                                                                                                                                                       
]                                                                                                                                                                                        
-----------------------------------------------------                                                                                                                                    
2019/12/17 17:54:33 [DEBUG] Kubernetes API Response Details:                                                                                                                             
---[ RESPONSE ]--------------------------------------                                                                                                                                    
HTTP/2.0 200 OK                                                                                                                                                                          
Content-Length: 628                                                                                                                                                                      
Cache-Control: no-cache, private                                                                                                                                                         
Content-Type: application/json                                                                                                                                                           
Date: Tue, 17 Dec 2019 16:54:33 GMT                                                                                                                                                      
                                                                                                                                                                                         
{                                                                                                                                                                                        
 "kind": "ClusterRoleBinding",                                                                                                                                                           
 "apiVersion": "rbac.authorization.k8s.io/v1",                                                                                                                                           
 "metadata": {
  "name": "tf-acc-test:gfudvbgvaz",
  "selfLink": "/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/tf-acc-test%3Agfudvbgvaz",
  "uid": "a0ebc4a6-f1c6-4b20-9429-56995ffdece5",
  "resourceVersion": "1356964",
  "creationTimestamp": "2019-12-17T16:54:32Z"
 },
 "subjects": [
  {
   "kind": "User",
   "apiGroup": "rbac.authorization.k8s.io",
   "name": "notauser2",
   "namespace": "default"
  },
  {
   "kind": "User",
   "apiGroup": "rbac.authorization.k8s.io",
   "name": "notauser3",
   "namespace": "default"
  }
 ],
 "roleRef": {
  "apiGroup": "rbac.authorization.k8s.io",
  "kind": "ClusterRole",
  "name": "cluster-admin"
 }
}

Edit: the issue is that the delete operation for "path": "/subjects/1" took place after its update.

[pdecat]

Processing remove operations before replace operations resolves the issue:

diff --git kubernetes/structures_rbac.go kubernetes/structures_rbac.go
index 0533bc6e..8088d915 100644
--- kubernetes/structures_rbac.go
+++ kubernetes/structures_rbac.go
@@ -132,12 +132,6 @@ func patchRbacSubject(d *schema.ResourceData) PatchOperations {
        if common > len(oldsubjects) {
                common = len(oldsubjects)
        }
-       for i, v := range newsubjects[:common] {
-               ops = append(ops, &ReplaceOperation{
-                       Path:  "/subjects/" + strconv.Itoa(i),
-                       Value: v,
-               })
-       }
        if len(oldsubjects) > len(newsubjects) {
                for i := len(newsubjects); i < len(oldsubjects); i++ {
                        ops = append(ops, &RemoveOperation{
@@ -145,6 +139,12 @@ func patchRbacSubject(d *schema.ResourceData) PatchOperations {
                        })
                }
        }
+       for i, v := range newsubjects[:common] {
+               ops = append(ops, &ReplaceOperation{
+                       Path:  "/subjects/" + strconv.Itoa(i),
+                       Value: v,
+               })
+       }
        if len(newsubjects) > len(oldsubjects) {
                for i, v := range newsubjects[common:] {
                        ops = append(ops, &AddOperation{

2019/12/17 18:04:52 [DEBUG] Kubernetes API Request Details:
---[ REQUEST ]---------------------------------------
PATCH /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/tf-acc-test:4warucz1gx HTTP/1.1
Host: 192.168.39.31:8443
User-Agent: HashiCorp/1.0 Terraform/0.12.7-sdk
Content-Length: 322
Accept: application/json, */*
Content-Type: application/json-patch+json
Accept-Encoding: gzip

[
 {
  "path": "/subjects/1",
  "op": "remove"
 },
 {
  "path": "/subjects/0",
  "value": {
   "kind": "User",
   "apiGroup": "rbac.authorization.k8s.io",
   "name": "notauser2",
   "namespace": "default"
  },
  "op": "replace"
 },
 {
  "path": "/subjects/1",
  "value": {
   "kind": "User",
   "apiGroup": "rbac.authorization.k8s.io",
   "name": "notauser4",
   "namespace": "default"
  },
  "op": "replace"
 }
]
-----------------------------------------------------
2019/12/17 18:04:52 [DEBUG] Kubernetes API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Content-Length: 628
Cache-Control: no-cache, private
Content-Type: application/json
Date: Tue, 17 Dec 2019 17:04:52 GMT

{
 "kind": "ClusterRoleBinding",
 "apiVersion": "rbac.authorization.k8s.io/v1",
 "metadata": {
  "name": "tf-acc-test:4warucz1gx",
  "selfLink": "/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/tf-acc-test%3A4warucz1gx",
  "uid": "b166ca7e-3858-409d-b159-5702ed1de904",
  "resourceVersion": "1357731",
  "creationTimestamp": "2019-12-17T17:04:52Z"
 },
 "subjects": [
  {
   "kind": "User",
   "apiGroup": "rbac.authorization.k8s.io",
   "name": "notauser2",
   "namespace": "default"
  },
  {
   "kind": "User",
   "apiGroup": "rbac.authorization.k8s.io",
   "name": "notauser4",
   "namespace": "default"
  }
 ],
 "roleRef": {
  "apiGroup": "rbac.authorization.k8s.io",
  "kind": "ClusterRole",
  "name": "cluster-admin"
 }
}

[pdecat]

Acceptance test results with this change:

# TF_ACC=1 go test -run TestAccKubernetesClusterRoleBinding ./kubernetes/ -v
go: finding k8s.io/client-go v12.0.0+incompatible
go: finding k8s.io/client-go v12.0.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-google v2.17.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-google v2.17.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-aws v2.32.0+incompatible
go: finding github.com/terraform-providers/terraform-provider-aws v2.32.0+incompatible
=== RUN   TestAccKubernetesClusterRoleBinding
--- PASS: TestAccKubernetesClusterRoleBinding (0.78s)
=== RUN   TestAccKubernetesClusterRoleBinding_serviceaccount_subject
--- PASS: TestAccKubernetesClusterRoleBinding_serviceaccount_subject (0.43s)
=== RUN   TestAccKubernetesClusterRoleBinding_group_subject
--- PASS: TestAccKubernetesClusterRoleBinding_group_subject (0.44s)
=== RUN   TestAccKubernetesClusterRoleBinding_importBasic
--- PASS: TestAccKubernetesClusterRoleBinding_importBasic (0.59s)
=== RUN   TestAccKubernetesClusterRoleBindingBug
--- PASS: TestAccKubernetesClusterRoleBindingBug (0.80s)
PASS
ok      github.com/terraform-providers/terraform-provider-kubernetes/kubernetes 3.095s

[pdecat]

PR #714 fixes this.

[weeco]

@pdecat I am using provider version 1.11.0 but patching the the role_binding resource still does not seem to work. In my case I added namespace = "xy". Thus I expected Terraform to delete the role binding in the default namespace and add it to the namespace xy but it doesn't show any changes in the diff.

[Xartos]

1.11.0 doesn't work for me either. I have a couple of role bindings with users as subjects and one Service account. And it was fine to create the subjects, but when I deleted one of the user subjects it tries to update the one I deleted in place to the Service account and it keeps the api_group = "rbac.authorization.k8s.io" even though if I have set api_group = "".

Like this: (test2@example.com is the one I deleted)

 ~ resource "kubernetes_cluster_role_binding" "cluster_admin" {
...
        subject {
            api_group = "rbac.authorization.k8s.io"
            kind      = "User"
            name      = "test1@example.com"
            namespace = "default"
        }
      ~ subject {
            api_group = "rbac.authorization.k8s.io"
            kind      = "User"
          ~ name      = "test2@example.com" -> "test3@example.com"
            namespace = "default"
        }
      ~ subject {
            api_group = "rbac.authorization.k8s.io"
          ~ kind      = "User" -> "ServiceAccount"
          ~ name      = "test3@example.com" -> "sa-name"
          ~ namespace = "default" -> "namespace-name"
        }
      - subject {
          - kind      = "ServiceAccount" -> null
          - name      = "sa-name" -> null
          - namespace = "namespace-name" -> null
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

This gives an error when applying.

[pdecat]

@pdecat I am using provider version 1.11.0 but patching the the role_binding resource still does not seem to work. In my case I added namespace = "xy". Thus I expected Terraform to delete the role binding in the default namespace and add it to the namespace xy but it doesn't show any changes in the diff.

That sounds like another issue that I cannot reproduce, e.g. with this change:

# git diff
diff --git custom-metrics-stackdriver-adapter.tf custom-metrics-stackdriver-adapter.tf
index a017f19..b0e4ed3 100644
--- custom-metrics-stackdriver-adapter.tf
+++ custom-metrics-stackdriver-adapter.tf
@@ -26,7 +26,7 @@ resource "kubernetes_cluster_role_binding" "custom_metrics_system_auth_delegator
 resource "kubernetes_role_binding" "custom_metrics_auth_reader" {
   metadata {
     name      = "custom-metrics-auth-reader"
-    namespace = "kube-system"
+    namespace = "test"
   }

   subject {

I properly get a destruction/recreation:

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # kubernetes_role_binding.custom_metrics_auth_reader must be replaced
-/+ resource "kubernetes_role_binding" "custom_metrics_auth_reader" {
      ~ id = "kube-system/custom-metrics-auth-reader" -> (known after apply)

      ~ metadata {
          - annotations      = {} -> null
          ~ generation       = 0 -> (known after apply)
          - labels           = {} -> null
            name             = "custom-metrics-auth-reader"
          ~ namespace        = "kube-system" -> "test" # forces replacement
          ~ resource_version = "55078730" -> (known after apply)
          ~ self_link        = "/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings/custom-metrics-auth-reader" -> (known after apply)
          ~ uid              = "3cf9c95f-85dc-1234-1234-42010a880006" -> (known after apply)
        }

        role_ref {
            api_group = "rbac.authorization.k8s.io"
            kind      = "Role"
            name      = "extension-apiserver-authentication-reader"
        }

      ~ subject {
          + api_group = (known after apply)
            kind      = "ServiceAccount"
            name      = "custom-metrics-stackdriver-adapter"
            namespace = "custom-metrics"
        }
    }

Plan: 1 to add, 0 to change, 1 to destroy.

------------------------------------------------------------------------

Could you please open another issue with all details and a debug log?

[pdecat]

1.11.0 doesn't work for me either. I have a couple of role bindings with users as subjects and one Service account. And it was fine to create the subjects, but when I deleted one of the user subjects it tries to update the one I deleted in place to the Service account and it keeps the api_group = "rbac.authorization.k8s.io" even though if I have set api_group = "".

Like this: (test2@example.com is the one I deleted)

 ~ resource "kubernetes_cluster_role_binding" "cluster_admin" {
...
        subject {
            api_group = "rbac.authorization.k8s.io"
            kind      = "User"
            name      = "test1@example.com"
            namespace = "default"
        }
      ~ subject {
            api_group = "rbac.authorization.k8s.io"
            kind      = "User"
          ~ name      = "test2@example.com" -> "test3@example.com"
            namespace = "default"
        }
      ~ subject {
            api_group = "rbac.authorization.k8s.io"
          ~ kind      = "User" -> "ServiceAccount"
          ~ name      = "test3@example.com" -> "sa-name"
          ~ namespace = "default" -> "namespace-name"
        }
      - subject {
          - kind      = "ServiceAccount" -> null
          - name      = "sa-name" -> null
          - namespace = "namespace-name" -> null
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

This gives an error when applying.

That sounds like another issue, could you please open another issue with all details and a debug log?

[aareet]

I'm closing this since it seems to be resolved with #714 and the subsequent reports are new issues. Please reopen if you can reproduce this with the latest version of the provider.