Dev: acceptance tests needing Cluster Admin Roles fail

[pdecat]

Acceptance tests needing Cluster Admin Roles fail

Terraform Version

HEAD from master after 1.5.0 release:

# git log -n 1
commit a44233c85e55cc6bc70850667c5f089ab7a3357e (HEAD -> master, upstream/master, upstream/HEAD)
Author: tf-release-bot <terraform@hashicorp.com>
Date:   Mon Jan 14 14:18:53 2019 +0000

    Cleanup after v1.5.0 release

Affected Tests

• TestAccKubernetesRole_basic
• TestAccKubernetesRole_importBasic
• TestAccKubernetesRole_generatedName

Terraform Configuration Files

N/A

Debug Output

N/A

Panic Output

N/A

Expected Behavior

All acceptance tests pass out of the box.

Actual Behavior

Acceptance tests needing Admin Cluster Roles fail with:

--- FAIL: TestAccKubernetesRole_basic (0.22s)
    testing.go:538: Step 0 error: Error applying: 1 error occurred:
                * kubernetes_role.test: 1 error occurred:
                * kubernetes_role.test: roles.rbac.authorization.k8s.io "tf-acc-test-it1c4fw9d0" is forbidden: attempt to grant extra privileges: [{[list] [] [pods] [foo] []} {[watch] [] [pods] [foo] []} {[get] [] [pods] [foo] []} {[list] [apps] [deployments] [] []} {[get] [apps] [deployments] [] []}] user=&{p******@******  [system:authenticated] map[user-assertion.cloud.google.com:[******]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]

Steps to Reproduce

1. clone this repo
2. cd terraform-provider-kubernetes/kubernetes/test-infra/gke
3. terraform init
4. terraform apply (when prompted, enter any recent kubernetes version, e.g. 1.11.5-gke.5)
5. export environment variables needed for acceptance tests
6. make testacc TEST=./kubernetes TESTARGS='-run=TestAccKubernetes.* -count=1'

Important Factoids

N/A

References

• N/A

Work-around

1. get Cluster Admin Role on the GKE cluster with kubectl --user=$KUBE_CTX_AUTH_INFO --cluster=$KUBE_CTX_CLUSTER create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value account)
2. enable legacy ABAC on the GKE cluster (not recommended for production clusters)

[pdecat]

Either the kubernetes/test-infra/gke/README.md file should be updated with additional instructions to acquire the cluster admin role, or the google_container_cluster.primary resource in the kubernetes/test-infra/gke/main.tf file be updated with enable_legacy_abac = "false".

WDYT?

[alexsomesan]

@pdecat In my case I don't have to take any extra steps. The Role tests pass out of the box.
I just build the GKE cluster like you described and then run the tests.
Here's how my config looks:

~/workspace/terraform-provider-kubernetes(master) » kubectl config view                                                                                                                      alex@alexs-macbook
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://REDACTED
  name: tf-acc-test-4eb6d977b47d757622e9
contexts:
- context:
    cluster: tf-acc-test-4eb6d977b47d757622e9
    namespace: default
    user: 1579a1af2b71a07d146819f84eda
  name: tf-k8s-gcp-test
current-context: tf-k8s-gcp-test
kind: Config
preferences:
  colors: true
users:
- name: 1579a1af2b71a07d146819f84eda
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED
    password: REDACTED
    username: 1579a1af2b71a07d146819f84eda
------------------------------------------------------------

Then I run the tests like this:

~/workspace/terraform-provider-kubernetes(master) » TESTARGS="-run '^TestAccKubernetesRole_'" make testacc                                                                                   alex@alexs-macbook
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test $(go list ./... |grep -v 'vendor') -v -run '^TestAccKubernetesRole_' -timeout 120m
?       github.com/terraform-providers/terraform-provider-kubernetes    [no test files]
=== RUN   TestAccKubernetesRole_basic
--- PASS: TestAccKubernetesRole_basic (0.67s)
=== RUN   TestAccKubernetesRole_importBasic
--- PASS: TestAccKubernetesRole_importBasic (0.36s)
=== RUN   TestAccKubernetesRole_generatedName
--- PASS: TestAccKubernetesRole_generatedName (0.32s)
PASS
ok      github.com/terraform-providers/terraform-provider-kubernetes/kubernetes (cached)
------------------------------------------------------------

Not sure yet, but it might be that the admin role is inherited in some way from the GCloud identity used to build the cluster?
I'd have to look into it a bit.

[pdecat]

What version of the google provider are you using?

[alexsomesan]

~/workspace/terraform-provider-kubernetes/kubernetes/test-infra/gke(master) » terraform init                                                                                                 alex@alexs-macbook

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.google: version = "~> 1.20"
* provider.local: version = "~> 1.1"
* provider.random: version = "~> 2.0"
* provider.template: version = "~> 1.0"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
------------------------------------------------------------

[pdecat]

Ok, same as me, so not related to the provider version.

What version of kubernetes do you pass when creating the GKE cluster?

[pdecat]

Oh, I'm not using the generated kubeconfig file, I use the one from gcloud container clusters get-credentials.

[alexsomesan]

1.11.5-gke.5

[pdecat]

I've submitted #277 to fix other issues regarding the environment variables setup.

[pdecat]

I've created a new test cluster and can confirm it works without requiring additional changes when properly following the instructions from the README:

export KUBE_HOST=https://$(terraform output kube_endpoint)
export KUBE_USER=$(terraform output kube_username)
export KUBE_PASSWORD=$(terraform output kube_password)
export KUBE_CLIENT_CERT_DATA="$(terraform output kube_client_certificate_b64 | base64 -d -)"
export KUBE_CLIENT_KEY_DATA="$(terraform output kube_client_key_b64 | base64 -d -)"
export KUBE_CLUSTER_CA_CERT_DATA="$(terraform output kube_cluster_ca_certificate_b64 | base64 -d -)"
export GOOGLE_ZONE=$(terraform output google_zone)

# TESTARGS="-run '^TestAccKubernetesRole_'" make testacc
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test $(go list ./... |grep -v 'vendor') -v -run '^TestAccKubernetesRole_' -timeout 120m
?       github.com/terraform-providers/terraform-provider-kubernetes    [no test files]
=== RUN   TestAccKubernetesRole_basic
--- PASS: TestAccKubernetesRole_basic (3.05s)
=== RUN   TestAccKubernetesRole_importBasic
--- PASS: TestAccKubernetesRole_importBasic (1.61s)
=== RUN   TestAccKubernetesRole_generatedName
--- PASS: TestAccKubernetesRole_generatedName (1.45s)
PASS
ok      github.com/terraform-providers/terraform-provider-kubernetes/kubernetes 6.141s

Thanks @alexsomesan and sorry for the noise!