terraform test for module with k8s_secret resource not failing correctly

[pindge]

Terraform Version

(base) ubuntu@:~/datakube/modules/terraform_test$ terraform version
Terraform v1.1.4
on linux_amd64

Terraform Configuration Files

the main.tf file

variable "input" {
  type = string
}

variable "namespace" {
  type = string
  default = "processing"
}

variable "test_process" {
  type = string
  default = "test"
}


output "foo" {
  value = "foo value ${var.input}"
}

output "k8s_creds" {
    value = kubernetes_secret.uplifted_user_creds.metadata[0].name
}

resource "kubernetes_secret" "uplifted_user_creds" {
  metadata {
    name      = "${var.namespace}-aws-creds-${var.test_process}"
    namespace = var.namespace
  }

  data = {
  }

  type = "Opaque"
}

the test.tf file

terraform {
  required_providers {
    test = {
      source = "terraform.io/builtin/test"
    }
  }
}

module "main" {
  source = "../.."

  input = "boop"
}

resource "test_assertions" "foo" {
  component = "foo"

  equal "output" {
    description = "output \"foo\" value"
    got         = module.main.foo
    want        = "foo value boop"
  }
}

resource "test_assertions" "creds" {
  component = "cred"

  equal "output" {
    description = "output \"foo\" value"
    got         = module.main.k8s_creds
    want        = "foo value boop"
  }
}

Debug Output

(base) ubuntu@:~/datakube/modules/terraform_test$ terraform test
╷
│ Warning: The "terraform test" command is experimental
│ 
│ We'd like to invite adventurous module authors to write integration tests for their modules using this command, but all of the behaviors of this command are currently experimental and may change based on feedback.
│ 
│ For more information on the testing experiment, including ongoing research goals and avenues for feedback, see:
│     https://www.terraform.io/docs/language/modules/testing-experiment.html
╵
Success! All of the test assertions passed.

Expected Behavior

the test_assertions creds should fail as the output of kubernetes_secret.uplifted_user_creds.metadata[0].name should be processing-aws-creds-test.

Actual Behavior

the test for it passed, indicating terraform test sees the output of kubernetes_secret.uplifted_user_creds.metadata[0].name equal to foo value boop

Steps to Reproduce

terraform test

Additional Context

Nil

References

N/A

[alisdair]

Hi @pindge, thanks for reporting this.

When I run your tests, I also see the same behaviour, which I agree is confusing. What appears to be happening here is that the Kubernetes provider is erroring on apply, which terraform test harness is not reporting. The result is that the creds assertion is skipped rather than failed.

This does seem like something we ought to fix in the terraform test implementation, as it's surprising behaviour. I'm not yet sure of the root cause.

Are you able to share the full debug logs using TF_LOG=trace terraform test? You may see Kubernetes API errors in there. Running locally against a blank minikube, I see:

2022-02-04T09:31:06.512-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: -----------------------------------------------------
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: 2022/02/04 09:31:06 [DEBUG] Kubernetes API Response Details:
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: ---[ RESPONSE ]--------------------------------------
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: HTTP/2.0 404 Not Found
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: Content-Length: 200
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: Cache-Control: no-cache, private
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: Content-Type: application/json
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: Date: Fri, 04 Feb 2022 14:31:06 GMT
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: X-Kubernetes-Pf-Flowschema-Uid: 57a76d72-653a-42dc-97c7-6a099dee5b6d
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: X-Kubernetes-Pf-Prioritylevel-Uid: e24af498-cbf2-4f62-9f95-e288456122da
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: {
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:  "kind": "Status",
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:  "apiVersion": "v1",
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:  "metadata": {},
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:  "status": "Failure",
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:  "message": "namespaces \"processing\" not found",
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:  "reason": "NotFound",
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:  "details": {
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:   "name": "processing",
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:   "kind": "namespaces"
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:  },
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:  "code": 404
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: }
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5:
2022-02-04T09:31:06.577-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.7.1_x5: -----------------------------------------------------
2022-02-04T09:31:06.577-0500 [TRACE] maybeTainted: module.main.kubernetes_secret.uplifted_user_creds encountered an error during creation, so it is now marked as tainted
2022-02-04T09:31:06.577-0500 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.main.kubernetes_secret.uplifted_user_creds
2022-02-04T09:31:06.577-0500 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for module.main.kubernetes_secret.uplifted_user_creds
2022-02-04T09:31:06.577-0500 [TRACE] evalApplyProvisioners: module.main.kubernetes_secret.uplifted_user_creds is tainted, so skipping provisioning
2022-02-04T09:31:06.577-0500 [TRACE] maybeTainted: module.main.kubernetes_secret.uplifted_user_creds was already tainted, so nothing to do
2022-02-04T09:31:06.577-0500 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.main.kubernetes_secret.uplifted_user_creds
2022-02-04T09:31:06.577-0500 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for module.main.kubernetes_secret.uplifted_user_creds
2022-02-04T09:31:06.577-0500 [ERROR] vertex "module.main.kubernetes_secret.uplifted_user_creds" error: namespaces "processing" not found
2022-02-04T09:31:06.577-0500 [TRACE] vertex "module.main.kubernetes_secret.uplifted_user_creds": visit complete, with errors

[alisdair]

Noting that the confusing behaviour of terraform test when resources error out is covered by this similar issue: #30133