Terraform state secret contents are wiped if a plan fails with an error.

[mp998]

Describe the bug
The secret containing the state gets wiped and starts fresh if a plan or apply fails. This results in orphaned resources.

To Reproduce
Steps to reproduce the behavior:

1. Create a CloudResource or Configuration
2. Check the contents of the state secret for the configuration and you'll see that it's contents is only 159 bytes
3. Edit the annotation for the CloudResource or Configuration to apply the changes
4. Check the contents of the state secret for the configuration and you'll see that it is much larger that the initial 159 bytes
5. Make a change to the CloudResource or Configuration that will result in an error in the plan.
6. Check the contents of the state secret for the configuration again and you'll see that its contents as back to only 159 bytes.

Expected behavior
The existing state should persist even if a plan or apply fails.

[mp998]

I configured a custom state backend using an s3 bucket and this doesn't appear to happen with that configuration so the default kubernetes state backend might be the only backend that is affected.

[gambol99]

Hi @mp998 .. thanks for raising this one … were gonna try and replicate this one today ..

[gambol99]

Hi @mp998

Sorry the delay .. I’m having trouble replicating this issue. Here’s what i have done so far

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.29.2) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️ 
…

# Installed the latest release of the controller 
$ helm repo add appvia https://terranetes-controller.appvia.io
$ helm repo update
$ helm install -n terraform-system terranetes-controller appvia/terranetes-controller --create-namespace

# verified the version 
$ kubectl -n terraform-system get deployments.apps -o yaml | grep image:
          image: ghcr.io/appvia/terranetes-controller:v0.4.9

# Sourced in some environment variableds and created an aws credential to the terraform namespace
$ make aws-credentials 

# Created a provider 
$ kubectl apply -f examples/provider.yaml
provider.terraform.appvia.io/aws created
provider.terraform.appvia.io/aws-irsa created

# Updated the bucket name to something unique and applied to a namespace 
$ kubectl create namespace apps
namespace/apps created

$ git diff examples/
diff --git a/examples/configuration.yaml b/examples/configuration.yaml
index 9950586c..bf544715 100644
--- a/examples/configuration.yaml
+++ b/examples/configuration.yaml
@@ -69,7 +69,7 @@ spec:
   variables:
     # -- The name of the bucket. If omitted, Terraform will assign a random,
     # unique name
-    bucket: terranetes-controller-ci-bucket
+    bucket: terranetes-controller-ci-XXXXXXXX
     # -- The canned ACL to apply
     acl: private
     # -- Manage S3 Bucket Ownership Controls on this bucket.

$ kubectl -n apps apply -f examples/configuration.yaml
configuration.terraform.appvia.io/bucket create
NAME     MODULE                                                                             SECRET   DRIFT DETECTION   ESTIMATED     SYNCHRONIZED   AGE
bucket   https://github.com/terraform-aws-modules/terraform-aws-s3-bucket.git?ref=v3.10.1   test                       Not Enabled   OutOfSync      1m

# Approved the configuration 
$ tnctl approve configuration -n apps bucket

# The plans and applied worked .. I then altered the `spec.variables` in the configuration to have a new key `hello: world`, just to trigger off a new plan and apply. Adding a new command `bin/tnctl state get -n apps bucket | jq` to grab, decode and print the terraform state secret. The secret at this one look good.

# Created an error in the plan stage 
$ git diff examples/configuration.yaml
diff --git a/examples/configuration.yaml b/examples/configuration.yaml
index 9950586c..8c7145ca 100644
--- a/examples/configuration.yaml
+++ b/examples/configuration.yaml
@@ -69,7 +69,7 @@ spec:
   variables:
     # -- The name of the bucket. If omitted, Terraform will assign a random,
     # unique name
-    bucket: terranetes-controller-ci-bucket
+    bucket: terranetes-controller-ci-rohith-112
     # -- The canned ACL to apply
     acl: private
     # -- Manage S3 Bucket Ownership Controls on this bucket.
@@ -78,7 +78,7 @@ spec:
     object_ownership: ObjectWriter
     # -- Map containing versioning configuration
     versioning:
-      enabled: true
+      enabled: THIS_IS_BAD
     # -- Whether Amazon S3 should block public ACLs for this bucket
     block_public_acls: true
     # -- Whether Amazon S3 should block public bucket policies for this bucket

# Apply the update 
$ kubectl -n apps apply -f examples/configuration.yaml
configuration.terraform.appvia.io/bucket configured

# The plan failed 
$ k -n apps get po
NAME                   READY   STATUS      RESTARTS   AGE
bucket-1-apply-chwpk   0/1     Completed   0          49m
bucket-1-plan-sfxhn    0/1     Completed   0          50m
bucket-2-apply-sxj68   0/1     Completed   0          44m
bucket-2-plan-k9tgd    0/1     Completed   0          45m
bucket-3-plan-7c5wd    0/1     Error       0          43m
bucket-4-plan-kbcms    0/1     Completed   0          9m31s
bucket-5-plan-sb49c    0/1     Error       0          50s

$ kubectl -n apps logs bucket-5-plan-sb49c --tail=20
    ├────────────────
    │ while calling try(expressions...)
    │ var.versioning is map of string with 1 element
    │ var.versioning["enabled"] is "THIS_IS_BAD"

Call to function "try" failed: no expression succeeded:
- Incorrect condition type (at main.tf:144,18-43)
  The condition expression must be of type bool.
- Invalid index (at main.tf:144,92-102)
  The given key does not identify an element in this collection value.
- Invalid index (at main.tf:144,157-167)
  The given key does not identify an element in this collection value.

At least one expression must produce a successful result.
time="2024-04-30T12:43:56Z" level=error msg="failed to execute command successfully" command=0 error="exit status 1"
Error: exit status 1
[Error] exit status 1
[build] completed
time="2024-04-30T12:43:56Z" level=error msg="failed to execute command successfully" command=0 error="exit status 1"
[error] Job failure occurred, view the Terraform logs for more info.

# I rechecked the secret and it was fine 
$ bin/tnctl state get -n apps bucket | jq | tail -n 30
      "type": "aws_s3_bucket_versioning",
      "name": "this",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "index_key": 0,
          "schema_version": 0,
          "attributes": {
            "bucket": "terranetes-controller-ci-XXXXXXXX”,
            "expected_bucket_owner": "",
            "id": "terranetes-controller-ci-XXXXXXXX”,
            "mfa": null,
            "versioning_configuration": [
              {
                "mfa_delete": "",
                "status": "Enabled"
              }
            ]
          },
          "sensitive_attributes": [],
          "private": "bnVsbA==",
          "dependencies": [
            "aws_s3_bucket.this"
          ]
        }
      ]
    }
  ],
  "check_results": null
}

[gambol99]

Closing the issue, as we were unable to replicate the behavior - Feel free to open @mp998 if the above still doesn’t work for you