Empty state after terraform refresh, when external resource still exist

[vaspahomov]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

bash-5.1$ terraform -v
Terraform v1.2.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.72.0

Affected Resource(s)

All resources, but this time on aws_nat_gateway

• aws_nat_gateway

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

{
   "provider":{
      "aws":{
         "access_key":"<secret>",
         "region":"us-east-2",
         "secret_key":"<secret>",
         "token":""
      }
   },
   "resource":{
      "aws_nat_gateway":{
         "nat-gw-vaspahomov-testing-public-b":{
            "allocation_id":"eipalloc-0081931dc37bd4679",
            "connectivity_type":"public",
            "lifecycle":{
               "prevent_destroy":true
            },
            "subnet_id":"subnet-00adfbd49eaa70973",
            "tags":{
               "Name":"nat-gw-vaspahomov-testing-public-b",
               "crossplane-kind":"natgateway.ec2.aws.jet.crossplane.io",
               "crossplane-name":"nat-gw-vaspahomov-testing-public-b",
               "crossplane-providerconfig":"default"
            }
         }
      }
   },
   "terraform":{
      "required_providers":{
         "aws":{
            "source":"hashicorp/aws",
            "version":"3.75.2"
         }
      }
   }
}

Debug Output

Log from terraform refresh - https://gist.github.com/vaspahomov/94bff9821713b0d374888cf0ecdf29bd

Panic Output

Expected Behavior

State still exists after refresh and will be like -

{
  "version": 4,
  "terraform_version": "1.2.5",
  "serial": 1,
  "lineage": "8294f584-e139-450b-a1d3-30d4b50eab19",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "aws_nat_gateway",
      "name": "nat-gw-vaspahomov-testing-public-b",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "allocation_id": "eipalloc-0081931dc37bd4679",
            "connectivity_type": "public",
            "id": "nat-0b3adcf4b99392853",
            "network_interface_id": "eni-0e4763855684f6071",
            "private_ip": "10.0.16.6",
            "public_ip": "3.138.255.64",
            "subnet_id": "subnet-00adfbd49eaa70973",
            "tags": {
              "Name": "nat-gw-vaspahomov-testing-public-b",
              "crossplane-kind": "natgateway.ec2.aws.jet.crossplane.io",
              "crossplane-name": "nat-gw-vaspahomov-testing-public-b",
              "crossplane-providerconfig": "default"
            },
            "tags_all": {
              "Name": "nat-gw-vaspahomov-testing-public-b",
              "crossplane-kind": "natgateway.ec2.aws.jet.crossplane.io",
              "crossplane-name": "nat-gw-vaspahomov-testing-public-b",
              "crossplane-providerconfig": "default"
            }
          },
          "sensitive_attributes": [],
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}

Actual Behavior

State disappeared.

{
  "version": 4,
  "terraform_version": "1.2.5",
  "serial": 1,
  "lineage": "8294f584-e139-450b-a1d3-30d4b50eab19",
  "outputs": {},
  "resources": [  ]
}

Steps to Reproduce

I can not reproduce stable. The problem happens sometimes after terraform apply -refresh-only -auto-approve -input=false -lock=false -json

Important Factoids

I've found in log that provider 'produced an unexpected new value during refresh'

{"@level":"warn","@message":"Provider \"registry.terraform.io/hashicorp/aws\" produced an unexpected new value for aws_nat_gateway.nat-gw-vaspahomov-testing-public-b during refresh.\n      - Root resource was present, but now absent","@timestamp":"2022-07-28T08:22:20.173631Z"}
{"@level":"info","@message":"aws_nat_gateway.nat-gw-vaspahomov-testing-public-b: Refresh complete","@module":"terraform.ui","@timestamp":"2022-07-28T08:22:20.173680Z","hook":{"resource":{"addr":"aws_nat_gateway.nat-gw-vaspahomov-testing-public-b","module":"","resource":"aws_nat_gateway.nat-gw-vaspahomov-testing-public-b","implied_provider":"aws","resource_type":"aws_nat_gateway","resource_name":"nat-gw-vaspahomov-testing-public-b","resource_key":null}},"type":"refresh_complete"}
{"@level":"trace","@message":"NodeAbstractResouceInstance.writeResourceInstanceState to refreshState for aws_nat_gateway.nat-gw-vaspahomov-testing-public-b","@timestamp":"2022-07-28T08:22:20.173763Z"}
{"@level":"trace","@message":"NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for aws_nat_gateway.nat-gw-vaspahomov-testing-public-b","@timestamp":"2022-07-28T08:22:20.173786Z"}
{"@level":"trace","@message":"NodeAbstractResouceInstance.writeResourceInstanceState to workingState for aws_nat_gateway.nat-gw-vaspahomov-testing-public-b","@timestamp":"2022-07-28T08:22:20.173808Z"}
{"@level":"trace","@message":"NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for aws_nat_gateway.nat-gw-vaspahomov-testing-public-b","@timestamp":"2022-07-28T08:22:20.173825Z"}

References

• #0000

[vaspahomov]

Same problem on latest provider version (v4.23.0) - https://gist.github.com/vaspahomov/61e29f53115ca02e69d7fe07b295663a

[justinretzolk]

Hey @vaspahomov 👋 Thank you for taking the time to raise this! I took a quick look at the debug logs and happened to notice this line:

{"@level":"debug","@message":"New state was assigned lineage \"70a64b6c-df8c-fa50-367a-840210016653\"","@timestamp":"2022-07-28T10:10:44.456262Z"}

With that line in mind, was this refresh performed after the resources have been created, or is this a new configuration where you're running terraform refresh before any terraform apply has occurred?

[vaspahomov]

Hi @justinretzolk!

I've created resource with terraform apply. After that I'm calling refresh repeatedly. And sometimes (very rarely) refresh ends with 'empty state' (resources field empty)

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.