Terraform ends up with blank state file after error in apply

[gek0]

Terraform Version

Terraform v0.12.13
+ provider.aws v2.27.0

Terraform Configuration Files

# module for creating account and similiar resources....

resource "aws_organizations_organizational_unit" "environment_ou" { ... }
resource "aws_organizations_account" "account" { ... }
data "aws_iam_policy_document" "account_scp_policy" { ... }
resource "aws_organizations_policy" "account_scp_policy" { ... }
resource "aws_organizations_policy_attachment" "account_scp_policy" { ... }

# here another module is called (inside the one for account) that creates keypairs and security groups in all regions...
module "multi-region-resources" { ... }

# providers for all those resources are declared per region as it's needed, for eg.
provider "aws" {
  profile = var.profile
  region  = "us-east-1"
  alias   = "us-east-1"

  assume_role {
    role_arn     = local.formatted_role_arn
    session_name = var.account_assumed_role_session_name # got this from module call
  }
}

Apply Output

aws_organizations_organizational_unit.project: Creating...
module.prod.aws_organizations_policy.account_scp_policy: Creating...
module.prod.aws_organizations_policy.account_scp_policy: Creation complete after 2s [id=<REDACTED>]
aws_organizations_organizational_unit.project: Creation complete after 2s [id=<REDACTED>]
module.prod.aws_organizations_organizational_unit.environment_ou: Creating...
module.prod.aws_organizations_organizational_unit.environment_ou: Creation complete after 3s [id=<REDACTED>]
module.prod.aws_organizations_account.account: Creating...
module.prod.aws_ram_principal_association.ou_shared_resource[0]: Creating...
module.prod.aws_organizations_account.account: Still creating... [10s elapsed]
module.prod.aws_organizations_account.account: Creation complete after 15s [id=<REDACTED>]
module.prod.aws_organizations_policy_attachment.account_scp_policy: Creating...
module.prod.aws_organizations_policy_attachment.account_scp_policy: Creation complete after 1s [id=<REDACTED>:<REDACTED>]
module.prod.aws_iam_role.account_admin_role: Creating...
module.prod.aws_iam_role.account_team_role["team_one"]: Creating...
module.prod.aws_iam_role.account_admin_role: Creation complete after 1s [id=<REDACTED>]
module.prod.aws_iam_role.account_team_role["team_one"]: Creation complete after 1s [id=<REDACTED>]
module.prod.aws_iam_role_policy.prod_account_role_policy: Creating...
module.prod.aws_iam_role_policy.prod_account_role_policy: Creation complete after 1s [id=<REDACTED>:AdministratorAccess]
module.prod.aws_iam_role_policy.team_policy["team_one"]: Creating...
module.prod.module.multi-region-resources.aws_key_pair.key_pair_ca_central_1: Creating...
module.prod.aws_iam_role_policy.team_policy["team_one"]: Creation complete after 2s [id=<REDACTED>:ServicesAccess]
module.prod.module.multi-region-resources.aws_key_pair.key_pair_us_east_2: Creating...
module.prod.module.multi-region-resources.aws_key_pair.key_pair_eu_central_1: Creating...
module.prod.module.multi-region-resources.module.ec2_default_security_group_eu_central_1.module.default_sg.aws_security_group.this[0]: Creating...
module.prod.module.multi-region-resources.aws_key_pair.key_pair_eu_west_1: Creating...
module.prod.module.multi-region-resources.aws_key_pair.key_pair_us_west_1: Creating...
module.prod.module.multi-region-resources.aws_key_pair.key_pair_us_west_1: Creation complete after 3s [id=ib-default]
module.prod.module.multi-region-resources.aws_key_pair.key_pair_ap_northeast_2: Creating...
module.prod.module.multi-region-resources.aws_key_pair.key_pair_ap_northeast_2: Creation complete after 2s [id=ib-default]

Error: The role "arn:aws:iam::<REDACTED>:role/<REDACTED>_prod_admin" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

  on .terraform/modules/prod.multi-region-resources/02-providers.tf line 3, in provider "aws":
   3: provider "aws" {

Error: The role "arn:aws:iam::<REDACTED>:role/<REDACTED>_prod_admin" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

  on .terraform/modules/prod.multi-region-resources/02-providers.tf line 201, in provider "aws":
 201: provider "aws" {

Error: Error import KeyPair: OptInRequired: You are not subscribed to this service. Please go to http://aws.amazon.com to subscribe.
	status code: 401, request id: 1572e382-c3c0-47d2-a5dc-cd6d29f6c923

  on .terraform/modules/prod.multi-region-resources/10-main.tf line 28, in resource "aws_key_pair" "key_pair_ca_central_1":
  28: resource "aws_key_pair" "key_pair_ca_central_1" {



Error: Error import KeyPair: OptInRequired: You are not subscribed to this service. Please go to http://aws.amazon.com to subscribe.
	status code: 401, request id: 20654046-7fab-4ecd-8c8b-832fa94152f4

  on .terraform/modules/prod.multi-region-resources/10-main.tf line 34, in resource "aws_key_pair" "key_pair_eu_central_1":
  34: resource "aws_key_pair" "key_pair_eu_central_1" {



Error: Error import KeyPair: OptInRequired: You are not subscribed to this service. Please go to http://aws.amazon.com to subscribe.
	status code: 401, request id: 4b290a9b-d5cb-42a9-96c8-f822690d9e32

  on .terraform/modules/prod.multi-region-resources/10-main.tf line 46, in resource "aws_key_pair" "key_pair_eu_west_1":
  46: resource "aws_key_pair" "key_pair_eu_west_1" {



Error: Error import KeyPair: OptInRequired: You are not subscribed to this service. Please go to http://aws.amazon.com to subscribe.
	status code: 401, request id: 3af2a504-2b9b-4c65-b29a-2d07924c64c9

  on .terraform/modules/prod.multi-region-resources/10-main.tf line 76, in resource "aws_key_pair" "key_pair_us_east_2":
  76: resource "aws_key_pair" "key_pair_us_east_2" {



Error: Error associating principal with RAM resource share: UnknownResourceException: OrganizationalUnit <REDACTED> <REDACTED> could not be found.
	status code: 400, request id: 388f5849-5c48-40c9-81c0-9f5811b93376

  on .terraform/modules/prod/10-main.tf line 127, in resource "aws_ram_principal_association" "ou_shared_resource":
 127: resource "aws_ram_principal_association" "ou_shared_resource" {



Error: Error creating Security Group: OptInRequired: You are not subscribed to this service. Please go to http://aws.amazon.com to subscribe.
	status code: 401, request id: 84ef146d-4f5a-419c-a19d-51afa6185e2e

  on .terraform/modules/prod.multi-region-resources.ec2_default_security_group_eu_central_1.default_sg/terraform-aws-modules-terraform-aws-security-group-a332a3b/main.tf line 15, in resource "aws_security_group" "this":
  15: resource "aws_security_group" "this" {

Crash Output

• there is no crash

Expected Behavior

• I would expect that I can create resources using assumed role since the role is created and that is confirmed

• but those fail with errors seen in the apply output

• is this related to AWS API and how it is asynchronous?

 Error: Error import KeyPair: OptInRequired: You are not subscribed to this service. Please go to http://aws.amazon.com to subscribe

Actual Behavior

• some resources that have been marked as created really are, but nothing is saved to terraform state after that error occurs!

Steps to Reproduce

• terraform plan -out=file and terraform apply file

Additional Context

• we are running terraform using Atlantis in our CI system but this has been known to happen a few times even using terraform without any additional wrapper
• our biggest issue and unknown problem is that terraform should save its state and not create some resources but never save them to state!
• terraform state file only contains output variables structure and nothing else so created resources need to be imported (aws_organizations_account is especially fragile since it cannot be destroyed with pure IaaC)...
• after the import, terraform no longer has problems with creating all other resources - maybe AWS has really created all others now in the backend?
• important note, this does not happen every time so it's not deterministic from our view. Happened 2/10 times (and with lower version) in a period of a few weeks

References