Why does terraform apply fail?

[HsnVahedi]

I'm trying to follow the tutorial, But terraform apply fails.

Steps to reproduce:

1. Create an aws account.
2. Create access keys (access key ID and secret access key)
3. Run aws-cli docker image:
   docker run --rm -it --entrypoint bash amazon/aws-cli:latest
4. In the running container, install terraform.
5. Provide your access keys:
   aws configure
6. Clone the repository:
   git clone https://github.com/hashicorp/learn-terraform-provision-eks-cluster
7. Run terraform apply:
   cd learn-terraform-provision-eks-cluster && terraform apply --auto-approve

I have provided my root user access keys (aws configure) and did the above and now the eks is created and the autoscaling groups are also created and ec2 instances are running. But the terraform apply fails with this output:

.
.
.
module.eks.null_resource.wait_for_cluster[0]: Still creating... [50s elapsed]
module.eks.aws_autoscaling_group.workers[0]: Still creating... [40s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Still creating... [40s elapsed]
module.eks.aws_autoscaling_group.workers[0]: Creation complete after 41s [id=education-eks-Lof9Mf4j-worker-group-120210315132910683800000014]
module.eks.null_resource.wait_for_cluster[0]: Still creating... [1m0s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Still creating... [50s elapsed]
module.eks.null_resource.wait_for_cluster[0]: Creation complete after 1m5s [id=9027141840267261715]
data.aws_eks_cluster_auth.cluster: Reading...
data.aws_eks_cluster.cluster: Reading...
data.aws_eks_cluster_auth.cluster: Read complete after 0s [id=education-eks-Lof9Mf4j]
data.aws_eks_cluster.cluster: Read complete after 0s [id=education-eks-Lof9Mf4j]
module.eks.kubernetes_config_map.aws_auth[0]: Creating...
module.eks.aws_autoscaling_group.workers[1]: Still creating... [1m0s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Still creating... [1m10s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Creation complete after 1m12s [id=education-eks-Lof9Mf4j-worker-group-220210315132910701500000015]

Error: Unauthorized

And here is the last snapshot in logs:

-----------------------------------------------------: timestamp=2021-03-15T15:17:57.245Z
2021-03-15T15:17:57.246Z [INFO]  plugin.terraform-provider-aws_v3.25.0_x5: 2021/03/15 15:17:57 [DEBUG] [aws-sdk-go] <DescribeAutoScalingGroupsResponse xmlns="http://autoscaling.amazonaws.com/doc/2011-01-01/">
  <DescribeAutoScalingGroupsResult>
    <AutoScalingGroups>
      <member>
        <HealthCheckType>EC2</HealthCheckType>
        <Instances>
          <member>
            <LaunchConfigurationName>education-eks-BuAAAvsR-worker-group-220210315151705272900000013</LaunchConfigurationName>
            <LifecycleState>InService</LifecycleState>
            <InstanceId>i-04fb7f82204a614b1</InstanceId>
            <HealthStatus>Healthy</HealthStatus>
            <InstanceType>t2.medium</InstanceType>
            <ProtectedFromScaleIn>false</ProtectedFromScaleIn>
            <AvailabilityZone>us-east-2c</AvailabilityZone>
          </member>
        </Instances>
        <TerminationPolicies>
          <member>Default</member>
        </TerminationPolicies>
        <DefaultCooldown>300</DefaultCooldown>
        <AutoScalingGroupARN>arn:aws:autoscaling:us-east-2:008082804869:autoScalingGroup:66e234e4-1b87-4bb7-aef5-eae21601a813:autoScalingGroupName/education-eks-BuAAAvsR-worker-group-220210315151716231500000014</AutoScalingGroupARN>
        <EnabledMetrics/>
        <MaxSize>3</MaxSize>
        <AvailabilityZones>
          <member>us-east-2a</member>
          <member>us-east-2b</member>
          <member>us-east-2c</member>
        </AvailabilityZones>
        <TargetGroupARNs/>
onfigurationName>education-eks-BuAAAvsR-worker-group-220210315151705272900000013</LaunchConfigurationName>
        <AutoScalingGroupName>education-eks-BuAAAvsR-worker-group-220210315151716231500000014</AutoScalingGroupName>
        <HealthCheckGracePeriod>300</HealthCheckGracePeriod>
        <NewInstancesProtectedFromScaleIn>false</NewInstancesProtectedFromScaleIn>
        <CreatedTime>2021-03-15T15:17:16.848Z</CreatedTime>
        <MinSize>1</MinSize>
        <LoadBalancerNames/>
        <Tags>
          <member>
            <ResourceId>education-eks-BuAAAvsR-worker-group-220210315151716231500000014</ResourceId>
            <PropagateAtLaunch>true</PropagateAtLaunch>
            <Value>training</Value>
            <Key>Environment</Key>
            <ResourceType>auto-scaling-group</ResourceType>
          </member>
          <member>
            <ResourceId>education-eks-BuAAAvsR-worker-group-220210315151716231500000014</ResourceId>
            <PropagateAtLaunch>true</PropagateAtLaunch>
            <Value>terraform-aws-modules</Value>
            <Key>GithubOrg</Key>
            <ResourceType>auto-scaling-group</ResourceType>
          </member>
          <member>
            <ResourceId>education-eks-BuAAAvsR-worker-group-220210315151716231500000014</ResourceId>
            <PropagateAtLaunch>true</PropagateAtLaunch>
            <Value>terraform-aws-eks</Value>
            <Key>GithubRepo</Key>
            <ResourceType>auto-scaling-group</ResourceType>
          </member>
          <member>
            <ResourceId>education-eks-BuAAAvsR-worker-group-220210315151716231500000014</ResourceId>
            <PropagateAtLaunch>true</PropagateAtLaunch>
            <Value>education-eks-BuAAAvsR-worker-group-2-eks_asg</Value>
            <Key>Name</Key>
            <ResourceType>auto-scaling-group</ResourceType>
          </member>
          <member>
            <ResourceId>education-eks-BuAAAvsR-worker-group-220210315151716231500000014</ResourceId>
            <PropagateAtLaunch>true</PropagateAtLaunch>
alue>owned</Value>
            <Key>k8s.io/cluster/education-eks-BuAAAvsR</Key>
            <ResourceType>auto-scaling-group</ResourceType>
          </member>
          <member>
            <ResourceId>education-eks-BuAAAvsR-worker-group-220210315151716231500000014</ResourceId>
            <PropagateAtLaunch>true</PropagateAtLaunch>
            <Value>owned</Value>
            <Key>kubernetes.io/cluster/education-eks-BuAAAvsR</Key>
            <ResourceType>auto-scaling-group</ResourceType>
          </member>
        </Tags>
        <ServiceLinkedRoleARN>arn:aws:iam::008082804869:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling</ServiceLinkedRoleARN>
        <SuspendedProcesses>
          <member>
            <ProcessName>AZRebalance</ProcessName>
            <SuspensionReason>User suspended at 2021-03-15T15:17:56Z</SuspensionReason>
          </member>
        </SuspendedProcesses>
        <DesiredCapacity>1</DesiredCapacity>
        <VPCZoneIdentifier>subnet-07746a16929a6d25e,subnet-0fabb7770ed770def,subnet-0dd048ae9087d8308</VPCZoneIdentifier>
      </member>
    </AutoScalingGroups>
  </DescribeAutoScalingGroupsResult>
  <ResponseMetadata>
    <RequestId>d5dab1f1-a153-45ec-a6ba-990de96cc822</RequestId>
  </ResponseMetadata>
</DescribeAutoScalingGroupsResponse>: timestamp=2021-03-15T15:17:57.246Z
2021/03/15 15:17:57 [WARN] Provider "registry.terraform.io/hashicorp/aws" produced an unexpected new value for module.eks.aws_autoscaling_group.workers[0], but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .placement_group: was null, but now cty.StringVal("")
      - .capacity_rebalance: was null, but now cty.False
2021/03/15 15:17:57 [WARN] Provider "registry.terraform.io/hashicorp/aws" produced an unexpected new value for module.eks.aws_autoscaling_group.workers[1], but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .placement_group: was null, but now cty.StringVal("")
      - .capacity_rebalance: was null, but now cty.False
2021-03-15T15:17:57.342Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-03-15T15:17:57.349Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.25.0/linux_amd64/terraform-provider-aws_v3.25.0_x5 pid=481
2021-03-15T15:17:57.349Z [DEBUG] plugin: plugin exited
2021-03-15T15:17:57.416Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-03-15T15:17:57.416Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-03-15T15:17:57.418Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-03-15T15:17:57.418Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/kubernetes/2.0.1/linux_amd64/terraform-provider-kubernetes_v2.0.1_x5 pid=507
2021-03-15T15:17:57.418Z [DEBUG] plugin: plugin exited
2021-03-15T15:17:57.418Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=408
2021-03-15T15:17:57.418Z [DEBUG] plugin: plugin exited
2021-03-15T15:17:57.418Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=359
2021-03-15T15:17:57.418Z [DEBUG] plugin: plugin exited

Any suggestions?

[im2nguyen]

Thanks for reporting!

This is strange and seems similar to this issue and this one.

Did terraform finish provisioning all the resources? Can you re-run terraform apply?

[HsnVahedi]

@im2nguyen I reproduced the situation and ran terraform apply again. I get the exact error and here is the last snapshot in logs:

-----------------------------------------------------: timestamp=2021-03-15T16:55:08.676Z
2021/03/15 16:55:10 [DEBUG] module.eks.kubernetes_config_map.aws_auth[0]: apply errored, but we're indicating that via the Error pointer rather than returning it: Unauthorized
2021-03-15T16:55:10.940Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-03-15T16:55:10.941Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-03-15T16:55:10.941Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-03-15T16:55:10.948Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=1191
2021-03-15T16:55:10.948Z [DEBUG] plugin: plugin exited
2021-03-15T16:55:10.948Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=1143
2021-03-15T16:55:10.948Z [DEBUG] plugin: plugin exited
2021-03-15T16:55:10.951Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/kubernetes/2.0.2/linux_amd64/terraform-provider-kubernetes_v2.0.2_x5 pid=1281
2021-03-15T16:55:10.951Z [DEBUG] plugin: plugin exited

[im2nguyen]

Can you check if the IAM user you're using to provision the cluster has the right permissions?

[HsnVahedi]

@im2nguyen I'm providing my root user access keys (when running aws configure). So I suppose the root user can do everything, right?

[im2nguyen]

It should... hrm... I'm stumped but will try digging into it more (unable to reproduce on my end). In the meantime, can you raise an issue in the terraform-aws-eks repo?

[kaykhancheckpoint]

@im2nguyen Im having the same problem right now, did you find a fix for this?

[HsnVahedi]

Have you provided "region" when authenticating with "aws configure" ? @kaykhancheckpoint

My problem was because of that.

[kaykhancheckpoint]

yes i did, i am actually providing it in the terraform provider aws config. I think my error is a bit more specific its to do with the auth not being created

Error: Unauthorized

  on .terraform/modules/eks/aws_auth.tf line 65, in resource "kubernetes_config_map" "aws_auth":
  65: resource "kubernetes_config_map" "aws_auth" { 

Everything else creates fine.

[alanszlosek]

This should now be fixed by #64 and #65. Please let us know if you are still having issues.