eksctl create cluster failed because of MalformedPolicyDocument

[Vieufoux]

I am try to use eksctl to start a eks cluster by using eksctl create cluster -f cluster.yaml but i got the below error:

AWS::IAM::Policy/PolicyEBS: CREATE_FAILED – "Partition \"aws\" is not valid for resource \"arn:aws:ec2:*:*:volume/*\". (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 78a4be11-60d5-4444-8669-c3375d453948; Proxy: null)"

Here is the full log:

2021-10-09 06:59:40 [ℹ]  eksctl version 0.69.0
2021-10-09 06:59:40 [ℹ]  using region cn-northwest-1
2021-10-09 06:59:40 [ℹ]  setting availability zones to [cn-northwest-1a cn-northwest-1b cn-northwest-1c]
2021-10-09 06:59:40 [ℹ]  subnets for cn-northwest-1a - public:192.168.0.0/19 private:192.168.96.0/19
2021-10-09 06:59:40 [ℹ]  subnets for cn-northwest-1b - public:192.168.32.0/19 private:192.168.128.0/19
2021-10-09 06:59:40 [ℹ]  subnets for cn-northwest-1c - public:192.168.64.0/19 private:192.168.160.0/19
2021-10-09 06:59:40 [ℹ]  nodegroup "eks-test-ng-1" will use "" [AmazonLinux2/1.21]
2021-10-09 06:59:40 [ℹ]  using EC2 key pair "eks-cn"
2021-10-09 06:59:40 [ℹ]  using Kubernetes version 1.21
2021-10-09 06:59:40 [ℹ]  creating EKS cluster "cnnx-eks-test" in "cn-northwest-1" region with managed nodes
2021-10-09 06:59:40 [ℹ]  1 nodegroup (eks-test-ng-1) was included (based on the include/exclude rules)
2021-10-09 06:59:40 [ℹ]  will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s)
2021-10-09 06:59:40 [ℹ]  will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s)
2021-10-09 06:59:40 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=cn-northwest-1 --cluster=cnnx-eks-test'
2021-10-09 06:59:40 [ℹ]  CloudWatch logging will not be enabled for cluster "cnnx-eks-test" in "cn-northwest-1"
2021-10-09 06:59:40 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=cn-northwest-1 --cluster=cnnx-eks-test'
2021-10-09 06:59:40 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "cnnx-eks-test" in "cn-northwest-1"
2021-10-09 06:59:40 [ℹ]  2 sequential tasks: { create cluster control plane "cnnx-eks-test", 2 sequential sub-tasks: { wait for control plane to become ready, create managed nodegroup "eks-test-ng-1" } }
2021-10-09 06:59:40 [ℹ]  building cluster stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 06:59:40 [ℹ]  deploying stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:00:10 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:00:40 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:01:40 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:02:40 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:03:40 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:04:40 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:05:40 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:06:40 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:07:40 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:08:40 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:09:40 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:10:40 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-cluster"
2021-10-09 07:12:41 [ℹ]  building managed nodegroup stack "eksctl-cnnx-eks-test-nodegroup-eks-test-ng-1"
2021-10-09 07:12:41 [ℹ]  deploying stack "eksctl-cnnx-eks-test-nodegroup-eks-test-ng-1"
2021-10-09 07:12:41 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-nodegroup-eks-test-ng-1"
2021-10-09 07:12:58 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-nodegroup-eks-test-ng-1"
2021-10-09 07:13:14 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-nodegroup-eks-test-ng-1"
2021-10-09 07:13:34 [ℹ]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-nodegroup-eks-test-ng-1"
2021-10-09 07:13:34 [✖]  unexpected status "ROLLBACK_COMPLETE" while waiting for CloudFormation stack "eksctl-cnnx-eks-test-nodegroup-eks-test-ng-1"
2021-10-09 07:13:34 [ℹ]  fetching stack events in attempt to troubleshoot the root cause of the failure
2021-10-09 07:13:34 [!]  AWS::EC2::SecurityGroup/SSH: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Role/NodeInstanceRole: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::EC2::LaunchTemplate/LaunchTemplate: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyFSX: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyCertManagerGetChange: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyCertManagerChangeSet: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyEBS: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyCertManagerHostedZones: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyAWSLoadBalancerController: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyAutoScaling: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyAppMesh: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyExternalDNSHostedZones: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyExternalDNSChangeSet: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyEFSEC2: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyEFS: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [!]  AWS::IAM::Policy/PolicyServiceLinkRole: DELETE_IN_PROGRESS
2021-10-09 07:13:34 [✖]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyAutoScaling: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyExternalDNSHostedZones: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyCertManagerChangeSet: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyEFSEC2: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyFSX: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyEFS: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyCertManagerHostedZones: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyCertManagerGetChange: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyAppMesh: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyAWSLoadBalancerController: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyServiceLinkRole: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyExternalDNSChangeSet: CREATE_FAILED – "Resource creation cancelled"
2021-10-09 07:13:34 [✖]  AWS::IAM::Policy/PolicyEBS: CREATE_FAILED – "Partition \"aws\" is not valid for resource \"arn:aws:ec2:*:*:volume/*\". (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 78a4be11-60d5-4444-8669-c3375d453948; Proxy: null)"
2021-10-09 07:13:34 [!]  1 error(s) occurred and cluster hasn't been created properly, you may wish to check CloudFormation console
2021-10-09 07:13:34 [ℹ]  to cleanup resources, run 'eksctl delete cluster --region=cn-northwest-1 --name=cnnx-eks-test'
2021-10-09 07:13:34 [✖]  waiting for CloudFormation stack "eksctl-cnnx-eks-test-nodegroup-eks-test-ng-1": ResourceNotReady: failed waiting for successful resource state

here is cluster.yaml:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cnnx-eks-test
  region: cn-northwest-1
  version: '1.21'

managedNodeGroups:
  - name: eks-test-ng-1
    instanceType: t3.small
    desiredCapacity: 2
    minSize: 1
    maxSize: 5
    ssh:
      publicKeyName: eks-cn
      allow: true
    iam:
      withAddonPolicies:
        externalDNS: true
        certManager: true
        albIngress: true
        appMesh: true
        autoScaler: true
        cloudWatch: true
        ebs: true
        efs: true
        fsx: true

when i remove the ebs field in yaml file, everything works fine

eksctl Versions
eksctl version: 0.69.0
kubectl version: v1.21.2-13+d2965f0db10712
OS: linux

[aclevername]

Hi @Vieufoux, it might be that we are missing some aws-cn references in the policy arns we attach. Similar to this PR #4218. We will investigate 😄

[aclevername]

Specifically in this file https://github.com/weaveworks/eksctl/blob/main/pkg/cfn/builder/statement.go#L16

We need to be able to swap our aws for aws-cn when in the china region I think

Is there any update @aclevername?
We identified that the issue seems to be in https://github.com/weaveworks/eksctl/blob/main/pkg/cfn/builder/statement.go#L299 where "aws" is still hardcoded in the resources of the ebs Statements. I guess this could be fixed by calling the "addARNPartitionPrefix"-function like elsewhere in this file?

[cPu1]

Is there any update @aclevername? We identified that the issue seems to be in https://github.com/weaveworks/eksctl/blob/main/pkg/cfn/builder/statement.go#L299 where "aws" is still hardcoded in the resources of the ebs Statements. I guess this could be fixed by calling the "addARNPartitionPrefix"-function like elsewhere in this file?

Right, the resource names need to have aws-cn instead of aws for China regions in that list of statements. We'll add this to our backlog and work on it soon.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[jnalley]

Hi @cPu1. The hard coded resource names cause the same issue with aws-us-gov. The PR #4369 seems to fix it.

[sunhaolin]

+1

[jonathan-strzalka-saggs]

Can we get #4369 merged into a release. This is causing issues for us too.

[Skarlso]

Fixed by #4547

[Skarlso]

Also, related issue #4545 for the record.