eks: Unable to create fargate clusters

[Gum-Christopher-bah]

Describe the bug

Attempting to create an eks fargate cluster fails during the Custom::AWSCDK-EKS-Cluster creation step, stating that a cluster already exists with that name. Attempted on the following aws-cdk-lib versions (cdk cli tool also set to corresponding version): 2.78.0, 2.80.0, 2.83.0, 2.86.0, 2.87.0. Previously (At the time of the cdk version's release) I was able to deploy a fargate cluster using CDK version 2.54.0, which added support for EKS version 1.24, using the same account and local deployment role I'm using now. Re-bootstrapped account to latest as well to rule that out.

Expected Behavior

Able to create new clusters using latest CDK version

Current Behavior

Error:

12:38:18 PM | CREATE_FAILED        | Custom::AWSCDK-EKS-Cluster            | FargateCluster/Resource/Resource/Default
Received response status [FAILED] from custom resource. Message returned: Cluster already exists with name: FargateCluster019F03E8-563be2cd723c4ce8914ec2bd0a015ded

Logs: /aws/lambda/DummyEKSFargateCluster-awsc-OnEventHandler42BEBAE0-tVt2m9ZK4pZU

at deserializeAws_restJson1ResourceInUseExceptionResponse (/var/runtime/node_modules/@aws-sdk/client-eks/dist-cjs/protocols/Aws_restJson1.js:2544:23)
at deserializeAws_restJson1CreateClusterCommandError (/var/runtime/node_modules/@aws-sdk/client-eks/dist-cjs/protocols/Aws_restJson1.js:1020:25)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:20
at async StandardRetryStrategy.retry (/var/runtime/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
at async ClusterResourceHandler.onCreate (/var/task/cluster.js:1:984) (RequestId: 563be2cd-723c-4ce8-914e-c2bd0a015ded)

The log group above contains 2 streams, one with a timestamp of 12:22 that shows the cluster is in a creating state:

2023-07-07T16:22:14.293Z	f5bdf59b-ace0-42f8-bfba-244194e38392	INFO	{
  clientName: 'EKSClient',
  commandName: 'CreateClusterCommand',
  input: {
    resourcesVpcConfig: {
      endpointPrivateAccess: true,
      securityGroupIds: [Array],
      endpointPublicAccess: false,
      subnetIds: [Array]
    },
    roleArn: 'REMOVED:role/DummyEKSFargateCluster-FargateClusterRole8E36B33A-ZNQ0FGMS020D',
    version: '1.26',
    kubernetesNetworkConfig: { ipFamily: 'ipv4' },
    logging: undefined,
    name: 'FargateCluster019F03E8-563be2cd723c4ce8914ec2bd0a015ded'
  },
  output: {
    cluster: {
      arn: 'REMOVED:cluster/FargateCluster019F03E8-563be2cd723c4ce8914ec2bd0a015ded',
      certificateAuthority: [Object],
      clientRequestToken: undefined,
      connectorConfig: undefined,
      createdAt: 2023-07-07T16:22:14.166Z,
      encryptionConfig: undefined,
      endpoint: undefined,
      health: undefined,
      id: undefined,
      identity: undefined,
      kubernetesNetworkConfig: [Object],
      logging: [Object],
      name: 'FargateCluster019F03E8-563be2cd723c4ce8914ec2bd0a015ded',
      outpostConfig: undefined,
      platformVersion: 'eks.4',
      resourcesVpcConfig: [Object],
      roleArn: 'REMOVED:role/DummyEKSFargateCluster-FargateClusterRole8E36B33A-ZNQ0FGMS020D',
      status: 'CREATING',
      tags: {},
      version: '1.26'
    }
  },
  metadata: {
    httpStatusCode: 200,
    requestId: 'ccd4a0d1-b841-4437-a5b1-a1605f1818f7',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}

And another at 12:38 where it calls onCreate a second time and fails

2023-07-07T16:38:17.143Z	9fe11b44-3350-43c0-bd7e-3146dfbbcce2	ERROR	Invoke Error 	
{
    "errorType": "ResourceInUseException",
    "errorMessage": "Cluster already exists with name: FargateCluster019F03E8-563be2cd723c4ce8914ec2bd0a015ded",
    "name": "ResourceInUseException",
    "$fault": "client",
    "$metadata": {
        "httpStatusCode": 409,
        "requestId": "ce88cb92-4429-4be3-9e03-fc2b4e65d803",
        "attempts": 1,
        "totalRetryDelay": 0
    },
    "clusterName": "FargateCluster019F03E8-563be2cd723c4ce8914ec2bd0a015ded",
    "nodegroupName": null,
    "addonName": null,
    "stack": [
        "ResourceInUseException: Cluster already exists with name: FargateCluster019F03E8-563be2cd723c4ce8914ec2bd0a015ded",
        "    at deserializeAws_restJson1ResourceInUseExceptionResponse (/var/runtime/node_modules/@aws-sdk/client-eks/dist-cjs/protocols/Aws_restJson1.js:2544:23)",
        "    at deserializeAws_restJson1CreateClusterCommandError (/var/runtime/node_modules/@aws-sdk/client-eks/dist-cjs/protocols/Aws_restJson1.js:1020:25)",
        "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
        "    at async /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24",
        "    at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:20",
        "    at async StandardRetryStrategy.retry (/var/runtime/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)",
        "    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22",
        "    at async ClusterResourceHandler.onCreate (/var/task/cluster.js:1:984)"
    ]
}

Reproduction Steps

Create a stack with just a cluster in a file

import { Construct } from 'constructs';
import {
  aws_ec2 as ec2,
  aws_eks as eks,
  Stack,
} from 'aws-cdk-lib';
import { KubectlV26Layer } from '@aws-cdk/lambda-layer-kubectl-v26';


export class DummyFargateClusterStack extends Stack {

  public cluster: eks.FargateCluster;

  constructor(scope: Construct, id: string, props: eks.FargateClusterProps){
    super(scope, id);

    this.cluster = new eks.FargateCluster(this, 'FargateEKSCluster', {
      version: props.version,
      vpc: props.vpc,
      endpointAccess: eks.EndpointAccess.PRIVATE,
      kubectlLayer: new KubectlV26Layer(this, 'kubectl-v26-layer'),
      placeClusterHandlerInVpc: true,
      vpcSubnets: [{
        subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS
      }],
    });
  }
}

Deploy it with an app in another file

import { DummyFargateClusterStack } from "../path/file";
const app = new App();
new DummyFargateClusterStack(app, 'Dummy2EKSFargateCluster', {
      vpc: existing_vpc,
      version: eks.KubernetesVersion.V1_26,
    });
    

Where existing_vpc is a vpc referenced from lookup in the app using default context to get account/region

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.87.0, 2.86.0, 2.83.0, 2.80.0, 2.78.0

Framework Version

No response

Node.js Version

v18.13.0

OS

Mac

Language

Typescript

Language Version

No response

Other information

No response

[Gum-Christopher-bah]

Closing, I needed to add states.{{myregion}}.amazonaws.com to our NFW allowlist

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.