aws-eks: Cannot update cluster endpoint access #21439

resnikb · 2022-08-03T11:30:23Z

Describe the bug

Changing endpoint access for an existing EKS cluster fails.

I have an existing EKS cluster with

endpointAccess: EndpointAccess.PUBLIC_AND_PRIVATE

If I change that to

endpointAccess: EndpointAccess.PRIVATE

CloudFormation update fails with error:

Received response status [FAILED] from custom resource. Message returned: Only one type of update can be allowed.

Expected Behavior

The update should succeed, and change the endpoint access to PRIVATE.

Current Behavior

Cluster update fails with:

Received response status [FAILED] from custom resource. Message returned: Only one type of update can be allowed.

Reproduction Steps

Create a new cluster with logging enabled:

   new Cluster(stack, 'Cluster', {
      clusterName: 'MyCluster',
      version: KubernetesVersion.V1_21,
      endpointAccess: EndpointAccess.PUBLIC_AND_PRIVATE,
      vpc,
      clusterLogging: [
        ClusterLoggingTypes.API,
        ClusterLoggingTypes.AUDIT,
        ClusterLoggingTypes.AUTHENTICATOR,
        ClusterLoggingTypes.CONTROLLER_MANAGER,
        ClusterLoggingTypes.SCHEDULER,
      ],
    });

Change endpoint access to EndpointAccess.PRIVATE and redeploy

Possible Solution

The cluster handler lambda specifies logging configuration even when only endpoint access needs to be updated. If logging configuration doesn't need updating, it should not be specified in the call to updateClusterConfig.

Additional Information/Context

CloudWatch logs for lambda execution:

2022-08-03T11:07:30.872Z	c627c94c-6ee5-4193-9d6b-18f1b376637a	INFO	onUpdate: {
    "updates": {
        "replaceName": false,
        "replaceVpc": false,
        "updateAccess": true,
        "replaceRole": false,
        "updateVersion": false,
        "updateEncryption": false,
        "updateLogging": false
    }
}

2022-08-03T11:07:31.798Z	c627c94c-6ee5-4193-9d6b-18f1b376637a	INFO	[AWS eks 400 0.925s 0 retries] updateClusterConfig({
  name: '<redacted>',
  logging: {
    clusterLogging: [
      {
        types: [
          'api',
          'audit',
          'authenticator',
          'controllerManager',
          'scheduler',
          [length]: 5
        ],
        enabled: true
      },
      { types: [ [length]: 0 ], enabled: true },
      [length]: 2
    ]
  },
  resourcesVpcConfig: {
    endpointPrivateAccess: true,
    endpointPublicAccess: false,
    publicAccessCidrs: undefined
  },
  clientRequestToken: '<redacted>'
})

2022-08-03T11:07:31.812Z	c627c94c-6ee5-4193-9d6b-18f1b376637a	ERROR	Invoke Error 	{
    "errorType": "InvalidParameterException",
    "errorMessage": "Only one type of update can be allowed.",
    "code": "InvalidParameterException",
    "message": "Only one type of update can be allowed.",
    "time": "2022-08-03T11:07:31.797Z",
    "requestId": "<redacted>",
    "statusCode": 400,
    "retryable": false,
    "retryDelay": 23.61820879655758,
    "stack": [
        "InvalidParameterException: Only one type of update can be allowed.",
        "    at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)",
        "    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)",
        "    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)"
    ]
}

CDK CLI Version

2.35.0 (build 5c23578)

Framework Version

No response

Node.js Version

v16.16.0

OS

Linux

Language

Typescript

Language Version

No response

Other information

No response

The text was updated successfully, but these errors were encountered:

resnikb · 2022-08-03T11:52:47Z

On further inspection, this might be related to #21436 and the changes made in #21185, as the clusterLogging JSON in the call to updateClusterConfig is now invalid.

juweeks · 2022-08-09T15:06:10Z

we're getting this too now. v2.33

johnnyhuy · 2022-09-09T23:45:45Z

One way we've worked around it was to turn off logging on the first run, switch-over endpoint access and then turn on logging. Not ideal since it doesn't resolve the root issue because we're doing three deploys instead of one.

pahud · 2022-11-16T22:08:32Z

I can reproduce this issue in cdk v2.50.0 and I'm assigning this to myself as p2. I'll look into it for the root cause and investigate if there's anything we can do to fix it.

pahud · 2022-11-16T22:32:42Z

I think we probably should fix here.

aws-cdk/packages/@aws-cdk/aws-eks/lib/cluster-resource-handler/cluster.ts

Lines 139 to 154 in 5b3d06d

    
           if (updates.updateLogging || updates.updateAccess) { 
        
             const config: aws.EKS.UpdateClusterConfigRequest = { 
        
               name: this.clusterName, 
        
               logging: this.newProps.logging, 
        
             }; 
        
             if (updates.updateAccess) { 
        
               // Updating the cluster with securityGroupIds and subnetIds (as specified in the warning here: 
        
               // https://awscli.amazonaws.com/v2/documentation/api/latest/reference/eks/update-cluster-config.html) 
        
               // will fail, therefore we take only the access fields explicitly 
        
               config.resourcesVpcConfig = { 
        
                 endpointPrivateAccess: this.newProps.resourcesVpcConfig.endpointPrivateAccess, 
        
                 endpointPublicAccess: this.newProps.resourcesVpcConfig.endpointPublicAccess, 
        
                 publicAccessCidrs: this.newProps.resourcesVpcConfig.publicAccessCidrs, 
        
               }; 
        
             } 
        
             const updateResponse = await this.eks.updateClusterConfig(config);

According to the lambda logs:

{
    "updates": {
        "replaceName": false,
        "replaceVpc": false,
        "updateAccess": true,
        "replaceRole": false,
        "updateVersion": false,
        "updateEncryption": false,
        "updateLogging": false
    }
}

We actually need to updateAccess only, it's not clear to me why we should have this line:

aws-cdk/packages/@aws-cdk/aws-eks/lib/cluster-resource-handler/cluster.ts

Line 142 in 5b3d06d

logging: this.newProps.logging,

pahud · 2022-11-17T16:08:39Z

Just created a PR draft for a quick fix #22957

I can successfully update the stack by simply updating the endpoint access like

   new eks.Cluster(this, 'Cluster', {
      vpc,
      endpointAccess: eks.EndpointAccess.PRIVATE,
      version: eks.KubernetesVersion.V1_23,
      clusterLogging: [
        eks.ClusterLoggingTypes.API,
        eks.ClusterLoggingTypes.AUDIT,
        eks.ClusterLoggingTypes.AUTHENTICATOR,
        eks.ClusterLoggingTypes.CONTROLLER_MANAGER,
        eks.ClusterLoggingTypes.SCHEDULER,
      ],
    });

Will look into previous commits to see if I miss anything.

jaredhancock31 · 2022-12-05T23:06:31Z

We're seeing a similar problem when mutating the list of allowAccessFrom CIDRs.

Example:

deploy a cluster with something like the following:

     # assume clusterLogging also enabled here
      allowAccessFrom:
        - 2.4.6.0/24

Then, try to update it by adding 2 entries:

     # assume clusterLogging is still the same as before, no delta
      allowAccessFrom:
        - 2.4.6.0/24
        - 1.2.3.4/32 
        - 3.3.3.3/32

Observe the following error:

4:59:47 PM | UPDATE_FAILED        | Custom::AWSCDK-EKS-Cluster            | EKSClusterE11008B6
Received response status [FAILED] from custom resource. Message returned: Only one type of update can be allowed.

Logs: /aws/lambda/redacted-name-awscdkawseks-OnEventHandler42BEBAE0-5fCPxU8lMELN

at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12) (RequestId: 7e00f099-29a7-4f0e-8bc9-4e9a7e552922)

This was tested using cdk 2.50.
This will block a lot of our automation from progressing

…#22957) This PR addresses the following known issues: 1. When updating the cluster endpoint access type only with logging predefined yet unchanged, the cluster-resource-handler updates both the logging and access, which is not allowed and throws the SDK error. This PR fixed this and will update access type only, which is allowed. 2. When updating the cluster endpoint public cidr with exactly the same size of cidr, the `setsEqual` function should return correctly. 3. When updating the cluster endpoint public access from one cidr to multiple cidr with logging predefined yet unchanged, the update should return correctly. 4. Updating both access and logging now throws an error from CDK custom resource. This PR is just a temporary fix that does not implement multiple operations in the cluster-resource-handler custom resource provider(i.e. update both logging and access). Fixes: #21439 ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

github-actions · 2022-12-20T00:47:25Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

…aws#22957) This PR addresses the following known issues: 1. When updating the cluster endpoint access type only with logging predefined yet unchanged, the cluster-resource-handler updates both the logging and access, which is not allowed and throws the SDK error. This PR fixed this and will update access type only, which is allowed. 2. When updating the cluster endpoint public cidr with exactly the same size of cidr, the `setsEqual` function should return correctly. 3. When updating the cluster endpoint public access from one cidr to multiple cidr with logging predefined yet unchanged, the update should return correctly. 4. Updating both access and logging now throws an error from CDK custom resource. This PR is just a temporary fix that does not implement multiple operations in the cluster-resource-handler custom resource provider(i.e. update both logging and access). Fixes: aws#21439 ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

resnikb added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 3, 2022

github-actions bot added the @aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service label Aug 3, 2022

github-actions bot assigned otaviomacedo Aug 3, 2022

pahud added the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Nov 14, 2022

pahud self-assigned this Nov 16, 2022

pahud added p2 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Nov 16, 2022

pahud mentioned this issue Nov 17, 2022

fix(aws-eks): fail to update both logging and access at the same time #22957

Merged

4 tasks

pahud removed the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Nov 17, 2022

pahud unassigned otaviomacedo Nov 17, 2022

mergify bot closed this as completed in #22957 Dec 20, 2022

danquack mentioned this issue Nov 1, 2023

Remove force new on eks vpc change hashicorp/terraform-provider-aws#34209

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws-eks: Cannot update cluster endpoint access #21439

aws-eks: Cannot update cluster endpoint access #21439

resnikb commented Aug 3, 2022

resnikb commented Aug 3, 2022

juweeks commented Aug 9, 2022

johnnyhuy commented Sep 9, 2022

pahud commented Nov 16, 2022

pahud commented Nov 16, 2022

pahud commented Nov 17, 2022 •

edited

jaredhancock31 commented Dec 5, 2022 •

edited

github-actions bot commented Dec 20, 2022

aws-eks: Cannot update cluster endpoint access #21439

aws-eks: Cannot update cluster endpoint access #21439

Comments

resnikb commented Aug 3, 2022

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

resnikb commented Aug 3, 2022

juweeks commented Aug 9, 2022

johnnyhuy commented Sep 9, 2022

pahud commented Nov 16, 2022

pahud commented Nov 16, 2022

pahud commented Nov 17, 2022 • edited

jaredhancock31 commented Dec 5, 2022 • edited

github-actions bot commented Dec 20, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

pahud commented Nov 17, 2022 •

edited

jaredhancock31 commented Dec 5, 2022 •

edited