‼️ NOTICE: aws-eks "error retrieving RESTMappings to prune: invalid resource networking.k8s.io/v1"

[mattchrist]

Please add your +1 👍 to let us know you have encountered this

---

Status: IN-PROGRESS

Overview:

Version 1.106.0 and later of the aws-eks construct library throw an error when trying to update a KubernetesManifest object, this includes objects used in the cluster.addManifest method.

Complete Error Message:

11:22:46 AM | UPDATE_FAILED        | Custom::AWSCDK-EKS-KubernetesResource | pdb/Resource/Default
Received response status [FAILED] from custom resource. Message returned: Error: b'poddisruptionbudget.policy/test-pdb configured\nerror: error retrieving RESTMappings to prune: invalid resource networking.k8s.io/v1, Kind=Ingress, Namespaced=true: no matches for kind "Ingress" in version "networking.k8s.io/v1"\n'

Workaround:

Downgrade to version 1.105.0 or below

---

Original opening post

When updating a KubernetesManifest, the deploy fails with an error like:

11:22:46 AM | UPDATE_FAILED        | Custom::AWSCDK-EKS-KubernetesResource | pdb/Resource/Default
Received response status [FAILED] from custom resource. Message returned: Error: b'poddisruptionbudget.policy/test-pdb configured\nerror: error retrieving RESTMappings to prune: invalid resource networking.k8s.io/v1, Kind=Ingress, Namespaced=true: no matches for kind "Ingress" in version "networking.k8s.io/v1"\n'

This issue occurs with Kubernetes versions 1.16, 1.17, and 1.20.

Reproduction Steps

1. Deploy a simple EKS stack with a manifest

import { Stack, App } from "@aws-cdk/core";
import {
  Cluster,
  KubernetesManifest,
  KubernetesVersion,
} from "@aws-cdk/aws-eks";

const app = new App();
const stack = new Stack(app, "repro-prune-invalid-resource", {
  env: {
    region: process.env.CDK_DEFAULT_REGION,
    account: process.env.CDK_DEFAULT_ACCOUNT,
  },
});

const cluster = new Cluster(stack, "cluster", {
  clusterName: "repro-prune-invalid-resource-test",
  version: KubernetesVersion.V1_16,
  prune: true,
});

const manifest = new KubernetesManifest(stack, `pdb`, {
  cluster,
  manifest: [
    {
      apiVersion: "policy/v1beta1",
      kind: "PodDisruptionBudget",
      metadata: {
        name: "test-pdb",
        namespace: "default",
      },
      spec: {
        maxUnavailable: 1,
        selector: {
          matchLabels: { app: "thing" },
        },
      },
    },
  ],
});

app.synth();

This deploys successfully.

2. Make a small change to the manifest, such as changing maxUnavailable: 1 to maxUnavailable: 2 and deploy again

This results in the error above.

What did you expect to happen?

I would have expected the deploy to have succeeded, and updated the maxUnavailable field in the deployed Manifest from 1 to 2.

What actually happened?

11:22:46 AM | UPDATE_FAILED        | Custom::AWSCDK-EKS-KubernetesResource | pdb/Resource/Default
Received response status [FAILED] from custom resource. Message returned: Error: b'poddisruptionbudget.policy/test-pdb configured\nerror: error retrieving RESTMappings to prune: invalid resource networking.k8s.io/v1, Kind=Ingress, Namespaced=true: no matches for kind "Ingr
ess" in version "networking.k8s.io/v1"\n'

Logs: /aws/lambda/repro-prune-invalid-resource-awscd-Handler886CB40B-hFxU42VXJuOz

at invokeUserFunction (/var/task/framework.js:95:19)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async onEvent (/var/task/framework.js:19:27)
at async Runtime.handler (/var/task/cfn-response.js:48:13) (RequestId: 1be7dfcb-288d-4309-8b8c-cadafb97fd09)

Environment

• CDK CLI Version : 1.108.0
• Framework Version: 1.108.0
• Node.js Version: v12.18.4
• OS : Linux
• Language (Version): Typescript 4.3.2

Other

---

This is 🐛 Bug Report

[cweidinger]

This is also a problem for us +1. Also happens with 1_20 on our real cluster

[saggar]

+1 Same error on EKS cluster v1.18. Currently blocking deployment of K8s manifest/ YAML changes via CDK.

[otaviomacedo]

The recent bump in the kubectl version from 1.20.0 to 1.21.0 broke KubernetesManifest updates.

Marking this as a p1, given the other comments from people facing the same issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[iliapolo]

We are preparing a patch release for this fix. Will update once available.

[NetaNir]

version 1.110.1 was released with the patch.

[DarkmatterVale]

Out of curiosity, why isn't the kubectl handler version always configured to be the same version as the Kubernetes cluster?

[asgerjensen]

Seeing error retrieving RESTMappings to prune: invalid resource extensions/v1beta1, Kind=Ingress, N amespaced=true: no matches for kind "Ingress" in version "extensions/v1beta1"\n' on EKS version 1.23, using cdk 2.51.1

[robertd]

@asgerjensen Can you please paste a code snippet showing how you’re creating your cluster? Did you provide a custom kubectl layer or using defaults?

[charlesakalugwu]

@robertd I just ran into this problem as well.

Fargate EKS 1.23 built using CDK 2.53.0

Cluster looks as follows:

      kubernetes_cluster = eks.Cluster(
          self,
          id=f"{prefix}-cluster",
          version=version,
          vpc=vpc,
          vpc_subnets=[
              ec2.SubnetSelection(
                  subnet_group_name="private-subnet",
              ),
          ],
          cluster_logging=[
              eks.ClusterLoggingTypes.AUDIT,
          ],
          default_capacity=0,
          endpoint_access=eks.EndpointAccess.PUBLIC_AND_PRIVATE,
          kubectl_layer=kubectl_v23.KubectlV23Layer(self, id=f"{prefix}-kubectl"),
          masters_role=masters_role,
          output_masters_role_arn=False,
          place_cluster_handler_in_vpc=True,
          secrets_encryption_key=kms_key_data,
          output_cluster_name=False,
          output_config_command=False,
          tags=tags,
      )

As you can see I supplied the matching kubectl layer for k8s 1.23. Nevertheless I keep seeing the error:

Received response status [FAILED] from custom resource. Message returned: Error: b'configmap/foo configured\nerror: error retrieving RESTMappings to prune: invalid resource extensions/v1beta1, Kind=Ingress, Namespaced=true: no matches for kind "Ingress" in version "extensions/v1beta1"\n'

I have upgraded CDK to 2.55.0 and upgraded EKS to 1.24 and I saw the error again.

@asgerjensen Did you make any progress?