Cilium Direct Routing Mode broken in latest release of EKS-A (v0.21.0) on vSphere with Ubuntu OVA

[thecloudgarage]

What happened:

• EKS-A cluster on vSphere not created
• Nodes have come up on vSphere and everything is fine
• However, cluster never comes up
• kubectl get machines -A says all 3 CP and 2 Worker nodes are in Running status
• When I SSH into one of the nodes, I get the following error

2024-11-18T15:46:22.547517231Z stderr F time="2024-11-18T15:46:22Z" level=fatal msg="auto-direct-node-routes cannot be used with tunneling. Packets must be routed through the tunnel device." subsys=daemon

• My configuration for the cluster CNI spec is as follows:

apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
 name: workload-cluster-name
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/17
    services:
      cidrBlocks:
      - 10.96.0.0/17
    cniConfig:
      cilium:
        skipUpgrade: true
        routingMode: "direct"

• I can confirm that there is no conflict with any of the CIDR blocks for pods and services
• Additionally, no conflict with API server IP
• So basically, everything is correct
• The moment I comment the line routingMode: "direct", the cluster comes up perfectly

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
Use EKS A v0.21.0 and ubuntu 20.04 k8s 1.30 release channel
Anything else we need to know?:
I have also tried it with release channels 1.28 and 1.29 ... same issue
Environment:

• EKS Anywhere Release:

 eksctl anywhere version
Version: v0.21.0
Release Manifest URL: https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml
Bundle Manifest URL: https://anywhere-assets.eks.amazonaws.com/releases/bundles/81/manifest.yaml

• EKS Distro Release: 1.30

[vivek-koppuru]

@thecloudgarage Quick question regarding this, since you have skipUpgrade: true, are you not using the bundled version of Cilium for subsequent upgrades? But this is an error during a new cluster creation and not an upgrade from an older version of EKS Anywhere right?

[vivek-koppuru]

We release both v0.21.7 and v0.22.0 that has a potential fix for this issue. We are going to update our release notes shortly, but if you would like to try it out in the meantime you can.

[thecloudgarage]

Ok.. thanks

…

On Mon, 3 Mar 2025, 06:22 Vivek Koppuru, ***@***.***> wrote: We release both v0.21.7 and v0.22.0 that has a potential fix for this issue. We are going to update our release notes shortly, but if you would like to try it out in the meantime you can. — Reply to this email directly, view it on GitHub <#9020 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJNKQ3RDXG4PKALD4L3EWDD2SORV7AVCNFSM6AAAAABSAA7C7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJTGAYTSNJSG4> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: vivek-koppuru]*vivek-koppuru* left a comment (aws/eks-anywhere#9020) <#9020 (comment)> We release both v0.21.7 and v0.22.0 that has a potential fix for this issue. We are going to update our release notes shortly, but if you would like to try it out in the meantime you can. — Reply to this email directly, view it on GitHub <#9020 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJNKQ3RDXG4PKALD4L3EWDD2SORV7AVCNFSM6AAAAABSAA7C7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJTGAYTSNJSG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ndeksa]

issues has been fixed and validated in the lab, closing as completed!