Unable to Access kops Cluster on AWS (Account Has Limited Permissions)

[lmeandry]

/kind support

1. What kops version are you running? The command kops version, will display
this information.

Client version: 1.31.0 (git-v1.31.0)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
My kubectl is...

$ kubectl version --client
Client Version: v1.32.3
Kustomize Version: v5.5.0

My test kops cluster is unreachable. I cannot use KUBECONFIG=~/.kube/config kubectl cluster-info

3. What cloud provider are you using?
AWS cloud account in us-east-1. Our VPCs and Subnets are setup by our gov't agency sponsor. We do however have the ability to create and manage EC2/EBS, Security Groups, S3, Route53 resources.

**4. What commands did you run? What is the simplest way to reproduce this issue?**kubect

5. What happened after the commands executed?

6. What did you expect to happen?
That I could run kubectl cluster-info with the exported kubeconfig.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

---
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: '2025-05-08T20:06:57Z'
  generation: 2
  name: <REDACTED>platform-kops.<REDACTED_ORG>.com
spec:
  api:
    dns: {}
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://<REDACTED_ORG>-ocp-<REDACTED>-testing/<REDACTED>platform-kops.<REDACTED_ORG>.com
  dnsZone: Z0772711NMWNNROOWA4O
  etcdClusters:
    - cpuRequest: 200m
      etcdMembers:
        - encryptedVolume: true
          instanceGroup: control-plane-us-east-1b
          name: b
      manager:
        backupRetentionDays: 90
      memoryRequest: 100Mi
      name: main
    - cpuRequest: 100m
      etcdMembers:
        - encryptedVolume: true
          instanceGroup: control-plane-us-east-1b
          name: b
      manager:
        backupRetentionDays: 90
      memoryRequest: 100Mi
      name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
    - 0.0.0.0/0
    - ::/0
  kubernetesVersion: 1.31.7
  networkCIDR: 10.175.55.0/24
  networkID: vpc-0e17e4d5cab5c90db
  networking:
    amazonvpc: {}
  nonMasqueradeCIDR: 10.175.55.0/24
  sshAccess:
    - 0.0.0.0/0
    - ::/0
  subnets:
    - cidr: 10.175.55.32/27
      egress: External
      id: subnet-0e2a1ff61e1784941
      name: us-east-1b
      type: Private
      zone: us-east-1b
    - cidr: 10.175.55.0/27
      egress: External
      id: subnet-0871bf04b10f3c364
      name: us-east-1c
      type: Private
      zone: us-east-1c
  topology:
    dns:
      type: Private
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: '2025-05-08T20:06:57Z'
  labels:
    kops.k8s.io/cluster: <REDACTED>platform-kops.<REDACTED_ORG>.com
  name: control-plane-us-east-1b
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20250305
  machineType: t3.small
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
    - us-east-1b
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: '2025-05-08T20:06:57Z'
  labels:
    kops.k8s.io/cluster: <REDACTED>platform-kops.<REDACTED_ORG>.com
  name: nodes-us-east-1b
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20250305
  machineType: t3.small
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
    - us-east-1b
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: '2025-05-08T20:06:57Z'
  labels:
    kops.k8s.io/cluster: <REDACTED>platform-kops.<REDACTED_ORG>.com
  name: nodes-us-east-1c
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20250305
  machineType: t3.small
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
    - us-east-1c

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

$ KUBECONFIG=~/.kube/config kubectl cluster-info -v 10
I0508 16:17:46.854658   33423 loader.go:402] Config loaded from file:  /home/ec2-user/.kube/config
I0508 16:17:46.855281   33423 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0508 16:17:46.855319   33423 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0508 16:17:46.855341   33423 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0508 16:17:46.855355   33423 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0508 16:17:46.855492   33423 discovery_client.go:253] "Request Body" body=""
I0508 16:17:46.855599   33423 round_trippers.go:473] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.32.3 (linux/amd64) kubernetes/32cc146" 'https://api.internal.<REDACTED>platform-kops.<REDACTED_ORG>.com/api?timeout=32s'
I0508 16:17:46.859084   33423 round_trippers.go:502] HTTP Trace: DNS Lookup for api.internal.<REDACTED>platform-kops.<REDACTED_ORG>.com resolved to [{203.0.113.123 }]
^C

9. Anything else do we need to know?
Additional background....
I work for a defense contractor supporting a US government agency. The agency allocates and manages the cloud resources and has some limits placed via IAM roles and policies.

Currently we build and deploy K8s clusters via the Rancher/RKE2 Kubernetes distribution. We use Ansible to run Terraform that creates some AWS resources within the AWS account. We use the Ansible lablabs/ansible-role-rke2 role to install RKE2. The terraform also generates the necessary load balancers pointing to cluster's service ingresses. Once installed an Ansible task creates the kubeconfig using /etc/rancher/rke2/rke2.yaml from one of the master nodes, and update the server line to point to the API server. This works out well to let us use helm, kubectl and the Ansible kubernetes/core collections.

I'm investigating kops because we have to move away from using Rancher/RKE2.

After a little trial and error, it seems like almost all the necessary AWS resources are set up with a cluster.

However the newly built kops cluster is inaccessible with k8s utilities.

[hakman]

@lmeandry please list the commands that you use to create the cluster.

[lmeandry]

This is what I used after setting environment variables for the cluster name and S3 bucket to store the TF state:

$ kops create cluster $kopsClusterName  \
--cloud aws \
--dns private \
--zones=us-east-1b,us-east-1c \
--control-plane-size=t3.small \
--node-size=t3.small \
--dns-zone=Z0772711NMWNNROOWA4O \
--networking=amazonvpc \
--network-id=vpc-0e17e4d5cab5c90db \
--subnets=subnet-0e2a1ff61e1784941,subnet-0871bf04b10f3c364

Then I used "edit" to adjust the networking according to "Run kOps in an existing VPC".

And finally, a "kops update cluster $kopsClusterName --yes --admin".

[hakman]

Based on the info, you probably use a VPN to access the internal network and the DNS records for the cluster API are not forwarded (--dns private).
If you create a load balancer, could you connect to it?

[lmeandry]

Not a VPN. Our developers use AWS Workspaces for code development and then SSH to an EC2 instance to execute Ansible and Terraform to build K8s cluster(s). To access deployed web applications with web browsers on Workspaces, we set the /etc/resolv.conf with a DNS server from Route 53 that points to an ELB directing to the cluster's Ingress. Since this is a development environment with sensitive data, we do not allow ingress from the public Internet.

Whenever I try to use --dns private combined with --topology=private, I receive errors that I cannot have a mix of public and private subnets.

ec2-user@ip-10-175-55-219:~/src/research_rancher_alternatives/kops$ kops create cluster \
>   --state=$KOPS_STATE_STORE \
>   --name=$KOPS_CLUSTER_NAME \
>   --cloud=aws \
>   --networking=amazonvpc \
>   --dns=private \
>   --topology=private \
>   --subnets="subnet-0e2a1ff61e1784941,subnet-0871bf04b10f3c364" \
>   --zones="us-east-1b,us-east-1c" \
>   --network-id=vpc-0e17e4d5cab5c90db \
>   --dns-zone=Z0772711NMWNNROOWA4O \
>   --control-plane-size=t3.small \
>   --node-size=t3.small \
>   --admin-access="10.175.55.0/24" \
>   --cloud-labels="clap_off=yes"
I0513 07:58:44.069774    2639 create_cluster.go:881] Using SSH public key: /home/ec2-user/.ssh/<REDACTED>.pub
I0513 07:58:44.438877    2639 new_cluster.go:1461] Cloud Provider ID: "aws"
I0513 07:58:44.948961    2639 subnets.go:205] Assigned CIDR 10.175.55.64/29 to subnet utility-us-east-1b
I0513 07:58:44.948998    2639 subnets.go:205] Assigned CIDR 10.175.55.72/29 to subnet utility-us-east-1c
Error: [spec.networking.subnets[2].id: Forbidden: cannot mix subnets with specified ID and unspecified ID, spec.networking.subnets[3].id: Forbidden: cannot mix subnets with specified ID and unspecified ID]

Honestly from the start I've been a little confused of the options I should use to get this working for us.

Comparing to what I know that works with RKE2, should I instead use --network=calico instead of --network=amazonvpc? I ask because I believe that's what RKE2 uses by default, and we didn't have any problems deploying on AWS.

I apologize for my lack of Kubernetes and AWS networking experience. For what it's worth, I'm continuing to review the Kubernetes Networking Options and official Kubernetes Cluster Networking pages to help my understanding.

[lmeandry]

Since there is such a large timezone difference, I have been experimenting with other documented options. Reduced to just one subnet and AZ for now. I also updated to version v1.32.0. At first I was still having issues. But now I have managed to access the cluster through the API.

I'm starting to think this was more of a support issue rather than bug. I tried updating the label, but probably don't have permission to triage. I'll let the developers decide. Since trying other options, I don't feel this is a reproducible bug yet, except maybe for clarity in documentation.

Anyway, I came across another issue mentioning --dns=none' (because I was trying to not have to use a DNS entry to access the API server) and read about "Load Balancer Class". So I've been trying these options out along with the others in "Run kOps in an existing VPC".

So at this point the load balancer points pointing to target groups, and IP within the subnet instead of a "public" address. Earlier although AWS resources were created, I still could not access the cluster via the API with kubectl.

But now I can use kubectl to access the cluster. Somewhere its because of additional updates to the kOps configuration I made.

ec2-user@ip-10-175-55-219:~/src/research_rancher_alternatives/kops$ { date ; KUBECONFIG=~/.kube/config kubectl cluster-info ; }
Tue May 13 16:32:23 MST 2025
Kubernetes control plane is running at https://api-<REDACTED>-kops-<REDACTED>-c-tsu37d-90e6f18202b39600.elb.us-east-1.amazonaws.com
CoreDNS is running at https://api-<REDACTED>-kops-<REDACTED>-c-tsu37d-90e6f18202b39600.elb.us-east-1.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Here's my most recent CLI used:

$ kops create cluster   --state=$KOPS_STATE_STORE   --name=$KOPS_CLUSTER_NAME   --cloud=aws   --networking=calico   --subnets=subnet-0871bf04b10f3c364   --zones=us-east-1c   --network-id=vpc-0e17e4d5cab5c90db  --control-plane-size=t3.small   --node-size=t3.small  --dns=private --dns-zone=Z0772711NMWNNROOWA4O --node-security-groups=sg-0262a92b2a3a1b913  --ssh-public-key=~/.ssh/id_rsa.pub --associate-public-ip=false

I was thinking allowed access was SG related. But in the end I don't think it mattered because it was applied to worker node and not the control plane. Now I think it's because left off the --internal for the kubeconfig command.

Still used "edit" to switch to Private and Internal, then an Update.

Here's my most current "get config":

---
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2025-05-13T21:54:45Z"
  generation: 2
  name: <REDACTED>-kops.<REDACTED>.com
spec:
  api:
    loadBalancer:
      class: Network
      type: Internal
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://<REDACTED>-ocp-kops-<REDACTED>e-testing/<REDACTED>-kops.<REDACTED>.com
  dnsZone: Z0772711NMWNNROOWA4O
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-1c
      name: c
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-us-east-1c
      name: c
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.32.4
  networkCIDR: 10.175.55.0/24
  networkID: vpc-0e17e4d5cab5c90db
  networking:
    calico: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 10.175.55.0/27
    egress: External
    id: subnet-0871bf04b10f3c364
    name: us-east-1c
    type: Private
    zone: us-east-1c
  topology:
    dns:
      type: Private

The final piece of the puzzle is figuring out how to use my custom SSH keypair so we can get into the nodes in case of troubleshooting.

[hakman]

@lmeandry Happy to see you figured it out. sshKeyName might fit your needs: https://kops.sigs.k8s.io/security/.
For --network, amazonvpc is more picky with the node OS and has quite bad support for anything other than Amazon Linux, but it should have support from AWS. Calico is pretty stable and reliable, and there are others supported with more or less features.

[lmeandry]

@hakman thank you for your help!

[hakman]

@hakman thank you for your help!

Happy to help, was nice to see that everything worked out in the end. Don't hesitate to open issues if you have more Q.