Unexpected "Subnet.CIDR: Ifield is immutable:" error - not trying to change CIDR

[kmadathil]

1. What kops version are you running? The command kops version, will display
this information.
Version 1.14.1 (git-b7c25f9a9)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"211047e9a1922595eaa3a1127ed365e9299a6c23", GitTreeState:"clean", BuildDate:"2019-10-15T12:02:12Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?
AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

I first did

kops create cluster ${CLUSTER_NAME} \
  --node-count ${NODE_COUNT} \
  --zones $ZONES \
  --node-size $NODE_SIZE \
  --master-size $MASTER_SIZE \
  --master-zones $ZONES \
  --vpc $VPC_ID \

kops replace cluster ${CLUSTER_NAME} \
  -f kops-cluster.yaml

kops update cluster ${CLUSTER_NAME} --yes

kops-cluster.yaml has a modified subnet CIDR, so that I can run on a pre-determined VPC without collisions. So far so good, everything runs well.

Now, I need to reduce the number of nodes to zero temporarily. I ran kops edit ig, made the change, and then ran kops update cluster --yes.

5. What happened after the commands executed?
I see the following error.

error running task "Subnet/us-west-2a.clustername.k8s.local" (8m58s remaining to succeed): Subnet.CIDR: Invalid value: "172.31.128.0/19": field is immutable: old="172.31.144.0/19" new="172.31.128.0/19"

I am not trying to modify the subnet CIDR, only maxSize/minSize so this error should not occur.

6. What did you expect to happen?
No error, number of nodes in the ig scales down to zero

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2019-11-16T00:51:07Z
  generation: 1
  name: clustername.k8s.local
spec:
  api:
    loadBalancer:
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://clustername-m-example-com-state-store/clustername.k8s.local
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-us-west-2a
      name: a
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-us-west-2a
      name: a
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.14.8
  masterInternalName: api.internal.clustername.k8s.local
  masterPublicName: api.clustername.k8s.local
  networkCIDR: 172.31.0.0/16
  networkID: vpc-redacted
  networking:
    kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.31.144.0/19
    name: us-west-2a
    type: Public
    zone: us-west-2a
  topology:
    dns:
      type: Public
    masters: public
    nodes: public                                                                                                                                                                                           
  

kind: InstanceGroup                                                                                                                                                                                         
metadata:                                                                                                                                                                                                   
  creationTimestamp: 2019-12-03T20:10:16Z                                                                                                                                                                   
  labels:                                                                                                                                                                                                   
    kops.k8s.io/cluster: clustername.k8s.local                                                                                                                                                              
  name: master-us-west-2a                                                                                                                                                                                   
spec:                                                                                                                                                                                                       
  image: kope.io/k8s-1.14-debian-stretch-amd64-hvm-ebs-2019-09-26                                                                                                                                           
  machineType: m3.medium                                                                                                                                                                                    
  maxSize: 1                                                                                                                                                                                                
  minSize: 1                                                                                                                                                                                                
  nodeLabels:                                                                                                                                                                                               
    kops.k8s.io/instancegroup: master-us-west-2a                                                                                                                                                            
  role: Master                                                                                                                                                                                              
  subnets:                                                                                                                                                                                                  
  - us-west-2a                                                                                                                                                                                              
                                                                                                                                                                                                            
---                                                                                                                                                                                                         
                                                                                                                                                                                                            
apiVersion: kops/v1alpha2                                                                                                                                                                                   
kind: InstanceGroup                                                                                                                                                                                         
metadata:                                                                                                                                                                                                   
  creationTimestamp: 2019-12-03T20:10:16Z                                                                                                                                                                   
  generation: 1                                                                                                                                                                                             
  labels:                                                                                                                                                                                                   
    kops.k8s.io/cluster: clustername.k8s.local
  name: nodes
spec:
  image: kope.io/k8s-1.14-debian-stretch-amd64-hvm-ebs-2019-09-26
  machineType: t2.medium
  maxSize: 0
  minSize: 0
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  subnets:
  - us-west-2a

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

1203 16:52:47.507066   29692 changes.go:174] comparing slices: 0 *awstasks.SecurityGroup {"Name":null,"Lifecycle":null,"ID":"sg-0a7afea3f6340fc3a","Description":null,"VPC":null,"RemoveExtraRules":null,"Shared":null,"Tags":null} *awstasks.SecurityGroup {"Name":"nodes.clustername.k8s.local","Lifecycle":"Sync","ID":"sg-0a7afea3f6340fc3a","Description":"Security group for nodes","VPC":{"Name":"clustername.k8s.local","Lifecycle":"Sync","ID":"vpc-e4405380","CIDR":"172.31.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true,"Tags":null},"RemoveExtraRules":["port=22"],"Shared":null,"Tags":{"KubernetesCluster":"clustername.k8s.local","Name":"nodes.clustername.k8s.local","kubernetes.io/cluster/clustername.k8s.local":"owned"}}
I1203 16:52:47.508588   29692 tagbuilder.go:95] tags: [_aws _k8s_1_6]
I1203 16:52:47.508673   29692 urls.go:165] Using cached protokube location: "https://kubeupv2.s3.amazonaws.com/kops/1.14.1/images/protokube.tar.gz"
I1203 16:52:47.509759   29692 request_logger.go:45] AWS request: autoscaling/DescribeLaunchConfigurations
W1203 16:52:47.597024   29692 executor.go:130] error running task "Subnet/us-west-2a.clustername.k8s.local" (9m58s remaining to succeed): Subnet.CIDR: Invalid value: "172.31.128.0/19": field is immutable: old="172.31.144.0/19" new="172.31.128.0/19"
I1203 16:52:47.597050   29692 executor.go:103] Tasks: 81 done / 88 total; 1 can run

9. Anything else do we need to know?
Not that I can think of

[rifelpet]

Rather than running kops create cluster first followed by kops replace -f can you try just using kops replace --force -f when creating a new cluster? That way it will use the subnet CIDR in kops-cluster.yaml from the beginning rather than choosing its own subnet CIDRs based off of the VPC.

[kmadathil]

I get the same error with that approach as well. I ran

kops replace -f kops-cluster.yaml --force
kops create secret --name $CLUSTER_NAME admin -i $SSH_KEY
kops update cluster --yes

My cluster comes up as usual.
Now I ran kops edit ig, made the change, and then ran kops update cluster --yes.

I see the same error

[kmadathil]

Any other suggestions? I can get this working as expected if I don't use a VPC_ID, but the use model I have in mind includes one.

[fgbreel]

Seems the subnet cidr you provided is in fact a host address.

Try to use this subnet address instead:

subnets:
-  - cidr: 172.31.144.0/19
+  - cidr: 172.31.128.0/19

[kmadathil]

Thank you! That change fixes the issue. 144 was an arithmetic error. Perhaps kops could throw a warning (or error) if I specify a subnet cidr incorrectly as above that could cause an issue.

[fgbreel]

It happens!

This is a nice tool to help you calculate networks in the terminal: https://packages.debian.org/search?keywords=ipcalc

Enjoy kops!

:)