Skip to content
This repository has been archived by the owner on Feb 10, 2022. It is now read-only.

Support configuring service cluster ip and pod network #220

Closed
wants to merge 1 commit into from
Closed

Support configuring service cluster ip and pod network #220

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions jobs/apply-specs/spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ properties:
description: The admin username for the Kubernetes cluster
admin-password:
description: The admin password for the Kubernetes cluster
kubedns-service-ip:
description: The service cluster IP for kube-dns, must reside within service-cluster-cidr set for kube-apiserver
default: "10.100.200.10"
timeout-sec:
description: Timeout for system spec deployment
default: 1200
Expand Down
2 changes: 1 addition & 1 deletion jobs/apply-specs/templates/specs/kube-dns.yml.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ metadata:
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.100.200.10
clusterIP: <%= kubedns-service-ip %>
ports:
- name: dns
port: 53
Expand Down
5 changes: 4 additions & 1 deletion jobs/flanneld/spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,10 @@ packages:
- flanneld
- cni

properties: {}
properties:
pod-network-cidr:
description: The pod networking cidr for pod network overlay
default: "10.200.0.0/16"

consumes:
- name: etcd
Expand Down
2 changes: 1 addition & 1 deletion jobs/flanneld/templates/bin/flanneld_ctl.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ start_flanneld() {
--cert-file /var/vcap/jobs/flanneld/config/etcd-client.crt \
--key-file /var/vcap/jobs/flanneld/config/etcd-client.key \
--ca-file /var/vcap/jobs/flanneld/config/etcd-ca.crt \
set /coreos.com/network/config '{"Network":"10.200.0.0/16","Backend":{"Type":"vxlan"}}'
set /coreos.com/network/config '{"Network":"<%= pod-network-cidr %>","Backend":{"Type":"vxlan"}}'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the format is incorrect for properties.
It should be like

 set /coreos.com/network/config '{"Network":"<%= p('pod-network-cidr') %>","Backend":{"Type":"vxlan"}}'

otherwise bosh is complaining

- Error filling in template 'flanneld_ctl.erb' (line 54: undefined local variable or method `pod' for #<Bosh::Template::EvaluationContext:0x000055e7f7843278>)


flanneld -etcd-endpoints=<%= etcd_endpoints %> \
--ip-masq \
Expand Down
3 changes: 3 additions & 0 deletions jobs/kube-apiserver/spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ properties:
default: 1235
route-sync-password:
description: The password for the route-sync user
service-cluster-cidr:
description: The service cluster IP cidr for hosting cluster services
default: "10.100.200.0/24"
tls.kubernetes.ca:
description: "CA Certificate for the Kubernetes master"
tls.kubernetes.certificate:
Expand Down
2 changes: 1 addition & 1 deletion jobs/kube-apiserver/templates/config/bpm.yml.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ processes:
- --secure-port=8443
- --service-account-key-file=/var/vcap/jobs/kube-apiserver/config/service-account-public-key.pem
- --service-account-lookup
- --service-cluster-ip-range=10.100.200.0/24
- --service-cluster-ip-range=<%= service-cluster-cidr %>
- --service-node-port-range=30000-32767
- --tls-cert-file=/var/vcap/jobs/kube-apiserver/config/kubernetes.pem
- --tls-private-key-file=/var/vcap/jobs/kube-apiserver/config/kubernetes-key.pem
Expand Down
3 changes: 3 additions & 0 deletions jobs/kube-controller-manager/spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ properties:
description: https_proxy env var for the kubernetes-controller-manager binary (i.e. for cloud provider interactions)
no_proxy:
description: no_proxy env var for cloud provider interactions, i.e. for the kubelet
service-cluster-cidr:
description: The service cluster IP for hosting cluster services
default: "10.100.200.0/24"
tls.kubo-ca:
description: "Certificate and private key for the Kubo CA"
service-account-private-key:
Expand Down
2 changes: 1 addition & 1 deletion jobs/kube-controller-manager/templates/config/bpm.yml.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ processes:
- --leader-elect
- --root-ca-file=/var/vcap/jobs/kube-controller-manager/config/ca.pem
- --service-account-private-key-file=/var/vcap/jobs/kube-controller-manager/config/service-account-private-key.pem
- --service-cluster-ip-range=10.100.200.0/24
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am no sure if we need that at all.
According to the doc

--cluster-cidr string-
  | CIDR Range for Pods in cluster. Requires --allocate-node-cidrs to be true

and we are not using the allocate-node-cidrs=true

Copy link
Author

@rapenchukd rapenchukd Jun 19, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--service-cluster-ip-range is needed, it sets the address block for hosting kubernetes service IPs, if not set, kubernetes uses a 10.X.X.X default of some sort, which could potentially clash with users pod IP space, or even network routable IP space.

I think the default size is also a /24, which means a kubernetes cluster can only have an absolute max of 254 services.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, thats for the controller manager and not the apiserver, yeah you are correct, that should be fine to drop for now.

- --service-cluster-ip-range=<%= service-cluster-cidr %>
- --terminated-pod-gc-threshold=100
- --use-service-account-credentials
- --v=<%=p('logging-level') %>
Expand Down
3 changes: 3 additions & 0 deletions jobs/kube-proxy/spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,6 @@ properties:
logging-level:
description: "V-leveled logging at the specified level. See https://github.com/golang/glog"
default: 2
pod-network-cidr:
description: The pod networking cidr for pod network overlay
default: "10.200.0.0/16"
2 changes: 1 addition & 1 deletion jobs/kube-proxy/templates/config/config.yml.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ clientConnection:
contentType: application/vnd.kubernetes.protobuf
kubeconfig: /var/vcap/jobs/kube-proxy/config/kubeconfig
qps: 5
clusterCIDR: 10.200.0.0/16
clusterCIDR: <%= pod-network-cidr %>
configSyncPeriod: 15m0s
conntrack:
max: 0
Expand Down
3 changes: 3 additions & 0 deletions jobs/kubelet/spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,9 @@ properties:
logging-level:
description: "V-leveled logging at the specified level. See https://github.com/golang/glog"
default: 2
kubedns-service-ip:
description: The service IP for kube-dns, must match what is configured for kube-dns
default: "10.100.200.10"

consumes:
- name: cloud-provider
Expand Down
2 changes: 1 addition & 1 deletion jobs/kubelet/templates/config/kubeletconfig.yml.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ authentication:
authorization:
mode: Webhook
clusterDNS:
- 10.100.200.10
- <%= kubedns-service-ip %>
clusterDomain: cluster.local
failSwapOn: false
readOnlyPort: 0
Expand Down