Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set kube-proxy bindaddress through kubeadm config file failed #77876

Open
mzyfree opened this issue May 14, 2019 · 8 comments

Comments

Projects
None yet
3 participants
@mzyfree
Copy link

commented May 14, 2019

I want to set kube-proxy bindaddress to ipv6 address [::] through command:
kubeadm init --config=./kubeadm_config.yml --ignore-preflight-errors all
But result not function.
The kube-proxy bindadress is always 0.0.0.0
My kubeadm config file:
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: "v1.14.0"
controlPlaneEndpoint: "2000:1691:1111:2222:aaaa:bbbb:cccc:2000"
imageRepository: "matrix"
etcd:
external:
endpoints:

  • "http://[2000:1691:1111:2222:aaaa:bbbb:cccc:1180]:2379"
  • "http://[2000:1691:1111:2222:aaaa:bbbb:cccc:1181]:2379"
  • "http://[2000:1691:1111:2222:aaaa:bbbb:cccc:1182]:2379"
    apiServer:
    certSANs:
  • 2000:1691:1111:2222:aaaa:bbbb:cccc:2000
  • ::1
  • 2000:1691:1111:2222:aaaa:bbbb:cccc:1180
  • 2000:1691:1111:2222:aaaa:bbbb:cccc:1181
  • 2000:1691:1111:2222:aaaa:bbbb:cccc:1182
  • kubernetes
  • kubernetes.default
  • kubernetes.default.svc
  • kubernetes.default.svc.cluster
  • kubernetes.default.svc.cluster.local
    extraArgs:
    advertise-address: "2000:1691:1111:2222:aaaa:bbbb:cccc:2000"
    token-auth-file: /etc/kubernetes/pki/token.csv
    service-node-port-range: 1-65535
    networking:
    podSubnet: fd00:177:177::/112
    serviceSubnet: fd00:10:96::/112

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
bindAddress: "::"

My test command:
kubeadm init --config=./kubeadm_config.yml --ignore-preflight-errors all
OR
kubeadm config upload from-file --config /opt/matrix/k8s/kubeadm_config.yml

What happened:
[root@180 ~]# kubectl get cm kube-proxy -n kube-system -o yaml
apiVersion: v1
data:
config.conf: |-
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
acceptContentTypes: ""
burst: 10
contentType: application/vnd.kubernetes.protobuf
kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
qps: 5
clusterCIDR: fd00:177:177::/112
configSyncPeriod: 15m0s
conntrack:
max: null
maxPerCore: 32768
min: 131072
tcpCloseWaitTimeout: 1h0m0s
tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
masqueradeAll: false
masqueradeBit: 14
minSyncPeriod: 0s
syncPeriod: 30s
ipvs:
excludeCIDRs: null
minSyncPeriod: 0s
scheduler: ""
syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: ""
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
resourceContainer: /kube-proxy
udpIdleTimeout: 250ms
winkernel:
enableDSR: false
networkName: ""
sourceVip: ""
kubeconfig.conf: |-
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
server: https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443
name: default
contexts:
- context:
cluster: default
namespace: default
user: default
name: default
current-context: default
users:
- name: default
user:
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
kind: ConfigMap
metadata:
creationTimestamp: "2019-05-14T17:28:46Z"
labels:
app: kube-proxy
name: kube-proxy
namespace: kube-system
resourceVersion: "222"
selfLink: /api/v1/namespaces/kube-system/configmaps/kube-proxy
uid: ba52ade2-766d-11e9-8809-0cda411d2eca

What you expected to happen:
kube-proxy bindaddress set to be [::]

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): v1.14.0
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release): centos 7.4
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:
@neolit123

This comment has been minimized.

Copy link
Member

commented May 14, 2019

hm, i don't see this being a kubeadm problem.

bindAddress: "::"

is there a chance kube-proxy resolves this to 0.0.0.0.

what happens if you try to modify this config field?

healthzBindAddress: 0.0.0.0:10256

@neolit123

This comment has been minimized.

Copy link
Member

commented May 14, 2019

/sig cluster-lifecycle

@mzyfree

This comment has been minimized.

Copy link
Author

commented May 15, 2019

hm, i don't see this being a kubeadm problem.

bindAddress: "::"

is there a chance kube-proxy resolves this to 0.0.0.0.

what happens if you try to modify this config field?

healthzBindAddress: 0.0.0.0:10256

I change healthzBindAddress to 0.0.0.123 then it works,But bindAddress not work.
I try to read kubeadm code under initconfiguration.go~.~!a little bit hard

@mzyfree

This comment has been minimized.

Copy link
Author

commented May 15, 2019

Here is the log:
[root@180 k8s]# kubeadm init phase addon kube-proxy --config=./kubeadm_config.yml -v 6
I0515 03:50:54.291684 59421 initconfiguration.go:186] loading configuration from "./kubeadm_config.yml"
I0515 03:50:54.295023 59421 initconfiguration.go:105] detected and using CRI socket: /var/run/dockershim.sock
I0515 03:50:54.295647 59421 interface.go:384] Looking for default routes with IPv4 addresses
I0515 03:50:54.295662 59421 interface.go:389] Default route transits interface "eth0"
I0515 03:50:54.296598 59421 interface.go:196] Interface eth0 is up
I0515 03:50:54.296733 59421 interface.go:244] Interface "eth0" has 5 addresses :[10.99.212.140/24 2000:1691:1111:2222:aaaa:bbbb:cccc:3000/128 2000:1691:1111:2222:aaaa:bbbb:cccc:2000/128 2000:1691:1111:2222:aaaa:bbbb:cccc:1180/64 fe80::4448:9f73:479f:af7a/64].
I0515 03:50:54.296769 59421 interface.go:211] Checking addr 10.99.212.140/24.
I0515 03:50:54.296781 59421 interface.go:218] IP found 10.99.212.140
I0515 03:50:54.296796 59421 interface.go:250] Found valid IPv4 address 10.99.212.140 for interface "eth0".
I0515 03:50:54.296806 59421 interface.go:395] Found active IP 10.99.212.140
I0515 03:50:54.297045 59421 feature_gate.go:226] feature gates: &{map[]}
I0515 03:50:54.299206 59421 loader.go:359] Config loaded from file /etc/kubernetes/admin.conf
I0515 03:50:54.320455 59421 round_trippers.go:438] POST https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/api/v1/namespaces/kube-system/serviceaccounts 409 Conflict in 19 milliseconds
I0515 03:50:54.326518 59421 round_trippers.go:438] POST https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/api/v1/namespaces/kube-system/configmaps 409 Conflict in 3 milliseconds
I0515 03:50:54.337540 59421 round_trippers.go:438] PUT https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy 200 OK in 10 milliseconds
I0515 03:50:54.357913 59421 round_trippers.go:438] POST https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/apis/apps/v1/namespaces/kube-system/daemonsets 409 Conflict in 4 milliseconds
I0515 03:50:54.362305 59421 round_trippers.go:438] PUT https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/apis/apps/v1/namespaces/kube-system/daemonsets/kube-proxy 200 OK in 4 milliseconds
I0515 03:50:54.367381 59421 round_trippers.go:438] POST https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings 409 Conflict in 4 milliseconds
I0515 03:50:54.369524 59421 round_trippers.go:438] PUT https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kubeadm:node-proxier 200 OK in 1 milliseconds
I0515 03:50:54.377739 59421 round_trippers.go:438] POST https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles 409 Conflict in 6 milliseconds
I0515 03:50:54.381402 59421 round_trippers.go:438] PUT https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles/kube-proxy 200 OK in 3 milliseconds
I0515 03:50:54.387662 59421 round_trippers.go:438] POST https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings 409 Conflict in 4 milliseconds
I0515 03:50:54.391570 59421 round_trippers.go:438] PUT https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings/kube-proxy 200 OK in 3 milliseconds
[addons] Applied essential addon: kube-proxy

Here is the modify message put by kubeadm:
I0515 03:52:13.991457 14207 request.go:942] Request Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"kube-proxy","namespace":"kube-system","creationTimestamp":null,"labels":{"app":"kube-proxy"}},"data":{"config.conf":"apiVersion: kubeproxy.config.k8s.io/v1alpha1\nbindAddress: 0.0.0.0\nclientConnection:\n acceptContentTypes: ""\n burst: 10\n contentType: application/vnd.kubernetes.protobuf\n kubeconfig: /var/lib/kube-proxy/kubeconfig.conf\n qps: 5\nclusterCIDR: fd00:177:177::/112\nconfigSyncPeriod: 15m0s\nconntrack:\n max: null\n maxPerCore: 32768\n min: 131072\n tcpCloseWaitTimeout: 1h0m0s\n tcpEstablishedTimeout: 24h0m0s\nenableProfiling: false\nhealthzBindAddress: 0.0.0.123:10256\nhostnameOverride: ""\niptables:\n masqueradeAll: false\n masqueradeBit: 14\n minSyncPeriod: 0s\n syncPeriod: 30s\nipvs:\n excludeCIDRs: null\n minSyncPeriod: 0s\n scheduler: ""\n syncPeriod: 30s\nkind: KubeProxyConfiguration\nmetricsBindAddress: 127.0.0.1:10249\nmode: ""\nnodePortAddresses: null\noomScoreAdj: -999\nportRange: ""\nresourceContainer: /kube-proxy\nudpIdleTimeout: 250ms\nwinkernel:\n enableDSR: false\n networkName: ""\n sourceVip: ""","kubeconfig.conf":"apiVersion: v1\nkind: Config\nclusters:\n- cluster:\n certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n server: https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443\n name: default\ncontexts:\n- context:\n cluster: default\n namespace: default\n user: default\n name: default\ncurrent-context: default\nusers:\n- name: default\n user:\n tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token"}}
I0515 03:52:13.991513 14207 round_trippers.go:419] curl -k -v -XPUT -H "Content-Type: application/json" -H "Accept: application/json, /" -H "User-Agent: kubeadm/v1.14.0 (linux/amd64) kubernetes/641856d" 'https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy'
I0515 03:52:13.993862 14207 round_trippers.go:438] PUT https://[2000:1691:1111:2222:aaaa:bbbb:cccc:2000]:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy 200 OK in 2 milliseconds

The bindAddress of cm in kubeadm already is 0.0.0.0

@mzyfree

This comment has been minimized.

Copy link
Author

commented May 15, 2019

I think i have found the reason.In kubeadm code,kube-proxy bindAddress is bind with apiserver localAPIEndpoint struct,see the code below:

func SetClusterDynamicDefaults(cfg *kubeadmapi.ClusterConfiguration, advertiseAddress string, bindPort int32) error {
componentconfigs.Known.Default(cfg)

ip := net.ParseIP(advertiseAddress)
if ip.To4() != nil {
	cfg.ComponentConfigs.KubeProxy.BindAddress = kubeadmapiv1beta1.DefaultProxyBindAddressv4
} else {
	cfg.ComponentConfigs.KubeProxy.BindAddress = kubeadmapiv1beta1.DefaultProxyBindAddressv6
}

if err := NormalizeKubernetesVersion(cfg); err != nil {
	return err
}

if cfg.ControlPlaneEndpoint != "" {
	host, port, err := kubeadmutil.ParseHostPort(cfg.ControlPlaneEndpoint)
	if err != nil {
		return err
	}
	if port == "" {
		cfg.ControlPlaneEndpoint = net.JoinHostPort(host, strconv.FormatInt(int64(bindPort), 10))
	}
}
LowercaseSANs(cfg.APIServer.CertSANs)
return nil

}

And here is the function call:

func SetInitDynamicDefaults(cfg *kubeadmapi.InitConfiguration) error {
if err := SetBootstrapTokensDynamicDefaults(&cfg.BootstrapTokens); err != nil {
return err
}
if err := SetNodeRegistrationDynamicDefaults(&cfg.NodeRegistration, true); err != nil {
return err
}
if err := SetAPIEndpointDynamicDefaults(&cfg.LocalAPIEndpoint); err != nil {
return err
}
return SetClusterDynamicDefaults(&cfg.ClusterConfiguration, cfg.LocalAPIEndpoint.AdvertiseAddress, cfg.LocalAPIEndpoint.BindPort)
}

So Maybe i need to use InitConfiguration and set LocalAPIEndpoint to ipv6 address to make the bindAddress use [::] ipv6 address.
But the question is still exist: Why cann't user set the kube-proxy bindAddress through kubeadm KubeProxyConfiguration config params.

@mzyfree

This comment has been minimized.

Copy link
Author

commented May 15, 2019

@neolit123 After update my kubeadm_config.yml, Add LocalAPIEndpoint in InitConfiguration.All kube-proxy run ok.
Here is my new kubeadm config file:

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: "v1.14.0"
controlPlaneEndpoint: "2000:1691:1111:2222:aaaa:bbbb:cccc:2000"
imageRepository: "matrix"
etcd:
external:
endpoints:
- "http://[2000:1691:1111:2222:aaaa:bbbb:cccc:1180]:2379"
- "http://[2000:1691:1111:2222:aaaa:bbbb:cccc:1181]:2379"
- "http://[2000:1691:1111:2222:aaaa:bbbb:cccc:1182]:2379"
apiServer:
certSANs:
- 2000:1691:1111:2222:aaaa:bbbb:cccc:2000
- ::1
- 2000:1691:1111:2222:aaaa:bbbb:cccc:1180
- 2000:1691:1111:2222:aaaa:bbbb:cccc:1181
- 2000:1691:1111:2222:aaaa:bbbb:cccc:1182
- kubernetes
- kubernetes.default
- kubernetes.default.svc
- kubernetes.default.svc.cluster
- kubernetes.default.svc.cluster.local
extraArgs:
advertise-address: "2000:1691:1111:2222:aaaa:bbbb:cccc:2000"
token-auth-file: /etc/kubernetes/pki/token.csv
service-node-port-range: 1-65535
networking:
podSubnet: fd00:177:177::/112
serviceSubnet: fd00:10:96::/112
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: "2000:1691:1111:2222:aaaa:bbbb:cccc:1180"

@neolit123

This comment has been minimized.

Copy link
Member

commented May 15, 2019

But the question is still exist: Why cann't user set the kube-proxy bindAddress through kubeadm KubeProxyConfiguration config params.

this is definitely a bug.

could you please close this issue and log a new one in kubernetes/kubeadm so that we can track it better.
your example in this comment is great:
#77876 (comment)

  • just outline the problem in a couple of sentences.
    thanks.
@neolit123

This comment has been minimized.

Copy link
Member

commented May 15, 2019

/area kubeadm
/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.