Testing on Kind: APIServer proxy not working

[jayunit100]

We have a new development recipe in the PR #5 .... When i run it, everything comes up however

• kpng server wants to access the apiserver through 10.96.0.1
• apiserver on 10.96.0.1 is a route that normally is written by... kube proxy
• since now kube proxy is running (other then kpng), the route falls down dial tcp 10.96.0.1:443: i/o timeout.

to reproduce, you can directly run the hack/local-up-kpng.sh script (uncomment all 3 functions to reproduce from source

E0419 01:36:27.444110       1 reflector.go:138] k8s.io/client-go@v0.20.5/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout

It appears that the kube-proxy rules routing to the APIServer arent working.

If possible would like to use that dev recipe as a starting point, merge it, and then fix whatever the routing issue is.

I suppose maybe its a chicken/egg thing , i.e. the KPNG nftables rules , when setup to access the apiserver through the service IP dont work bc KPNG isnt up yet....

Im not sure how kube-proxy normally solves this -

[mcluseau]

hence the --master flag, that's the most direct way to specify an alternative target. One can also specify a HostAlias (with APIServer or a LB on port 443), inject a custom kubeconfig via a ConfigMap, etc.

[mcluseau]

btw it seems --master is replaced by --server nowadays, so I'll change that right now so we conform.

[jayunit100]

yup i changed it already :) - well hold on - we already have a --server flag, so i modified the ds.yaml to use that in my dev recipe

To solve my problem i think... KPNG server needs to tell the KPNG dataplane

"hey mister dataplane ! write apiserver routing rules on startup do this 10.0.96.1 IP !"

even before it can connect to the apiserver, right? or is this a feature request ? after all, its the service proxy's job to write host-network service APIs...

[jayunit100]

i guess im asking more of a theoretical question then anything else - https://groups.google.com/g/kubernetes-sig-network/c/GKwCWXlpx04

[mcluseau]

not that it's a priority, but I wonder if I could override the gRPC version in kpng to access the balancing code. Le lun. 19 avr. 2021 à 15:42, jay vyas ***@***.***> a écrit :

…

i guess im asking more of a theoretical question then anything else - https://groups.google.com/g/kubernetes-sig-network/c/GKwCWXlpx04 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5QROBMTU7UCSAYRGNFBNLTJQXMTANCNFSM43E3F3ZA> .

[jayunit100]

ya interesting idea

[jayunit100]

Solutions:

• make an apiserver client token ?
• keepalived + host-alias
• run kpng on the apiserver, --server=IP (IP is in the kube generated cert)

[uablrek]

Start with kube-apiserver --token-auth-file=/srv/kubernetes/known_tokens.csv .... Set env before starting kpng KUBECONFIG=/etc/kubernetes/kubeconfig.token

The kubeconfig.token file;

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://192.168.1.1:6443
  name: xcluster
contexts:
- context:
    cluster: xcluster
    user: root
  name: xcluster
current-context: xcluster
kind: Config
preferences: {}
users:
- name: root
  user:
    token: kallekula

kpng will use the $KUBECONFIG and access the api-server without problem.

[uablrek]

Generate the token file;

##   kubeconfig_sec
##     Generate secure kubeconfig's
##
cmd_kubeconfig_sec() {
	local cfg=$dir/default/etc/kubernetes/kubeconfig

	export KUBECONFIG=$cfg.token
	cp $cfg $KUBECONFIG
	kubectl config set-cluster xcluster --server=https://192.168.1.1:6443
	kubectl config set-cluster xcluster --insecure-skip-tls-verify=true
	kubectl config set-credentials root --token=kallekula
}

[uablrek]

My /srv/kubernetes/known_tokens.csv

kallekula,root,root
X4o3adoGo5Yx6oi4LUo2JB2GO8b0CatJ,kubelet,kubelet
tHDwA7KmaQ0jTLSKcPV2p3LrMhXP2e3m,kube_proxy,kube_proxy
cBvKaWGjXE5TnOsOW7juaO0JsXiNsxN1,system:scheduler,system:scheduler
2gxkVhMhQvMKydr7RwYYS66vXEJxy47s,system:controller_manager,system:controller_manager
isb1TVpwRYzGfshMVfTfSEDvwx2txvQt,system:logging,system:logging
JF5hfsUcDPg1JrTzlw9T6aKAQwn2BGn8,system:monitoring,system:monitoring
lXjgo4cUaiyjyCHcrn0OnonOxme6G4AV,system:dns,system:dns

[jayunit100]

we found a few workarounds for this:

• using the kubelet config directly and increasing its api permissions
• using the kubeconfig config similar to what kindnormally does.... and just mounting it into kubeconfig

the hack/ recipes are up to date now and working... closing... thanks for your help ulabrek and mikael ...

We have a new issue though, where somehow coredns doesnt resolve 10.96.0.1 and were not sure why yet. that is a different follow on issue though, so ill file it separately.