Failed to interact with cluster in wsl2 #707
Comments
|
more infomations: the outupt of |
|
cc @PatrickLang |
|
@anjiawei1991 what´s the output of |
|
|
Not a WSL2 expert but it seems our Network setup is failing, can you get the logs from the kindnetd pod? ( Also, were you following https://kind.sigs.k8s.io/docs/user/using-wsl2/ ? |
can you check if iptables is working inside the container? what´s the output of |
@BenTheElder this is the log of kindnetd pod。 I was not following the using-wsl2 doc but the Microsoft's official doc. |
|
|
can you please try following the WSL2 doc? those instructions are from a microsoft employee :-) |
that's super helpful, that shouldn't happen. |
|
seems we might need a if that fixes things, we can probably ensure kindnetd runs it. |
|
great catch @BenTheElder , seems that the WSL2 kernel doesn´t have the netfilter modules needed |
I have glanced over those instructions, and I found no special in it, it's almost the same with I had done on WSL2 installation. But I do think there is some problem in WSL2's docker enviroment, because I just found that I can't run a very simple nginx in docker. I'm going to install some other vm. Also , I will save the WSL2 enviroment for this issue tracking. |
|
They do mention a particular way of starting docker fwiw, but the missing module is the real problem here. (#707 (comment)) |
|
Similar issue with docker-machine: kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5c98db65d4-h57mp 0/1 Pending 0 8m5s
kube-system coredns-5c98db65d4-tv76c 0/1 Pending 0 8m5s
kube-system etcd-kind-control-plane 1/1 Running 0 7m27s
kube-system kindnet-77trs 0/1 CrashLoopBackOff 6 7m50s
kube-system kindnet-rbhzn 0/1 CrashLoopBackOff 6 7m50s
kube-system kindnet-tvdtg 0/1 CrashLoopBackOff 6 8m6s
kube-system kube-apiserver-kind-control-plane 1/1 Running 0 7m4s
kube-system kube-controller-manager-kind-control-plane 1/1 Running 0 7m14s
kube-system kube-proxy-2hnkr 1/1 Running 0 8m6s
kube-system kube-proxy-ddvbr 1/1 Running 0 7m50s
kube-system kube-proxy-slxkg 1/1 Running 0 7m50s
kube-system kube-scheduler-kind-control-plane 1/1 Running 0 7m23s
kube-system tiller-deploy-7bf78cdbf7-xwrfq 0/1 Pending 0 5m27sRunning way: tee ~/.kind-config.yaml << EOF
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
apiServerAddress: $(docker-machine ip)
nodes:
- role: control-plane
- role: worker
- role: worker
EOFkind create cluster --config ~/.kind-config.yamlOS X Mojave
docker-machine version 0.16.1, build cce350d7
kind version v0.4.0
Docker version 18.09.2, build 6247962
kubectl Client Version: v1.15.0
kubectl Server Version: v1.15.0 |
|
@PatrickLang I suspect lacking the MASQUERADE module is a bit of a problem for many Kubernetes setups? Is it totally not available or just not loaded? |
|
microsoft/WSL#4165 looks related what kernel config does that relate to? |
|
Huh I would have thought https://cateee.net/lkddb/web-lkddb/IP_NF_TARGET_MASQUERADE.html aside: modules on a working host: |
|
on that same host (my linux workstation, just the first handy linux box I know it works on :^)): |
|
thanks, will do some more digging there. my initial guess was that it was actually missing the chains since WSL2 doesn't run init.d or systemd by default. I'll see if I can duplicate the default chains from a normal Ubuntu setup. |
|
actually looks like |
|
I'm pretty sure the error is somewhat misleading and that I'd definitely go with |
|
making some progress... kconfig diff diff --git a/Microsoft/config-wsl b/Microsoft/config-wsl
index 646309095..be2158f8c 100644
--- a/Microsoft/config-wsl
+++ b/Microsoft/config-wsl
@@ -1,13 +1,13 @@
#
# Automatically generated file; DO NOT EDIT.
-# Linux/x86 4.19.52 Kernel Configuration
+# Linux/x86 4.19.57 Kernel Configuration
#
#
-# Compiler: x86_64-msft-linux-gcc (GCC) 7.3.0
+# Compiler: gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
#
CONFIG_CC_IS_GCC=y
-CONFIG_GCC_VERSION=70300
+CONFIG_GCC_VERSION=70400
CONFIG_CLANG_VERSION=0
CONFIG_CC_HAS_ASM_GOTO=y
CONFIG_IRQ_WORK=y
@@ -869,7 +869,7 @@ CONFIG_NF_TABLES_INET=y
# CONFIG_NFT_CONNLIMIT is not set
# CONFIG_NFT_LOG is not set
# CONFIG_NFT_LIMIT is not set
-# CONFIG_NFT_MASQ is not set
+CONFIG_NFT_MASQ=y
# CONFIG_NFT_REDIR is not set
# CONFIG_NFT_NAT is not set
# CONFIG_NFT_TUNNEL is not set
@@ -1033,6 +1033,7 @@ CONFIG_NF_REJECT_IPV4=y
CONFIG_NF_NAT_IPV4=y
CONFIG_NF_NAT_MASQUERADE_IPV4=y
# CONFIG_NFT_CHAIN_NAT_IPV4 is not set
+CONFIG_NFT_MASQ_IPV4=y
CONFIG_NF_NAT_PROTO_GRE=y
CONFIG_NF_NAT_PPTP=y
CONFIG_NF_NAT_H323=y
@@ -1066,6 +1067,7 @@ CONFIG_IP_NF_ARP_MANGLE=y
CONFIG_NF_TABLES_IPV6=y
# CONFIG_NFT_CHAIN_ROUTE_IPV6 is not set
# CONFIG_NFT_CHAIN_NAT_IPV6 is not set
+CONFIG_NFT_MASQ_IPV6=y
# CONFIG_NFT_DUP_IPV6 is not set
# CONFIG_NFT_FIB_IPV6 is not set
# CONFIG_NF_DUP_IPV6 is not set
now kindnet gets further before crashing |
|
oh nevermind - kindnet restarted after that and now it's running :) |
|
Awesome! We should retry the node listing / rewrite as a proper controller 😅 Right now we panic a little too aggressively, but it does help surface problems and it should restart. |
|
Digging into a kube-proxy problem next: |
|
The module load failures at the top are not relevant. All the mods it's probing are compiled in already. The best clue as this point is the iptables-restore error which is truncated |
|
might have to up the kube-proxy verbosity to see more of what it was doing. |
|
I think technically #726 fixed this, but ideally long term we will hopefully not need those workarounds in a future version of WSL2 |
doc: update latest workaround for #707
|
If you are using docker with WSL 2 in May 2020 then do the following in a terminal: |
|
@jpvosloo is this something we need to add to the guide? |
|
May I know how to change the flag and recompile my wsl2 kernel? Seems this is also stopping docker directly installed inside wsl2 from working (as Docker Desktop is not there for Surface Pro X yet, so wanna try to get docker running inside wsl2 as a workaround for now) |
|
Related issue: docker/roadmap#91 |
|
I don't use WSL2, if anyone does and knows about this, please also look at #1740 |
|
Used https://kubernetes.io/blog/2020/05/21/wsl-docker-kubernetes-on-the-windows-desktop/ In writing that I see a pretty ominous server/client version warning there.. (flashed as per the wsl guide in k8s |
That doesn't matter yet, Please file a new bug with the full bug template details instead of commenting on a 2+ year old issue, thing will have changed quite a bit and we'll need to know things like the kind version you're using (which is in the bug template) if anyone can help. |
|
@BenTheElder Thanks for your comment. For whoever else this could help: I solved it earlier using #1740 (comment) and by upgrading my I am not sure if updating Since it's a known issue as linked above, I don't think it's worth creating a new issue, but thank you all for the help! |
I setup the cluster in wsl2, but can't interact with it.
How to reproduce it (as minimally and precisely as possible):
kind create clusterto create default cluster. and usingexport KUBECONFIG="$(kind get kubeconfig-path --name="kind")"to set KUBECONFIG env for kubectlkubectl cluster-infoto lookup clusterThen after a lot of time, I see
Error from server (InternalError): an error on the server ("") has prevented the request from succeedingAnything else we need to know?:
the output of
cat $KUBECONFIGis:the output of
netstat -ano | grep 45991is:the output of
docker exec kind-control-plane ps auxis:kind version): v0.4.0kubectl version):docker info):/etc/os-release):The text was updated successfully, but these errors were encountered: