WSL2: dockerd related missing kernel flags #4165

simonferquel · 2019-06-17T10:55:09Z

By running https://github.com/moby/moby/blob/master/contrib/check-config.sh within wsl2, I just produced a report of missing things for running dockerd with all features enabled:

simon@simonxps:~/go/src/github.com/docker/docker/contrib$ ./check-config.sh
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: missing
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: missing
- CONFIG_IP_NF_NAT: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_LEGACY_VSYSCALL_NONE: enabled
    (containers using eglibc <= 2.13 will not work. Switch to
     "CONFIG_VSYSCALL_[NATIVE|EMULATE]" or use "vsyscall=[native|emulate]"
     on kernel command line. Note that this will disable ASLR for the,
     VDSO which may assist in exploiting security vulnerabilities.)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_IOSCHED_CFQ: missing
- CONFIG_CFQ_GROUP_IOSCHED: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled
- CONFIG_IP_VS: enabled
- CONFIG_IP_VS_NFCT: missing
- CONFIG_IP_VS_PROTO_TCP: missing
- CONFIG_IP_VS_PROTO_UDP: missing
- CONFIG_IP_VS_RR: missing
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: missing
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: missing
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: missing
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: enabled
  - "macvlan":
    - CONFIG_MACVLAN: enabled
    - CONFIG_DUMMY: missing
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: missing
    - CONFIG_NF_CONNTRACK_FTP: missing
    - CONFIG_NF_NAT_TFTP: missing
    - CONFIG_NF_CONNTRACK_TFTP: missing
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

cc @benhillis, I think it should help.
I'll try to make my own kernel with all this setup accordingly, and will do experiment to setup kubernetes in a similar way as what we do in Docker Desktop to see how things go.

The text was updated successfully, but these errors were encountered:

simonferquel · 2019-06-17T14:53:55Z

I can confirm that adding missing flags makes it possible to run swarm and netwoking seems to work.
However, logs get polluted with warnings like:

time="2019-06-17T16:51:41+02:00" level=warning msg="Running modprobe nf_nat failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.43-microsoft-standard/modules.dep.bin'\nmodprobe: WARNING: Module nf_nat not found in directory /lib/modules/4.19.43-microsoft-standard`, error: exit status 1"

simonferquel · 2019-06-17T14:55:03Z

(it does also that for xt_conntrack)

benhillis · 2019-06-17T15:52:57Z

Thanks @simonferquel, I will make sure all those config options get added to the kernel config. Since the kernel that we are using for WSL2 includes modules statically instead of in a separate folder those warnings are benign. It would likely be good to add a flag to docker daemon to suppress those errors for WSL2.

simonferquel · 2019-06-17T15:58:57Z

That is what I suspected.
Got a try with kubernetes, we miss CONFIG_NETFILTER_XT_MATCH_COMMENT apparently (at least). Can you add it to the list ?

benhillis · 2019-06-17T16:01:13Z

@simonferquel - Yes, I will add that as well. If that is needed, should it be added to the check-config script?

Biswa96 · 2019-06-17T18:52:02Z

@simonferquel The WSL-ed Linux kernel source here https://thirdpartysource.microsoft.com/download/Windows%20Subsystem%20for%20Linux%20v2/May%202019/WSLv2-Linux-Kernel-master.zip. Found the kernel configs at WSLv2-Linux-Kernel-master\Microsoft\config-wsl.

simonferquel · 2019-06-17T19:00:25Z

Yep already working on it for starting prototyping Kubernetes enabled integration packs

…

________________________________ From: Biswapriyo Nath <notifications@github.com> Sent: Monday, June 17, 2019 8:52:29 PM To: microsoft/WSL Cc: Simon Ferquel; Mention Subject: Re: [microsoft/WSL] WSL2: dockerd related missing kernel flags (#4165) @simonferquel<https://github.com/simonferquel> The WSL-ed Linux kernel source here https://thirdpartysource.microsoft.com/download/Windows%20Subsystem%20for%20Linux%20v2/May%202019/WSLv2-Linux-Kernel-master.zip. Found the kernel configs at WSLv2-Linux-Kernel-master\Microsoft\config-wsl. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#4165?email_source=notifications&email_token=AAMZQFNBVO4GG6AFSDHQH4TP27MOZA5CNFSM4HYVPCFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX4DSTI#issuecomment-502806861>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAMZQFLD3ZBPE53DZ45ZET3P27MOZANCNFSM4HYVPCFA>.

therealkenc · 2019-06-18T00:51:52Z

Found the kernel configs at WSLv2-Linux-Kernel-master\Microsoft\config-wsl.

You can get that at runtime with:

$ sudo cat /proc/config.gz | gunzip -

Which is a nice touch, since we don't have /boot/config-4.19.43-microsoft-standard. [/proc/config.gz isn't really common on distros these days (including Ubuntu) because they have /boot.]

MortenRickiRasmussen · 2019-06-18T14:14:26Z

Will this issue fix the VXLAN issue?

I'am getting this when trying to use networl overlays in Docker Swarm
network sandbox join failed: subnet sandbox join failed for "172.172.0.0/16": error creating vxlan interface: operation not supported

simonferquel · 2019-06-18T14:17:01Z

@MortenRickiRasmussen yes. I have a built a kernel with a modified kconfig file, and I can expose ports on the ingress network, create and attach to overlay networks etc.

ibuildthecloud · 2019-06-19T23:51:41Z

Related to this is that Kubernetes needs CONFIG_NETFILTER_XT_MATCH_COMMENT as the use -m comment --comment ... on iptables.

Biswa96 · 2019-06-28T15:14:58Z

@simonferquel Can you mention which configs you have changed?

simonferquel · 2019-06-28T16:20:44Z

You can use the diff tool in the kernel sources script directory to look at the changes

…

________________________________ From: Biswapriyo Nath <notifications@github.com> Sent: Friday, June 28, 2019 5:15:25 PM To: microsoft/WSL Cc: Simon Ferquel; Mention Subject: Re: [microsoft/WSL] WSL2: dockerd related missing kernel flags (#4165) @simonferquel<https://github.com/simonferquel> Can you mention which configs you have changed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#4165?email_source=notifications&email_token=AAMZQFLLEBYJD7FSTZA4XJTP4YTI3A5CNFSM4HYVPCFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY2LM3Q#issuecomment-506771054>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAMZQFJX7Z6HRI2GA5JWR73P4YTI3ANCNFSM4HYVPCFA>.

andrelop-zz · 2019-07-10T21:10:00Z

Hello,

Plese see weaveworks/ignite#129 .

My guess is that the issue is somewhat related to CONFIG_DM_THIN_PROVISIONING not being enabled by default.

Woud you please consider enabling it for the next release ?

Biswa96 · 2019-07-11T03:42:39Z

@andrelop You can compile your own kernel and place it in System32\lxss\tools folder. We don't have to wait for next builds.

therealkenc · 2019-07-11T20:11:55Z

You can compile your own kernel and place it in System32\lxss\tools folder.

Or compile it as a module, in which case you don't need to muck lxss/tools. CONFIG_DM_THIN_PROVISIONING is =m on Real Ubuntu FWIW. Along with all the other CONFIG_DM_*.

andrelop-zz · 2019-07-13T02:43:53Z

Hello

@andrelop You can compile your own kernel and place it in System32\lxss\tools folder. We don't have to wait for next builds.

Reproducing here what I posted at weaveworks/ignite#129 to see if you could help with the correct procedure to build/use a custom kernel for a given WSL2 VM :

I tried building a custom kernel but the WSL2 VM cannot start after I change the C:\Windows\System32\lxss\tools\kernel using the vmlinux kernel produced by the kernel build process.

What I did :

Downloaded the last release on microsoft/WSL2-Linux-Kernel github repo (https://github.com/microsoft/WSL2-Linux-Kernel/archive/v4.19.57.tar.gz)
Unpacked it and used the provided kernel config on https://github.com/microsoft/WSL2-Linux-Kernel/blob/master/Microsoft/config-wsl
Ran make pointing to this config (ie. using make KCONFIG_CONFIG=config.wsl)
At the end of the build, renamed the existing C:\Windows\System32\lxss\tools\kernel file to something else (ie. C:\Windows\System32\lxss\tools\kernel.old) and renamed/moved the vmlinux generated file as C:\Windows\System32\lxss\tools\kernel
Forced all the WSL2 VMs shutdown using wsl --shutdown on PowerShell as Administrator
Started the WSL2 VM again, but it did not boot
Restored the previously renamed file in order to get the VM booting again

Not sure what else I could try or if what I did was really what needed to be done.

onomatopellan · 2019-07-13T09:28:32Z

@andrelop The renamed kernel that works for me is arch/x86/boot/bzImage which is 7Mb in size.

edit: Since there is no Microsoft folder in that v4.19.57.tar.gz file it will be better to download the WSL2 kernel from https://github.com/microsoft/WSL2-Linux-Kernel/archive/master.zip

PatrickLang · 2019-07-19T21:09:26Z

@simonferquel - where's your modified kconfig?

Biswa96 · 2019-07-21T15:17:29Z

@PatrickLang wsl kernel config is updated with those flags. See the commit on Jul 21, 2019.

ioanc · 2019-08-07T19:55:59Z

thank you guys and girls !!!
with the steps provided here, I compiled a new kernel with the new config-wsl
now running "k3s server" on Ubuntu 18.04 inside wls2 successfully

kamkie · 2019-11-16T10:10:34Z

I think this works now

goodwill · 2020-07-30T10:31:28Z

May I know how to use this insider build kernel? As I am trying to run docker on surface pro X wsl2 I seems come across the issue on iptable support of the kernel as well.

maxboone · 2021-05-25T10:53:26Z

May I know how to use this insider build kernel? As I am trying to run docker on surface pro X wsl2 I seems come across the issue on iptable support of the kernel as well.

The kernel has been available for the general public for a while, you're most probably using nftables on Debian or Ubuntu. Otherwise, you can check the following link: docs.microsoft.com/en-us/windows/wsl/install-win10

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

benhillis self-assigned this Jun 17, 2019

benhillis added the wsl2 Issue/feature applies to WSL 2 label Jun 17, 2019

therealkenc mentioned this issue Jun 17, 2019

[WSL2] Issues with running LXD (system containers) #4158

Closed

benhillis added the fixinbound label Jun 17, 2019

seanson mentioned this issue Jun 19, 2019

Can not run on wsl2 k3s-io/k3s#537

Closed

This was referenced Jul 2, 2019

[Feature Request] podman Windows WSL support containers/podman#3288

Closed

Failed to start service Docker on WSL2 WhitewaterFoundry/Pengwin#485

Open

rsrini7 mentioned this issue Jul 10, 2019

Cannot upgrade to version 2 even when Virtual Machine Platform is enabled MicrosoftDocs/WSL#404

Closed

luxas mentioned this issue Jul 11, 2019

Cannot start VMs on WSL2 weaveworks/ignite#129

Open

PatrickLang mentioned this issue Jul 18, 2019

Failed to interact with cluster in wsl2 kubernetes-sigs/kind#707

Closed

LouisStAmour mentioned this issue Sep 26, 2019

Support for Windows 10 WSL2 kubernetes/minikube#5392

Closed

benhillis closed this as completed Nov 16, 2019

therealkenc added fixedininsiderbuilds and removed fixinbound labels Feb 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WSL2: dockerd related missing kernel flags #4165

WSL2: dockerd related missing kernel flags #4165

simonferquel commented Jun 17, 2019

simonferquel commented Jun 17, 2019

simonferquel commented Jun 17, 2019

benhillis commented Jun 17, 2019

simonferquel commented Jun 17, 2019

benhillis commented Jun 17, 2019 •

edited

Biswa96 commented Jun 17, 2019

simonferquel commented Jun 17, 2019 via email

therealkenc commented Jun 18, 2019

MortenRickiRasmussen commented Jun 18, 2019

simonferquel commented Jun 18, 2019

ibuildthecloud commented Jun 19, 2019

Biswa96 commented Jun 28, 2019

simonferquel commented Jun 28, 2019 via email

andrelop-zz commented Jul 10, 2019

Biswa96 commented Jul 11, 2019

therealkenc commented Jul 11, 2019

andrelop-zz commented Jul 13, 2019

onomatopellan commented Jul 13, 2019 •

edited

PatrickLang commented Jul 19, 2019 •

edited

Biswa96 commented Jul 21, 2019

ioanc commented Aug 7, 2019 •

edited

kamkie commented Nov 16, 2019

goodwill commented Jul 30, 2020

maxboone commented May 25, 2021 •

edited

WSL2: dockerd related missing kernel flags #4165

WSL2: dockerd related missing kernel flags #4165

Comments

simonferquel commented Jun 17, 2019

simonferquel commented Jun 17, 2019

simonferquel commented Jun 17, 2019

benhillis commented Jun 17, 2019

simonferquel commented Jun 17, 2019

benhillis commented Jun 17, 2019 • edited

Biswa96 commented Jun 17, 2019

simonferquel commented Jun 17, 2019 via email

therealkenc commented Jun 18, 2019

MortenRickiRasmussen commented Jun 18, 2019

simonferquel commented Jun 18, 2019

ibuildthecloud commented Jun 19, 2019

Biswa96 commented Jun 28, 2019

simonferquel commented Jun 28, 2019 via email

andrelop-zz commented Jul 10, 2019

Biswa96 commented Jul 11, 2019

therealkenc commented Jul 11, 2019

andrelop-zz commented Jul 13, 2019

onomatopellan commented Jul 13, 2019 • edited

PatrickLang commented Jul 19, 2019 • edited

Biswa96 commented Jul 21, 2019

ioanc commented Aug 7, 2019 • edited

kamkie commented Nov 16, 2019

goodwill commented Jul 30, 2020

maxboone commented May 25, 2021 • edited

benhillis commented Jun 17, 2019 •

edited

onomatopellan commented Jul 13, 2019 •

edited

PatrickLang commented Jul 19, 2019 •

edited

ioanc commented Aug 7, 2019 •

edited

maxboone commented May 25, 2021 •

edited