Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WSL2: dockerd related missing kernel flags #4165

Open
simonferquel opened this issue Jun 17, 2019 · 21 comments

Comments

@simonferquel
Copy link

commented Jun 17, 2019

By running https://github.com/moby/moby/blob/master/contrib/check-config.sh within wsl2, I just produced a report of missing things for running dockerd with all features enabled:

simon@simonxps:~/go/src/github.com/docker/docker/contrib$ ./check-config.sh
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: missing
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: missing
- CONFIG_IP_NF_NAT: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_LEGACY_VSYSCALL_NONE: enabled
    (containers using eglibc <= 2.13 will not work. Switch to
     "CONFIG_VSYSCALL_[NATIVE|EMULATE]" or use "vsyscall=[native|emulate]"
     on kernel command line. Note that this will disable ASLR for the,
     VDSO which may assist in exploiting security vulnerabilities.)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_IOSCHED_CFQ: missing
- CONFIG_CFQ_GROUP_IOSCHED: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled
- CONFIG_IP_VS: enabled
- CONFIG_IP_VS_NFCT: missing
- CONFIG_IP_VS_PROTO_TCP: missing
- CONFIG_IP_VS_PROTO_UDP: missing
- CONFIG_IP_VS_RR: missing
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: missing
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: missing
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: missing
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: enabled
  - "macvlan":
    - CONFIG_MACVLAN: enabled
    - CONFIG_DUMMY: missing
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: missing
    - CONFIG_NF_CONNTRACK_FTP: missing
    - CONFIG_NF_NAT_TFTP: missing
    - CONFIG_NF_CONNTRACK_TFTP: missing
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

cc @benhillis, I think it should help.
I'll try to make my own kernel with all this setup accordingly, and will do experiment to setup kubernetes in a similar way as what we do in Docker Desktop to see how things go.

@simonferquel

This comment has been minimized.

Copy link
Author

commented Jun 17, 2019

I can confirm that adding missing flags makes it possible to run swarm and netwoking seems to work.
However, logs get polluted with warnings like:

time="2019-06-17T16:51:41+02:00" level=warning msg="Running modprobe nf_nat failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.43-microsoft-standard/modules.dep.bin'\nmodprobe: WARNING: Module nf_nat not found in directory /lib/modules/4.19.43-microsoft-standard`, error: exit status 1"
@simonferquel

This comment has been minimized.

Copy link
Author

commented Jun 17, 2019

(it does also that for xt_conntrack)

@benhillis

This comment has been minimized.

Copy link
Member

commented Jun 17, 2019

Thanks @simonferquel, I will make sure all those config options get added to the kernel config. Since the kernel that we are using for WSL2 includes modules statically instead of in a separate folder those warnings are benign. It would likely be good to add a flag to docker daemon to suppress those errors for WSL2.

@benhillis benhillis self-assigned this Jun 17, 2019

@benhillis benhillis added the wsl2 label Jun 17, 2019

@simonferquel

This comment has been minimized.

Copy link
Author

commented Jun 17, 2019

That is what I suspected.
Got a try with kubernetes, we miss CONFIG_NETFILTER_XT_MATCH_COMMENT apparently (at least). Can you add it to the list ?

@benhillis

This comment has been minimized.

Copy link
Member

commented Jun 17, 2019

@simonferquel - Yes, I will add that as well. If that is needed, should it be added to the check-config script?

@Biswa96

This comment has been minimized.

Copy link

commented Jun 17, 2019

@simonferquel The WSL-ed Linux kernel source here https://thirdpartysource.microsoft.com/download/Windows%20Subsystem%20for%20Linux%20v2/May%202019/WSLv2-Linux-Kernel-master.zip. Found the kernel configs at WSLv2-Linux-Kernel-master\Microsoft\config-wsl.

@simonferquel

This comment has been minimized.

Copy link
Author

commented Jun 17, 2019

@therealkenc

This comment has been minimized.

Copy link
Collaborator

commented Jun 18, 2019

Found the kernel configs at WSLv2-Linux-Kernel-master\Microsoft\config-wsl.

You can get that at runtime with:

$ sudo cat /proc/config.gz | gunzip -

Which is a nice touch, since we don't have /boot/config-4.19.43-microsoft-standard. [/proc/config.gz isn't really common on distros these days (including Ubuntu) because they have /boot.]

@MortenRickiRasmussen

This comment has been minimized.

Copy link

commented Jun 18, 2019

Will this issue fix the VXLAN issue?

I'am getting this when trying to use networl overlays in Docker Swarm
network sandbox join failed: subnet sandbox join failed for "172.172.0.0/16": error creating vxlan interface: operation not supported

@simonferquel

This comment has been minimized.

Copy link
Author

commented Jun 18, 2019

@MortenRickiRasmussen yes. I have a built a kernel with a modified kconfig file, and I can expose ports on the ingress network, create and attach to overlay networks etc.

@ibuildthecloud

This comment has been minimized.

Copy link

commented Jun 19, 2019

Related to this is that Kubernetes needs CONFIG_NETFILTER_XT_MATCH_COMMENT as the use -m comment --comment ... on iptables.

@Biswa96

This comment has been minimized.

Copy link

commented Jun 28, 2019

@simonferquel Can you mention which configs you have changed?

@simonferquel

This comment has been minimized.

Copy link
Author

commented Jun 28, 2019

@andrelop

This comment has been minimized.

Copy link

commented Jul 10, 2019

Hello,

Plese see weaveworks/ignite#129 .

My guess is that the issue is somewhat related to CONFIG_DM_THIN_PROVISIONING not being enabled by default.

Woud you please consider enabling it for the next release ?

@Biswa96

This comment has been minimized.

Copy link

commented Jul 11, 2019

@andrelop You can compile your own kernel and place it in System32\lxss\tools folder. We don't have to wait for next builds.

@therealkenc

This comment has been minimized.

Copy link
Collaborator

commented Jul 11, 2019

You can compile your own kernel and place it in System32\lxss\tools folder.

Or compile it as a module, in which case you don't need to muck lxss/tools. CONFIG_DM_THIN_PROVISIONING is =m on Real Ubuntu FWIW. Along with all the other CONFIG_DM_*.

@andrelop

This comment has been minimized.

Copy link

commented Jul 13, 2019

Hello

@andrelop You can compile your own kernel and place it in System32\lxss\tools folder. We don't have to wait for next builds.

Reproducing here what I posted at weaveworks/ignite#129 to see if you could help with the correct procedure to build/use a custom kernel for a given WSL2 VM :

I tried building a custom kernel but the WSL2 VM cannot start after I change the C:\Windows\System32\lxss\tools\kernel using the vmlinux kernel produced by the kernel build process.

What I did :

  • Downloaded the last release on microsoft/WSL2-Linux-Kernel github repo (https://github.com/microsoft/WSL2-Linux-Kernel/archive/v4.19.57.tar.gz)
  • Unpacked it and used the provided kernel config on https://github.com/microsoft/WSL2-Linux-Kernel/blob/master/Microsoft/config-wsl
  • Ran make pointing to this config (ie. using make KCONFIG_CONFIG=config.wsl)
  • At the end of the build, renamed the existing C:\Windows\System32\lxss\tools\kernel file to something else (ie. C:\Windows\System32\lxss\tools\kernel.old) and renamed/moved the vmlinux generated file as C:\Windows\System32\lxss\tools\kernel
  • Forced all the WSL2 VMs shutdown using wsl --shutdown on PowerShell as Administrator
  • Started the WSL2 VM again, but it did not boot
  • Restored the previously renamed file in order to get the VM booting again

Not sure what else I could try or if what I did was really what needed to be done.

@onomatopellan

This comment has been minimized.

Copy link

commented Jul 13, 2019

@andrelop The renamed kernel that works for me is arch/x86/boot/bzImage which is 7Mb in size.

edit: Since there is no Microsoft folder in that v4.19.57.tar.gz file it will be better to download the WSL2 kernel from https://github.com/microsoft/WSL2-Linux-Kernel/archive/master.zip

@PatrickLang

This comment has been minimized.

Copy link
Member

commented Jul 19, 2019

@simonferquel - where's your modified kconfig?

@Biswa96

This comment has been minimized.

Copy link

commented Jul 21, 2019

@PatrickLang wsl kernel config is updated with those flags. See the commit on Jul 21, 2019.

@ioanc

This comment has been minimized.

Copy link

commented Aug 7, 2019

thank you guys and girls !!!
with the steps provided here, I compiled a new kernel with the new config-wsl
now running "k3s server" on Ubuntu 18.04 inside wls2 successfully
wsl2-k3s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.