Kernel configuration for supporting docker in swarm mode

[atzoum]

This is mostly about adding IPVS & VXLAN support in the kernel.
To find out about the current missing kernel features docker has a check-config.sh script:

curl https://raw.githubusercontent.com/docker/docker/v1.12.5/contrib/check-config.sh | bash

Still, even with this change, some special vxlan configuration is necessary due to the old (3.10) kernel, which could be handled in the make_rootfs.sh script by adding something similar to the following

echo 'options vxlan udp_port=4789' >> /etc/modprobe.d/vxlan.conf

see moby/moby#18929 for details.

[adomenech73]

@atzoum I've been heading the same problem for a while.

Initially I could make overlay networks work by myself enabling CONFIG_VXLAN as module. Other problem I've seen is about OVERLAYFS_FS that has been renamed to OVERLAY_FS in kernels > 3.18 and probably would need some kind of backport.

So I've seen your PR and tried to apply the .config proposed changes manually.
Unfortunately I think that some kernel options here are not available for 3.10.104
I've seen now, they are suboptions, I've been asked while compiling, excuses!!

On the other hand I could apply this ones:
CONFIG_RT_GROUP_SCHED, CONFIG_IOSCHED_CFQ, CONFIG_CFQ_GROUP_IOSCHED, CONFIG_XFRM_USER, CONFIG_IP_VS, CONFIG_VXLAN

I'll regenerate my cards with this enabled options and let you know any feedback

[atzoum]

Hi, please note that OVERLAY_FS is not backwards compatible with OVERLAYFS_FS, that's why they did rename it and kernel 3.10 doesn't support the new version.
https://patchwork.kernel.org/patch/5337851/
However you may still have docker running on aufs driver which is the default.

[umiddelb]

You may consider to run a patched 4.9 mainline kernel on your Pine64. At this time µSD and GB Ethernet is supported but no USB. The mainline kernel needs a different firmware (u-boot) than the BSP kernel, so you need to prepare another µSD card.

[adomenech73]

@umiddelb I known that at the end the most reasonable thing to do, will be to use apritzel patched mainline kernel. I didn't think it was that advanced, cause last time I tried I even couldn't get it run. I'll try to build it on new SD's, as far as I know (correct me if I'm wrong) longsleep scripts are also usefull for that build.

About backporting OVERLAY_FS, I say it because I've seen this patchwork but I didn't realise it wasn't backwards compatible. I thougth it was related with overlay/overlay2 storage driver.

Initially CONFIG_VXLAN solved my carence of overlay networks on swarm cluster but now it seems it can't publish service ports throught bridge interface, I'm hopping to solve this with CONFIG_IP_VS activation.

@atzoum Do you know where to find aditional information about the Linux IO Schedulers, and their impact on docker?

[umiddelb]

I known that at the end the most reasonable thing to do, will be to use apritzel patched mainline kernel. I didn't think it was that advanced, cause last time I tried I even couldn't get it run. I'll try to build it on new SD's, as far as I know (correct me if I'm wrong) longsleep scripts are also usefull for that build.

Andre uses his own toolchain, but you can start with his prebuild firmware image and copy your existing pine64 userland to a newly crated second partition.

[atzoum]

Hi, I suppose you are referring to storage drivers? Docker does have sufficient documentation on the matter:
https://docs.docker.com/engine/userguide/storagedriver/selectadriver/

Update
Maybe you are referring to CONFIG_IOSCHED_CFQ... I am afraid I have no idea how docker is actually using it. This comment might be relevant:
moby/moby#21485 (comment)

[adomenech73]

@atzoum just tried your PR it's working nice and the first difference I could observe in docker info:

Network: bridge host macvlan null overlay

vs prior

Network: null bridge host overlay

Just for that it already worth it, for me!! nice work

About the question, no, I was more thinking about the Linux scheduler,because I saw in your PR that you enabled CONFIG_RT_GROUP_SCHED, CONFIG_IOSCHED_CFQ, CONFIG_CFQ_GROUP_IOSCHED options.

I already know that they where some problems for Linux scheduler managing multi core systems as pointed in this interesting paper

So I was asking, if you did this election for any specific requirement, or improvement on docker operation?

I understand that some kind of post configuration is needed to change the scheduler algorithm.

[atzoum]

@adomenech73 glad to hear it is working for you too. Regarding the other kernel options I simply enabled them because they were included in docker's own check-config.sh script. Shouldn't do any harm I guess :)

[adomenech73]

Absolutely ! just checked that services are now publishing as expected through bridge interface