Update or replace system-docker #28

dgiebert · 2020-12-13T11:11:24Z

Currently the version is currently pinned to 17.06

The buildsystem the following forks with the main change linked:

github.com/rancher/docker-cli changes
github.com/rancher/docker changes
github.com/rancher/docker-ce-packaging changes

Do you want to also create a fork or use patch files instead ?

olljanat · 2020-12-13T11:48:24Z

I have forked those repos under burmilla but I'm not sure if we really want upgrade system-docker because we don't really need those new features on it and binaries sizes has increased a lot of on those new versions.

docker-17.06.2-ce:
total 89M
-rwxr-xr-x 1 pi pi  18M Sep  5  2017 docker
-rwxr-xr-x 1 pi pi 9.5M Sep  5  2017 docker-containerd
-rwxr-xr-x 1 pi pi 8.6M Sep  5  2017 docker-containerd-ctr
-rwxr-xr-x 1 pi pi 3.7M Sep  5  2017 docker-containerd-shim
-rwxr-xr-x 1 pi pi  39M Sep  5  2017 dockerd
-rwxr-xr-x 1 pi pi 755K Sep  5  2017 docker-init
-rwxr-xr-x 1 pi pi 2.4M Sep  5  2017 docker-proxy
-rwxr-xr-x 1 pi pi 7.4M Sep  5  2017 docker-runc

docker-19.03.14:
total 200M
-rwxr-xr-x 1 pi pi  36M Dec  1 21:22 containerd
-rwxr-xr-x 1 pi pi 6.9M Dec  1 21:22 containerd-shim
-rwxr-xr-x 1 pi pi  19M Dec  1 21:22 ctr
-rwxr-xr-x 1 pi pi  59M Dec  1 21:22 docker
-rwxr-xr-x 1 pi pi  69M Dec  1 21:22 dockerd
-rwxr-xr-x 1 pi pi 693K Dec  1 21:22 docker-init
-rwxr-xr-x 1 pi pi 2.8M Dec  1 21:22 docker-proxy
-rwxr-xr-x 1 pi pi 9.2M Dec  1 21:22 runc

docker-20.10.0:
total 219M
-rwxr-xr-x 1 pi pi  38M Dec  8 20:59 containerd
-rwxr-xr-x 1 pi pi 7.0M Dec  8 20:59 containerd-shim
-rwxr-xr-x 1 pi pi 9.5M Dec  8 20:59 containerd-shim-runc-v2
-rwxr-xr-x 1 pi pi  21M Dec  8 20:59 ctr
-rwxr-xr-x 1 pi pi  53M Dec  8 20:59 docker
-rwxr-xr-x 1 pi pi  76M Dec  8 20:59 dockerd
-rwxr-xr-x 1 pi pi 693K Dec  8 20:59 docker-init
-rwxr-xr-x 1 pi pi 2.8M Dec  8 20:59 docker-proxy
-rwxr-xr-x 1 pi pi  13M Dec  8 20:59 runc

Also I can see that system-docker and system-dockerd binaries are smaller than same version from official Docker version so those are most probably patched somehow to only build only needed of the features.

docker-amd64-17.06-ros6:
total 67M
-rwxr-xr-x 1 pi pi 9.9M Feb 11  2019 system-docker
-rwxr-xr-x 1 pi pi 9.5M Feb 11  2019 system-docker-containerd
-rwxr-xr-x 1 pi pi 8.6M Feb 11  2019 system-docker-containerd-ctr
-rwxr-xr-x 1 pi pi 3.7M Feb 11  2019 system-docker-containerd-shim
-rwxr-xr-x 1 pi pi  25M Feb 11  2019 system-dockerd
-rwxr-xr-x 1 pi pi 755K Feb 11  2019 system-docker-init
-rwxr-xr-x 1 pi pi 2.4M Feb 11  2019 system-docker-proxy
-rwxr-xr-x 1 pi pi 7.4M Feb 11  2019 system-docker-runc

So most probably we should update ros command to all containerd directly so we can drop system-docker but I'm not sure if that is worth effort.

Alternative way is stay on 17.06 forever and only cherry-pick fixes from Moby if needed.

dgiebert · 2020-12-13T11:54:49Z

The reason for this is probably the removal of some features for system-docker:

I will try to compile and adapt this and post a size for system-docker

dgiebert · 2020-12-13T13:42:39Z

Actually after a successful built, the size did not shrink that much:

-rwxr-xr-x 1 dgiebert dgiebert  36M Dec 13 14:18 system-containerd*
-rwxr-xr-x 1 dgiebert dgiebert 6.9M Dec 13 14:18 system-containerd-shim*
-rwxr-xr-x 1 dgiebert dgiebert  19M Dec 13 14:18 system-ctr*
-rwxr-xr-x 1 dgiebert dgiebert  59M Dec 13 14:18 system-docker*
-rwxr-xr-x 1 dgiebert dgiebert  55M Dec 13 14:18 system-dockerd*
-rwxr-xr-x 1 dgiebert dgiebert 693K Dec 13 14:18 system-docker-init*
-rwxr-xr-x 1 dgiebert dgiebert 2.8M Dec 13 14:18 system-docker-proxy*
-rwxr-xr-x 1 dgiebert dgiebert 9.2M Dec 13 14:18 system-runc*

Checking the source code and diffs, a bit of google research showed the possibility of running multiple deamons: https://stackoverflow.com/questions/32334167/is-it-possible-to-start-multiple-docker-daemons-on-the-same-machine

This could be a better way to only maintain a single dockerd and just have a small wrapper script for system-docker

olljanat · 2020-12-13T14:07:35Z

Checking the source code and diffs, a bit of google research showed the possibility of running multiple deamons: https://stackoverflow.com/questions/32334167/is-it-possible-to-start-multiple-docker-daemons-on-the-same-machine

This could be a better way to only maintain a single dockerd and just have a small wrapper script for system-docker

@dgiebert first of all I would like to understand that what you are actually trying to achieve by updating system-docker? Main purpose of it is actually just works as package provider (like apt-get, yum, etc) for BurmillaOS so as long we don't have known bugs on it I don't see reason to change it.

olljanat · 2020-12-13T21:56:44Z

Go binaries growing looks to be generic problem golang/go#6853

I can see same on https://github.com/burmilla/docker/tree/test/update-build-scripts where I just took latest build scripts from Moby project so those gets build with later version of Go. That issue can be partly fixed by optimizing build parameters but not fully.

dgiebert · 2020-12-14T07:40:29Z

Also quite visible here in the official stable builds: https://download.docker.com/linux/static/stable/x86_64/
Will try to optimize the build with the compiler flags later today, any tipps ?

Also from the linked issue, seems that in 1.15 the size could be reduced but as of now moby is on 1.13.

olljanat · 2020-12-14T09:20:23Z

They are working on to get it updated moby/moby#40353

Also if we start working on this then we probably should target directly to 20.10.x (or even 21.03.x) because major refactoring have been done to Moby on that version (backend code have been removed from Moby used from containerd) and afaiu that work still continues.

dgiebert · 2020-12-15T11:38:45Z

Very good point indeed, changed to 20.10.1 !

Old Size (19.03.14): ~187M
New Size(20.10.1) using 1.15.5: ~177M

-rwxr-xr-x 1 root root  33M Dec 15 12:30 system-containerd*
-rwxr-xr-x 1 root root 6.5M Dec 15 12:30 system-containerd-shim*
-rwxr-xr-x 1 root root  21M Dec 15 12:30 system-ctr*
-rwxr-xr-x 1 root root  49M Dec 15 12:30 system-docker*
-rwxr-xr-x 1 root root  52M Dec 15 12:30 system-dockerd*
-rwxr-xr-x 1 root root 693K Dec 15 12:30 system-docker-init*
-rwxr-xr-x 1 root root 2.8M Dec 15 12:30 system-docker-proxy*
-rwxr-xr-x 1 root root  12M Dec 15 12:30 system-runc*

This was just plain 1.13.15 -> 1.15.5 changes, so I believe more can be achieved here

dgiebert · 2020-12-15T15:01:45Z

Following this article here: https://boyter.org/posts/trimming-golang-binary-fat/

-rwxr-xr-x 1 root root  11M Dec 15 14:08 system-containerd*
-rwxr-xr-x 1 root root 2.1M Dec 15 14:08 system-containerd-shim*
-rwxr-xr-x 1 root root  21M Dec 15 14:08 system-ctr*
-rwxr-xr-x 1 root root  25M Dec 15 14:08 system-docker*
-rwxr-xr-x 1 root root  15M Dec 15 14:08 system-dockerd*
-rwxr-xr-x 1 root root 277K Dec 15 14:08 system-docker-init*
-rwxr-xr-x 1 root root 1.6M Dec 15 14:08 system-docker-proxy*
-rwxr-xr-x 1 root root 5.0M Dec 15 14:08 system-runc*

Size after UPX compression: ~81M

olljanat · 2020-12-15T16:25:33Z

Interesting that Docker CLI is bigger than Docker daemon it selves but of course that makes sense as part of logic have been moved to containerd nowadays.

This made me thinking that it might possible actually replace "system-docker" with shell script which call "docker" command with needed parameters because boot process does not need it afaik:

$ sudo docker -H unix:///var/run/system-docker.sock ps
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS              PORTS               NAMES
19b253c34be2        burmilla/os-docker:19.03.14         "ros user-docker"        36 hours ago        Up 36 hours                             docker
13b35a796613        burmilla/os-console:v1.9.0          "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             console
f3e81544617a        burmilla/os-base:v1.9.0             "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             ntp
0229f9a12949        burmilla/os-base:v1.9.0             "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             network
0f86354032d3        burmilla/os-base:v1.9.0             "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             udev
37ac8ebfb9df        burmilla/container-crontab:v0.5.0   "container-crontab"      36 hours ago        Up 36 hours                             system-cron
742bdbd5c705        burmilla/os-syslog:v1.9.0           "/usr/bin/entrypoint…"   36 hours ago        Up 36 hours                             syslog
2c50b2258fda        burmilla/os-acpid:v1.9.0            "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             acpid

Also I think that we do not need "system-docker-containerd-ctr" / "system-ctr" at all and "system-docker-proxy" gets removed during build process on:

os/scripts/layout-initrd

Lines 33 to 34 in 93de612

    
           # we have diabled the user-proxy so we get rid of system-docker-proxy 
        
           rm -f ${INITRD_DIR}/usr/bin/system-docker-proxy

What comes to idea to use UPX, system-docker is actually packages inside of initrd which is already compressed so we should check how this affect size of it and iso file:

os/scripts/package-initrd

Lines 19 to 27 in 93de612

    
           if [ "$COMPRESS" == "" ]; then 
        
               COMPRESS="gzip -1" 
        
           fi 
        
           cd ${INITRD_DIR} 
        
           echo Creating ${ARTIFACTS}/${INITRD} 
        
           find | cpio -H newc -o | ${COMPRESS} > ${ARTIFACTS}/${INITRD} 
        
           echo Done creating ${ARTIFACTS}/${INITRD}

Also boot time and memory usage are areas which we need follow when we changing these as there is already comment about high CPU usage with too much compression:

os/scripts/tar-images

Lines 47 to 54 in 93de612

    
           if [ "$COMPRESS" == "" ]; then 
        
               ARCHIVE_CMD="gzip" 
        
           else 
        
               # system-docker can not load images which compressed by xz with a compression level of 9 
        
               # decompression consumes more memory if using level 9 
        
               # the default compression level for xz is 6 
        
               ARCHIVE_CMD="xz -4 -e" 
        
           fi

dgiebert · 2020-12-15T16:51:24Z

Yes, I think it is very easy to replace system-docker with a two line bash script in the console image.

You are correct did not think about the initrd compression, but I guess if we strip off all unneeded / duplicated executables the resulting size should not exceed the 17.06 build by much

olljanat · 2020-12-15T20:08:40Z

This could be a better way to only maintain a single dockerd and just have a small wrapper script for system-docker

Hmm. Maybe that is actually best way because if we decide that system-docker and default user docker versions are same then we can:

Cache files from https://github.com/burmilla/os-services/tree/master/images/10-docker-20.10.1 + downloaded Docker binaries on build.
Start system-docker directly from those binaries or with wrapper scripts (I'm not sure if we even need wrapper or maybe it can be simple which denies example swarm and plugin commands).
Start user docker with busybox image which actually mounts those same binaries inside container (so there is not need import those as Docker image like now).

Only special things needed on that one would be that we need modify settings on here:

os/os-config.tpl.yml

Lines 361 to 385 in 4e5d248

    
           docker: 
        
             {{if eq "amd64" .ARCH -}} 
        
             image: {{.OS_REPO}}/os-docker:{{.USER_DOCKER_VERSION}}{{.SUFFIX}} 
        
             {{else -}} 
        
             image: {{.OS_REPO}}/os-docker:{{.USER_DOCKER_VERSION}}{{.SUFFIX}} 
        
             {{end -}} 
        
             command: ros user-docker 
        
             environment: 
        
             - HTTP_PROXY 
        
             - HTTPS_PROXY 
        
             - NO_PROXY 
        
             labels: 
        
               io.rancher.os.scope: system 
        
               io.rancher.os.after: console 
        
             net: host 
        
             pid: host 
        
             ipc: host 
        
             uts: host 
        
             privileged: true 
        
             restart: always 
        
             volumes_from: 
        
             - all-volumes 
        
             volumes: 
        
             - /sys:/host/sys 
        
             - /var/lib/system-docker:/var/lib/system-docker:shared

Then it should (on theory) works similarly like consoles was earlier that default settings will be overridden when user uses sudo ros engine switch command (as on that point it need to use version specific configuration from os-service repo) and probably we need also add "default" option to that list which always points to Docker version which is bundled to ISO file.

olljanat · 2020-12-19T22:54:02Z

@dgiebert so how you now build those 20.10.1 binaries? I'm prepairing next 2.0.0 beta version so probably we should include it already and implement those optimizations later.

dgiebert · 2020-12-20T14:29:17Z

The main issue I see with the shared dockerd, would be that an upgrade of the engine would need a reboot.

I will try to get a build working today @olljanat

dgiebert · 2020-12-21T14:05:40Z

@olljanat seems like the console is starting and tells started but its not attaching. Do you have a pointer where this could be hanging ?

olljanat · 2020-12-21T18:56:20Z

@dgiebert can you share those codes which you are now using so it is easier to guess what might be wrong?

olljanat · 2021-04-25T10:25:20Z

Instead of updating system-docker we should also consider:

removing it like proposed on Proposal: New architecture for v3.0 #88
replacing it example with containerd and nerdctl combination

olljanat · 2023-01-22T12:53:37Z

I finally managed to build working version of system-docker from sources (earlier we have used binary build by Rancher) https://github.com/burmilla/os-system-docker/releases/tag/17.06.107

What I did find out that it is possible to build system-docker even with latest Go version (draft in here) but there is some change done in Go it selves which breaks it in way that runc is not able to start console container anymore.

replacing it example with containerd and nerdctl combination

I think that this would be best long term solution. We can keep size in control by disabling all those optional components https://github.com/containerd/containerd/blob/v1.6.15/BUILDING.md#build-containerd

olljanat · 2023-02-08T22:39:23Z

It looks to be that replacing system-docker with containerd + nerdctl would need massive refactoring for code from where ros binary is build because we use https://github.com/docker/libcompose/ and nerdctl uses https://github.com/compose-spec/compose-go

However https://github.com/balena-os/balena-engine potential drop-in replacement for system-docker so will test it some day.

olljanat · 2023-04-04T10:47:32Z

Update. Replacing system-docker with balena-engine is simple but looks that new runc version makes system crashing on boot. No matter if that is used with balena or system-docker.

Test versions of those exist in https://github.com/burmilla/os/tree/migrate-to-balena and https://github.com/burmilla/os-system-docker/releases/tag/17.06.109

pwFoo · 2023-04-04T11:13:34Z

Hi @olljanat
interesting topic... I heared about balena, but don't know about balena cloud / open balena...
Would it possible to manage instances with balena cloud / open balena if based on balena-engine?

Are the problems related to runc or maybe test it with crun?

olljanat · 2023-04-04T15:27:28Z

Would it possible to manage instances with balena cloud / open balena if based on balena-engine?

No. balena-engine is just minimized version of docker/moby engine (swarm mode is removed, etc) so very similar than how we use system-docker, just based on more modern version of docker and maintained by Balena team (the main reason to which to it).

Are the problems related to runc or maybe test it with crun?

That is new for me. Will look about it.

olljanat added the enhancement New feature or request label Dec 13, 2020

dgiebert changed the title ~~Update system-docker to 19.03.14~~ Update system-docker Dec 14, 2020

olljanat added the version/v2.0.x label Dec 15, 2020

dgiebert mentioned this issue Dec 22, 2020

WIP: Upgrade system-docker #39

Closed

tomaswarynyca linked a pull request Dec 22, 2020 that will close this issue

WIP: Upgrade system-docker #39

Closed

olljanat mentioned this issue Apr 24, 2021

Proposal: New architecture for v3.0 #88

Closed

olljanat changed the title ~~Update system-docker~~ Update or replace system-docker Apr 25, 2021

olljanat mentioned this issue Apr 25, 2021

How to upgrade os-kernel with source code from kernel.org? #78

Closed

PrplHaz4 mentioned this issue Jan 20, 2023

System-docker version number is incompatible with cAdvisor #151

Closed

olljanat mentioned this issue Apr 4, 2023

v2 branch updates? #134

Closed

olljanat closed this as not planned Won't fix, can't repro, duplicate, stale Aug 20, 2023

olljanat mentioned this issue Mar 5, 2024

v2.0.0-rc1 issue tracker #161

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update or replace system-docker #28

Update or replace system-docker #28

dgiebert commented Dec 13, 2020

olljanat commented Dec 13, 2020

dgiebert commented Dec 13, 2020

dgiebert commented Dec 13, 2020

olljanat commented Dec 13, 2020

olljanat commented Dec 13, 2020

dgiebert commented Dec 14, 2020

olljanat commented Dec 14, 2020

dgiebert commented Dec 15, 2020

dgiebert commented Dec 15, 2020

olljanat commented Dec 15, 2020

dgiebert commented Dec 15, 2020

olljanat commented Dec 15, 2020 •

edited

Loading

olljanat commented Dec 19, 2020

dgiebert commented Dec 20, 2020

dgiebert commented Dec 21, 2020

olljanat commented Dec 21, 2020

olljanat commented Apr 25, 2021

olljanat commented Jan 22, 2023

olljanat commented Feb 8, 2023 •

edited

Loading

olljanat commented Apr 4, 2023

pwFoo commented Apr 4, 2023

olljanat commented Apr 4, 2023

Update or replace system-docker #28

Update or replace system-docker #28

Comments

dgiebert commented Dec 13, 2020

olljanat commented Dec 13, 2020

dgiebert commented Dec 13, 2020

dgiebert commented Dec 13, 2020

olljanat commented Dec 13, 2020

olljanat commented Dec 13, 2020

dgiebert commented Dec 14, 2020

olljanat commented Dec 14, 2020

dgiebert commented Dec 15, 2020

dgiebert commented Dec 15, 2020

olljanat commented Dec 15, 2020

dgiebert commented Dec 15, 2020

olljanat commented Dec 15, 2020 • edited Loading

olljanat commented Dec 19, 2020

dgiebert commented Dec 20, 2020

dgiebert commented Dec 21, 2020

olljanat commented Dec 21, 2020

olljanat commented Apr 25, 2021

olljanat commented Jan 22, 2023

olljanat commented Feb 8, 2023 • edited Loading

olljanat commented Apr 4, 2023

pwFoo commented Apr 4, 2023

olljanat commented Apr 4, 2023

olljanat commented Dec 15, 2020 •

edited

Loading

olljanat commented Feb 8, 2023 •

edited

Loading