Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix docker-proxy and ctr not statically linked #43492

Closed
wants to merge 3 commits into from

Conversation

crazy-max
Copy link
Member

@crazy-max crazy-max commented Apr 15, 2022

- What I did

fix docker-proxy and ctr (containerd cli) not statically linked while building against the binary or cross targets.

- How I did it

while adding a static check in docker/docker-ce-packaging#665 (see docker/docker-ce-packaging@835cc08) for our bundles, I discovered some of our artifacts were not statically linked anymore: https://github.com/docker/docker-ce-packaging/runs/6032211065?check_suite_focus=true#step:5:1183

Seems to be a regression on the master branch of moby, not reproducible on 20.10 branch.

Let's take a look at the current static bundle with the latest stable release:

$ wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.14.tgz
$ tar -xf docker-20.10.14.tgz
$ file docker/*
docker/containerd:              ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, Go BuildID=eZIxaJEWHi0bBulGF6EL/hXw2jCrsLi4rG2f1RaIR/dx4aJMpl_IEa-ZTGoaKR/XQI4cLxIgIpFj-4z_4p5, BuildID[sha1]=650d0f073a08fade2e7c1df1c6017d9232acdb3d, stripped
docker/containerd-shim:         ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=LPvnNOZI29K8tZbn1saT/22N-w3YJMyDdeg_DlfvP/2X7HXzk8p99tA1-PyrLQ/XGXlCV4ZiZtHjsAD903Q, stripped
docker/containerd-shim-runc-v2: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=9xDrbF6O_ETO1uKx8mff/Aq-318-9bB8Ex_bloYOA/QP06ImNmuf-LLQHMnP59/w6YERncZcj4wb3Xf4bRw, stripped
docker/ctr:                     ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=iuAtm-XQYYwep4ytGxxt/X_Wqo-awuqq8NYm0k7K1/w7v-3tslz_kRBURRBtUe/t0UOaNUIXCWk7zEA7TdZ, stripped
docker/docker:                  ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=NdBw6yuLBt8CusYesHMg/d__QnDEolA243eJTXOMr/tY-SB-semM7n9YuDZuld/ZLc4WCIWPZVM87kDiZBb, with debug_info, not stripped
docker/docker-init:             ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=234a857818ee06ee47ec7fa11fcdf19299305723, stripped        
docker/docker-proxy:            ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=RcHPiVbNJ2oUN7Cb_4yv/3Y_H-Q3BgaPeQjfc5apR/RP65P_Xs1UVQ6dxVQ3wj/Pk229eBVNNBrElXFBSLp, not stripped
docker/dockerd:                 ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, Go BuildID=o2mmPyuLm6cxNDsXEEHd/dfpKHZEmLbU4mx64sZog/84W4j7SEAyvLpn5zxNOl/iZYF6xMiLVoLqm11vLPV, BuildID[sha1]=8b1dceac8e27a29ea76353405d6b563b0423b067, not stripped
docker/runc:                    ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, Go BuildID=efZOjB8_WTTvyro5kKVn/xbeNCRj_DCctGdFWNsAk/pS743qsXlPZEgFq2oCdz/JU-04DdVgzIvqGSL7uxA, BuildID[sha1]=9a06b901f37fab09ca5907c65f64ee07b571d22f, not stripped

As we can see all shipped artifacts are statically linked (except ctr).

If we create a static build on docker-ce-packaging against the master branch we have the following result:

bundles/cross/linux/amd64-daemon/containerd:                           ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=d5176ea843e7e2a4d4893a9b2f393bbca5d9b927, for GNU/Linux 3.2.0, stripped
bundles/cross/linux/amd64-daemon/containerd-shim:                      ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=urMD5YbT1XPvbaja9ZUU/lr8QECLdZxyrNTbNhUL8/uFAZD4Gh1AmP5bu65l1r/B8uM-uIrTzP-Q5ZrLWwF, stripped
bundles/cross/linux/amd64-daemon/containerd-shim-runc-v2:              ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=o-G871C2i-0As_lgwy5U/o93O4zYaIBjjrf4Hnwe3/JtovZiD4jBgWfpanYFw0/2mQTQVk1_Nfb-l_q8Kbd, stripped
bundles/cross/linux/amd64-daemon/ctr:                                  ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=VFfTO_vsiPDxSoARBquw/MAA7g05nJsw3Q39eMcsC/uiNhbgkhLJHLusRex-C2/b68DWBql3Dtp_KpCghvn, stripped
bundles/cross/linux/amd64-daemon/docker-init:                          ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=22b9ed474ff9b95396473ddadf16a1f5ba13ca7c, for GNU/Linux 3.2.0, stripped
bundles/cross/linux/amd64-daemon/docker-proxy-dev:                     ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=_YRanIyh2PHKViijXEgF/AxZAh5M9el3fls1KOeY_/ae5-aXhQAF3sORpAyQPl/wFEq0h73qrmbbYdklMqb, not stripped
bundles/cross/linux/amd64-daemon/dockerd-dev:                          ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=1b64025d7b05daddf6b7e2909a018e2796bd6291, for GNU/Linux 3.2.0, not stripped
bundles/cross/linux/amd64-daemon/rootlesskit:                          ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=vrHsirkFpCc4rnKra_bU/evJy7oTyXZNLz7SNFMHc/cT5yTrJ-PI7ehVbQA2FO/QFv6ZFjCngqxZHd68c3h, not stripped
bundles/cross/linux/amd64-daemon/rootlesskit-docker-proxy:             ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=9LtSP-cnB_UUL1c1Kmmp/sn8jO50xmeez8OQbmJBD/8rLTsyb7K6ZWdvs7Sgfw/1sHaG7NHV7YmTIuYzQ39, not stripped
bundles/cross/linux/amd64-daemon/runc:                                 ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=a6ada9a48b8d654a2128c28be07bc596725fd9c2, for GNU/Linux 3.2.0, not stripped
bundles/cross/linux/amd64-daemon/vpnkit:                               ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, with debug_info, not stripped

As you can see there is an issue with docker-proxy and ctr.

For docker-proxy it seems to be linked to #42539 where -linkmode=external is missing as it was previously. Adding it back looks good but pie mode might be enough? WDYT @tonistiigi @cpuguy83?

Same for ctr where -linkmode external is missing according to the documentation: https://github.com/containerd/containerd/blob/main/BUILDING.md#static-binaries cc @AkihiroSuda

It seems ctr is not statically linked since 20.10.8:

$ wget https://download.docker.com/linux/static/stable/aarch64/docker-20.10.7.tgz
$ tar -xf docker-20.10.7.tgz
$ file docker/*
docker/containerd:              ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, Go BuildID=x2buDIUb3kccO-81lMNg/C7koedI7iMDet295_hjz/WHdb6wE8QmAZGaBspjcH/IkvJroDls-y02lRJRI7j, BuildID[sha1]=f64014a1dec45671f1cc3719e0d8ee40d5b87392, stripped
docker/containerd-shim:         ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=1el2sAbHskn_vZyM4Trd/FsNq5PdwTRHERLZJOXM0/AA65zZhe3qgrOjxXH6RQ/K0YNiwqHF2JMPpPmsc7L, stripped
docker/containerd-shim-runc-v2: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=rsidWw6MWs0UzZIF7DTR/U6_h7ngNBrwMznkBRUrl/ZhiCBgk2r7_PCxIE_g27/_Lz7ykTrpprNy7fDNcoq, stripped
docker/ctr:                     ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, Go BuildID=BS9XvLvMUlUO0mMVMlsT/OTq-tBdAPtkrGXO1z6e1/aAgdeOgMJwXd6sU5m0nK/jB4BFgfL4hJXXybeYQOr, BuildID[sha1]=9bf1b4be4dd0511fbf65f4a221dfafbca208b34d, stripped
docker/docker:                  ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=GaaKWDU8PHSIM_zE9Bhv/MVIDF_vymvXAVSTHbas1/_OcNooi-r-apSQigQy4e/3r4ZgOQzElXWSCfk0ZZU, with debug_info, not stripped
docker/docker-init:             ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, BuildID[sha1]=b7f88c2a8a0f401c6c6dce469db51d28f8b68a4b, stripped
docker/docker-proxy:            ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=O5OdwQB0KY5_rtza4F4Z/pmQ24CNPjhQDFVRqdQms/NBlCj9DPTW0sdS8YO-eT/HnXW6okLCgmwSDwFTp9w, not stripped
docker/dockerd:                 ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, Go BuildID=YVBaTaYN7yGB7N5ghuPV/n_zlVnaWqmtNtHTMZ0U8/dwN1PUByEPF4SFznjzzO/X_AQZ1DZVgxJNinMn8ns, BuildID[sha1]=6d58f633a78b2c7caaf742334ea161e2d5bb218e, not stripped
docker/runc:                    ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, Go BuildID=7tQQtsVYZ9o4nquLOZC3/P4UqBZ-awzd7NKHFpPxw/FZaYbsId8bFN-7LyvIeT/NJemHh3hR5Cc3llo_G3J, BuildID[sha1]=5ddf3a437c0c9c66b23f34f8e93cb6e42b409915, not stripped

- How to verify it

$ docker buildx build --build-arg "CROSS=true" --build-arg "DOCKER_CROSSPLATFORMS=linux/amd64" --output "./bundles" --target cross .
$ file bundles/cross/linux/amd64-daemon/*
bundles/cross/linux/amd64-daemon/containerd:                           ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=6548032aca56c6a1771266f5d01891b4bf3e1c89, for GNU/Linux 3.2.0, stripped
bundles/cross/linux/amd64-daemon/containerd-shim:                      ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=7d9d9ae7edb9e405f6bf4650ded0d486f7266656, for GNU/Linux 3.2.0, stripped
bundles/cross/linux/amd64-daemon/containerd-shim-runc-v2:              ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=c55c45a8515beb1e34a04bbac2d90eb22557e2e7, for GNU/Linux 3.2.0, stripped
bundles/cross/linux/amd64-daemon/ctr:                                  ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=703b7854d30ba56b7941efdf1b3c095620f20561, for GNU/Linux 3.2.0, stripped
bundles/cross/linux/amd64-daemon/docker-init:                          ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=22b9ed474ff9b95396473ddadf16a1f5ba13ca7c, for GNU/Linux 3.2.0, stripped
bundles/cross/linux/amd64-daemon/docker-proxy-dev:                     ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=6be5e9b5172a653a0e3cb7b7a6e21d306526bc34, for GNU/Linux 3.2.0, not stripped
bundles/cross/linux/amd64-daemon/dockerd-dev:                          ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=e0b42577a6002e74a6c66cd0e1dc8fa6311d5b36, for GNU/Linux 3.2.0, not stripped
bundles/cross/linux/amd64-daemon/rootlesskit:                          ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=708FSPsRHNRZgF1L6uD2/UJGui4eOMyym2XhCKEAK/bOhGfDmaq7oQQGtogrIQ/JSjfQ3x0-m-8Z2_AFn-h, not stripped
bundles/cross/linux/amd64-daemon/rootlesskit-docker-proxy:             ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=TQ1YUYWqazoG16y8yavv/EunXQT_JraDDwgM8hESr/TFVqZgx_DToPaW2xEo6k/IW1Ni_QzartD4GbXJ4au, not stripped
bundles/cross/linux/amd64-daemon/runc:                                 ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=a6ada9a48b8d654a2128c28be07bc596725fd9c2, for GNU/Linux 3.2.0, not stripped
bundles/cross/linux/amd64-daemon/vpnkit:                               ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, with debug_info, not stripped

- Description for the changelog

fix docker-proxy and ctr not statically linked

@crazy-max crazy-max requested a review from tianon as a code owner April 15, 2022 13:12
@crazy-max
Copy link
Member Author

crazy-max commented Apr 15, 2022

cross for windows/arm64 of docker-proxy fails: https://github.com/moby/moby/runs/6038490251?check_suite_focus=true#step:5:2109

#47 76.28 Building: /build/bundles/cross/windows/arm64-daemon/docker-proxy-dev.exe
#47 76.28 GOOS="windows" GOARCH="arm64" GOARM=""
#47 76.29 +++++ go build -o /build/bundles/cross/windows/arm64-daemon/docker-proxy-dev.exe -tags 'netgo osusergo static_build apparmor seccomp journald' -installsuffix netgo -ldflags '-w 	-X "github.com/docker/docker/dockerversion.Version=dev" 	-X "github.com/docker/docker/dockerversion.GitCommit=HEAD" 	-X "github.com/docker/docker/dockerversion.BuildTime=2022-04-15T13:19:10.000000000+00:00" 	-X "github.com/docker/docker/dockerversion.IAmStatic=true" 	-X "github.com/docker/docker/dockerversion.PlatformName=" 	-X "github.com/docker/docker/dockerversion.ProductName=" 	-X "github.com/docker/docker/dockerversion.DefaultProductLicense="   -extldflags "-static" -linkmode=external' github.com/docker/docker/cmd/docker-proxy
#47 76.66 # github.com/docker/docker/cmd/docker-proxy
#47 76.66 loadinternal: cannot find runtime/cgo
#47 76.66 /usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
#47 76.66 gcc: error: unrecognized command-line option '-mconsole'

Afaik there is no aarch64-w64-mingw32-gcc toolchain available atm.

@crazy-max crazy-max force-pushed the fix-docker-proxy-and-ctr branch 2 times, most recently from b0222ef to e541b86 Compare April 15, 2022 15:13
@thaJeztah
Copy link
Member

I see some binaries are stripped and others not; wondering if we should strip the proxy one 🤔 (not directly related to this change, just wondering)

@crazy-max
Copy link
Member Author

I see some binaries are stripped and others not; wondering if we should strip the proxy one 🤔 (not directly related to this change, just wondering)

Yes indeed and we should also -trimpath for a bit of reproducibility.

@thaJeztah
Copy link
Member

Seeing some failures, which (from a quick glance) look related;

Stderr:   failed to start shim: start failed: : signal: trace/breakpoint trap (core dumped): unknown

@@ -2,9 +2,16 @@

set -e

case "$(go env GOOS)/$(go env GOARCH)" in
windows/arm64) ;;
# TODO remove windows/arm64 when aarch64-w64-mingw32-gcc toolchain is available
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need gcc for docker-proxy.
But windows/arm64 is a new golang target anyway.

@cpuguy83
Copy link
Member

It would help to review this without the changes that mess with the new-lines vs flattened into one line.

@thaJeztah
Copy link
Member

Related ticket in containerd; containerd/containerd#5824

@thaJeztah thaJeztah added this to the 22.04.0 milestone Apr 21, 2022
@crazy-max crazy-max marked this pull request as draft April 26, 2022 14:32
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
@thaJeztah
Copy link
Member

It would help to review this without the changes that mess with the new-lines vs flattened into one line.

I did a quick rebase and reduced the diff by removing the whitespace changes, and some other changes that didn't seem immediately related to the fix; https://github.com/moby/moby/compare/e541b867b97e7d26741e354f61cf7fc90fd4c3cc..d07cc5c85b7942d5c06c182234f91c6f8c45b112

@thaJeztah
Copy link
Member

Looks like things are segfaulting; think that's the containerd binary failing;

=== FAIL: amd64.integration.build TestBuildMultiStageCopy/copy_to_newdir (0.35s)
[2022-05-18T19:17:52.446Z]     build_test.go:182: {"stream":"Step 1/8 : FROM busybox AS base"}
[2022-05-18T19:17:52.446Z]         {"stream":"\n"}
[2022-05-18T19:17:52.446Z]         {"stream":" ---\u003e 1c35c4412082\n"}
[2022-05-18T19:17:52.446Z]         {"stream":"Step 2/8 : RUN mkdir existingdir"}
[2022-05-18T19:17:52.447Z]         {"stream":"\n"}
[2022-05-18T19:17:52.447Z]         {"stream":" ---\u003e Running in 55352d85beb4\n"}
[2022-05-18T19:17:52.447Z]         {"stream":"Removing intermediate container 55352d85beb4\n"}
[2022-05-18T19:17:52.447Z]         {"errorDetail":{"message":"failed to start shim: start failed: : signal: trace/breakpoint trap (core dumped): unknown"},"error":"failed to start shim: start failed: : signal: trace/breakpoint trap (core dumped): unknown"}

Let me open a quick PR with only the docker-proxy commit

1 similar comment
@thaJeztah
Copy link
Member

Looks like things are segfaulting; think that's the containerd binary failing;

=== FAIL: amd64.integration.build TestBuildMultiStageCopy/copy_to_newdir (0.35s)
[2022-05-18T19:17:52.446Z]     build_test.go:182: {"stream":"Step 1/8 : FROM busybox AS base"}
[2022-05-18T19:17:52.446Z]         {"stream":"\n"}
[2022-05-18T19:17:52.446Z]         {"stream":" ---\u003e 1c35c4412082\n"}
[2022-05-18T19:17:52.446Z]         {"stream":"Step 2/8 : RUN mkdir existingdir"}
[2022-05-18T19:17:52.447Z]         {"stream":"\n"}
[2022-05-18T19:17:52.447Z]         {"stream":" ---\u003e Running in 55352d85beb4\n"}
[2022-05-18T19:17:52.447Z]         {"stream":"Removing intermediate container 55352d85beb4\n"}
[2022-05-18T19:17:52.447Z]         {"errorDetail":{"message":"failed to start shim: start failed: : signal: trace/breakpoint trap (core dumped): unknown"},"error":"failed to start shim: start failed: : signal: trace/breakpoint trap (core dumped): unknown"}

Let me open a quick PR with only the docker-proxy commit

@thaJeztah
Copy link
Member

opened #43621 - let's see if that one does go green; the ctr binary is not super-important (only for debugging purposes)

@crazy-max
Copy link
Member Author

crazy-max commented May 20, 2022

closing as carry by #43621 as well as #43529 in a follow-up

@crazy-max crazy-max closed this May 20, 2022
@crazy-max crazy-max deleted the fix-docker-proxy-and-ctr branch May 20, 2022 15:55
@crazy-max crazy-max restored the fix-docker-proxy-and-ctr branch May 20, 2022 15:55
@crazy-max crazy-max deleted the fix-docker-proxy-and-ctr branch May 20, 2022 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants