Gödel build of linux/amd64 check_cpu doesn't seem to work in container

[asachs01]

Greetings NCR team!

I've been using the collection here across Windows systems, Linux systems, and Docker containers. I've noticed that there seems to be a way that the executables are built w/ Gödel vs building the executables natively, or with goreleaser.

The scenario that has surfaced this difference is that I'm running the check_cpu check across Linux, Windows and Docker. I have Linux entities that report back as linux/amd64 and Docker containers that report back as linux/amd64.

When the collection is downloaded on the Linux systems, the check runs without issue. However, in Docker containers, I receive an error:

/var/cache/sensu/sensu-agent/dca8a9389960ab9a5d419544a4f9c6bda0f133200e673e9cdb8e81f32c721b80ed2fc20b224f3b42630d12e50b056e6c2edeeacefd39c24ee2b6501d305c95ec/bin # ls
check_cpu          check_http         check_process      check_uptime
check_file_exists  check_memory       check_service      check_user_group
/var/cache/sensu/sensu-agent/dca8a9389960ab9a5d419544a4f9c6bda0f133200e673e9cdb8e81f32c721b80ed2fc20b224f3b42630d12e50b056e6c2edeeacefd39c24ee2b6501d305c95ec/bin # ./check_cpu
bin/sh: ./check_cpu: not found

Compare that with the Linux systems:

[root@sensu00 ~]# cd /var/cache/sensu/sensu-agent/dca8a9389960ab9a5d419544a4f9c6bda0f133200e673e9cdb8e81f32c721b80ed2fc20b224f3b42630d12e50b056e6c2edeeacefd39c24ee2b6501d305c95ec/bin/
[root@sensu00 bin]# ls
check_cpu          check_http    check_process  check_uptime
check_file_exists  check_memory  check_service  check_user_group
[root@sensu00 bin]# ./check_cpu
CheckAVGCPULoad OK - value = 0.250627 | pct_processor_time=0.250627

I found that if I build the executable natively and copy it into the container, it works:

 GOOS=linux GOARCH=amd64 go build github.com/ncr-devops-platform/nagiosfoundation/cmd/check_cpu
...
 docker cp check_cpu  sensu-go-docker_sensu-agent_1:/home   
# cd /home/
/home # ls
check_cpu
/home # ./check_cpu
CheckAVGCPULoad OK - value = 1.546392 | pct_processor_time=1.546392

Is there something that needs to be changed with Gödel's builds for them to work on containers?

[quaggoth]

Something else is amiss. The binary as fetched and extracted from the latest release on GitHub executes successfully in my Docker container:

$ docker container run --rm -it debian:10.2 /bin/bash
root@302184309fa0:/# apt-get update && apt-get install -y curl
Get:1...
...
done.
root@302184309fa0:/# curl -Ls https://github.com/ncr-devops-platform/nagiosfoundation/releases/download/0.2.2/nagiosfoundation-linux-amd64-0.2.2.tgz | tar -zxvf - bin/check_cpu
bin/check_cpu
root@302184309fa0:/# bin/check_cpu
CheckAVGCPULoad OK - value = 3.000000 | pct_processor_time=3.000000

[asachs01]

Hi @quaggoth, apologies--I neglected to mention that this is inside of an Alpine container. This still seems to be the case that the check doesn't work on alpine, though other checks that I've used that are for a linux/amd64 OS (which Alpine reports to be) work.

Bare Alpine container w/ nagiosfoundation asset

docker container run --rm -it alpine:latest /bin/sh             
/ # apk add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ca-certificates (20190108-r0)
(2/4) Installing nghttp2-libs (1.39.2-r0)
(3/4) Installing libcurl (7.66.0-r0)
(4/4) Installing curl (7.66.0-r0)
Executing busybox-1.30.1-r2.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 7 MiB in 18 packages
/ # curl -Ls https://github.com/ncr-devops-platform/nagiosfoundation/releases/download/0.2.2/nagiosfou
ndation-linux-amd64-0.2.2.tgz | tar -zxvf - bin/check_cpu
bin/check_cpu
/ # bin/check_cpu
/bin/sh: bin/check_cpu: not found

Bare Alpine container w/ asachs01/sensu-go-cpu-check asset

/ # curl -Ls https://github.com/asachs01/sensu-go-cpu-check/releases/download/0.0.4/sensu-go-cpu-check_0.0.4_linux_amd64.tar.gz | tar -xvzf - bin/sensu-go-cpu-check
/ # bin/sensu-go-cpu-check
CheckCPU OK - value = 4.90 | system_cpu=4.90

And looking at the two executables:

~/bin # ls
check_cpu           sensu-go-cpu-check
~/bin # file check_cpu
check_cpu: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=Pu00gzT2065S5Dy-zGy2/PdiIixQaDF-d5K6noDi8/GVfrQ4yBBPUAmTEfRPEV/LLEXmQiAuf1Cj5yupBBi, not stripped
~/bin # file sensu-go-cpu-check
sensu-go-cpu-check: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=FqwmBQw8H4gbO32qY26t/CzNPCRUB9lUTKimGZ4WK/exS17Huy1gQE-Q-RgqZE/W2Uo8KvQoRJRXR9cEgPj, stripped

Which looks like the check_cpu asset is dependent on the GNU libc library being present . That said, I think this is actually a result of using an Alpine image. I'm working with our eng folks to see if it's possible for us to add the libc6-compat package as part of the Docker container image.

[asachs01]

@quaggoth also in chatting with our engineers, the other possibility would be to ensure that CGO_ENABLED=0 is set as part of the build process.

[asachs01]

Discussed with our eng folk, it seems CGO_ENABLED=0 is the preferred way to handle this, if that's at all possible for y'all.

[quaggoth]

Ah. Alpine. Good times for you. Alpine is not based on glibc like Debian and many other Linux distributions. It's based on musl. For executables on Alpine, the checks would need to be compiled against musl. Another way to temporarily fool your container would be to create a glibc soft link to musl with

mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2

There may be some other ways around this but I certainly am not aware of them off the top of my head. It would probably involve statically linking glibc into all the binaries at build time.

Using the workaround above, here's a full example run using Alpine...

$ docker container run --rm -it alpine:3.10.3 /bin/sh
/ # apk add curl
fetch...
...
OK: 7 MiB in 18 packages
/ # curl -Ls https://github.com/ncr-devops-platform/nagiosfoundation/releases/download/0.2.2/nagiosfoundation-linux-amd64-0.2.2.tgz | tar -zxvf - bin/check_cpu
bin/check_cpu
/ # bin/check_cpu
/bin/sh: bin/check_cpu: not found
/ # mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2
/ # bin/check_cpu
CheckAVGCPULoad OK - value = 4.000000 | pct_processor_time=4.000000

[quaggoth]

Yes, I would take a pull request with CGO_ENABLED=0. It would be nice if you could make the changes, build (using godel) and then test the results on your various machines to validate the changes accomplish what you are wanting and to validate the change doesn't break on any of the various platforms.

Based on the documentation for it, I doubt it will cause any issues:
The cgo tool is enabled by default for native builds on systems where it is expected to work. It is disabled by default when cross-compiling.

[asachs01]

Yup! I actually got it working by adding that compatibility package. I'll work on testing w/ cgo disabled and get a PR in.