x-pack/elastic-agent-packaging-arm-arm - mage package failed

[kaiyan-sheng]

Flaky Test

• Test Name: mage pacakge
• Link: https://github.com/elastic/beats/blob/master/x-pack/elastic-agent/Jenkinsfile.yml#L98
• Branch: master
• Artifact Link: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/PR-26121/19/pipeline#step-7434-log-147

Stack Trace

[2021-06-09T18:31:10.772Z] WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

[2021-06-09T18:31:10.773Z] standard_init_linux.go:219: exec user process caused: exec format error

[2021-06-09T18:31:10.773Z] >> golangCrossBuild: Building for linux/amd64

[2021-06-09T18:31:11.293Z] WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

[2021-06-09T18:31:11.293Z] standard_init_linux.go:219: exec user process caused: exec format error

[2021-06-09T18:31:11.814Z] >> golangCrossBuild: Building for linux/arm64

[2021-06-09T18:31:11.814Z] >> Building using: cmd='build/mage-linux-arm64 golangCrossBuild', env=[GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64]

[2021-06-09T18:33:10.047Z] package ran for 21m24.303141201s

[2021-06-09T18:33:10.047Z] Error: running "docker run --env EXEC_UID=1162 --env EXEC_GID=1162 -v /var/lib/jenkins/workspace/PR-26121-19-580aec4b-3b28-49f5-a04e-098a494a058e/pkg/mod:/go/pkg/mod:ro --rm --env GOFLAGS=-mod=readonly --env MAGEFILE_VERBOSE= --env MAGEFILE_TIMEOUT= --env SNAPSHOT=true --env DEV=false -v /var/lib/jenkins/workspace/PR-26121-19-580aec4b-3b28-49f5-a04e-098a494a058e/src/github.com/elastic/beats:/go/src/github.com/elastic/beats -w /go/src/github.com/elastic/beats/x-pack/elastic-agent docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-main-debian7 --build-cmd build/mage-linux-arm64 buildGoDaemon -p linux/amd64" failed with exit code 1

[2021-06-09T18:33:10.047Z] running "docker run --env EXEC_UID=1162 --env EXEC_GID=1162 -v /var/lib/jenkins/workspace/PR-26121-19-580aec4b-3b28-49f5-a04e-098a494a058e/pkg/mod:/go/pkg/mod:ro --rm --env GOFLAGS=-mod=readonly --env MAGEFILE_VERBOSE= --env MAGEFILE_TIMEOUT= --env SNAPSHOT=true --env DEV=false -v /var/lib/jenkins/workspace/PR-26121-19-580aec4b-3b28-49f5-a04e-098a494a058e/src/github.com/elastic/beats:/go/src/github.com/elastic/beats -w /go/src/github.com/elastic/beats/x-pack/elastic-agent docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-main --build-cmd build/mage-linux-arm64 golangCrossBuild -p windows/amd64" failed with exit code 1

[2021-06-09T18:33:10.047Z] running "docker run --env EXEC_UID=1162 --env EXEC_GID=1162 -v /var/lib/jenkins/workspace/PR-26121-19-580aec4b-3b28-49f5-a04e-098a494a058e/pkg/mod:/go/pkg/mod:ro --rm --env GOFLAGS=-mod=readonly --env MAGEFILE_VERBOSE= --env MAGEFILE_TIMEOUT= --env SNAPSHOT=true --env DEV=false -v /var/lib/jenkins/workspace/PR-26121-19-580aec4b-3b28-49f5-a04e-098a494a058e/src/github.com/elastic/beats:/go/src/github.com/elastic/beats -w /go/src/github.com/elastic/beats/x-pack/elastic-agent docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin-debian10 --build-cmd build/mage-linux-arm64 golangCrossBuild -p darwin/amd64" failed with exit code 1

[2021-06-09T18:33:10.047Z] running "docker run --env EXEC_UID=1162 --env EXEC_GID=1162 -v /var/lib/jenkins/workspace/PR-26121-19-580aec4b-3b28-49f5-a04e-098a494a058e/pkg/mod:/go/pkg/mod:ro --rm --env GOFLAGS=-mod=readonly --env MAGEFILE_VERBOSE= --env MAGEFILE_TIMEOUT= --env SNAPSHOT=true --env DEV=false -v /var/lib/jenkins/workspace/PR-26121-19-580aec4b-3b28-49f5-a04e-098a494a058e/src/github.com/elastic/beats:/go/src/github.com/elastic/beats -w /go/src/github.com/elastic/beats/x-pack/elastic-agent docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-main-debian7 --build-cmd build/mage-linux-arm64 golangCrossBuild -p linux/amd64" failed with exit code 1

script returned exit code 1

[urso]

Checking the Jenkins logs, we see that the environment variables (PLATFORM, PACKAGES) are set create an arm docker image for :

[2021-06-09T18:10:55.577Z] + mage dumpVariables

[2021-06-09T18:11:32.294Z] ## Variables

[2021-06-09T18:11:32.294Z] 

[2021-06-09T18:11:32.294Z] GOOS             = linux

[2021-06-09T18:11:32.294Z] GOARCH           = arm64

[2021-06-09T18:11:32.294Z] GOARM            = 

[2021-06-09T18:11:32.294Z] Platform         = linux/arm64

[2021-06-09T18:11:32.294Z] BinaryExt        = 

[2021-06-09T18:11:32.294Z] XPackDir         = ../x-pack

[2021-06-09T18:11:32.294Z] BeatName         = beats

[2021-06-09T18:11:32.294Z] BeatServiceName  = beats

[2021-06-09T18:11:32.294Z] BeatIndexPrefix  = beats

[2021-06-09T18:11:32.294Z] BeatDescription  = 

[2021-06-09T18:11:32.294Z] BeatVendor       = Elastic

[2021-06-09T18:11:32.294Z] BeatLicense      = ASL 2.0

[2021-06-09T18:11:32.294Z] BeatURL          = https://www.elastic.co/beats/beats

[2021-06-09T18:11:32.294Z] BeatUser         = root

[2021-06-09T18:11:32.294Z] VersionQualifier = 

[2021-06-09T18:11:32.294Z] PLATFORMS        = linux/arm64

[2021-06-09T18:11:32.294Z] PACKAGES         = docker

[2021-06-09T18:11:32.294Z] 

[2021-06-09T18:11:32.294Z] ## Functions

[2021-06-09T18:11:32.294Z] 

[2021-06-09T18:11:32.294Z] beat_doc_branch              = master

[2021-06-09T18:11:32.294Z] beat_version                 = 8.0.0

[2021-06-09T18:11:32.294Z] commit                       = 91459f782054f5825c93d5ce85c7ab6f1e4d6a11

[2021-06-09T18:11:32.294Z] date                         = 2021-06-09T18:11:28Z

[2021-06-09T18:11:32.294Z] elastic_beats_dir            = /var/lib/jenkins/workspace/PR-26121-19-580aec4b-3b28-49f5-a04e-098a494a058e/src/github.com/elastic/beats

[2021-06-09T18:11:32.294Z] go_version                   = 1.16.4

[2021-06-09T18:11:32.294Z] repo.RootImportPath          = github.com/elastic/beats/v7

[2021-06-09T18:11:32.294Z] repo.CanonicalRootImportPath = github.com/elastic/beats

[2021-06-09T18:11:32.294Z] repo.RootDir                 = /var/lib/jenkins/workspace/PR-26121-19-580aec4b-3b28-49f5-a04e-098a494a058e/src/github.com/elastic/beats

[2021-06-09T18:11:32.294Z] repo.ImportPath              = github.com/elastic/beats/v7

[2021-06-09T18:11:32.294Z] repo.SubDir                  = .

[2021-06-09T18:11:32.294Z]

The mage package in the agent directory either seems to ignore the environment varialbe, or does not pass it along properly:

[2021-06-09T18:11:35.246Z] + mage package

[2021-06-09T18:11:44.300Z] Package 'filebeat-8.0.0-SNAPSHOT-darwin-x86_64.tar.gz' does not exist on path: /var/lib/jenkins/workspace/PR-26121-19-580aec4b-3b28-49f5-a04e-098a494a058e/src/github.com/elastic/beats/x-pack/filebeat/build/distributions/filebeat-8.0.0-SNAPSHOT-darwin-x86_64.tar.gz

Not sure if this is a typo, or if we really try to build an x86 image here.

The command to cross compile filebeat for ARM eventually succeeds:

[2021-06-09T18:11:59.907Z] >> Building filebeat.yml for linux/amd64

[2021-06-09T18:11:59.907Z] >> Building filebeat.reference.yml for linux/amd64

[2021-06-09T18:11:59.907Z] >> Building filebeat.docker.yml for linux/amd64

[2021-06-09T18:12:01.592Z] exec: go list -m

[2021-06-09T18:12:38.459Z] >> buildGoDaemon: Building for linux/arm64

[2021-06-09T18:12:38.459Z] >> Building using: cmd='build/mage-linux-arm64 buildGoDaemon', env=[GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64]

[2021-06-09T18:12:42.633Z] /tmp/cc5Oos4I.o: In function `main':

[2021-06-09T18:12:42.633Z] god.c:(.text+0x2ac): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

[2021-06-09T18:12:42.634Z] god.c:(.text+0x230): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

[2021-06-09T18:12:42.634Z] >> golangCrossBuild: Building for linux/arm64

[2021-06-09T18:12:42.634Z] >> Building using: cmd='build/mage-linux-arm64 golangCrossBuild', env=[GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64]

[2021-06-09T18:16:58.392Z] >> package: Building filebeat type=tar.gz for platform=linux/arm64

[2021-06-09T18:16:58.392Z] >> Testing package contents

[2021-06-09T18:17:04.686Z] package ran for 5m9.575802458s

For some reason mage tries to cross compile agent for windows and darwin as well:

[2021-06-09T18:30:35.828Z] >> golangCrossBuild: Building for windows/amd64

[2021-06-09T18:30:36.343Z] WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

[2021-06-09T18:30:36.855Z] standard_init_linux.go:219: exec user process caused: exec format error

[2021-06-09T18:30:36.855Z] >> buildGoDaemon: Building for linux/amd64

[2021-06-09T18:30:37.363Z] WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

[2021-06-09T18:30:37.871Z] standard_init_linux.go:219: exec user process caused: exec format error

[2021-06-09T18:30:37.871Z] >> golangCrossBuild: Building for darwin/amd64

[2021-06-09T18:30:37.871Z] Unable to find image 'docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin-debian10' locally

[2021-06-09T18:30:39.765Z] 1.16.4-darwin-debian10: Pulling from beats-dev/golang-crossbuild

linux/amd64, windows/amd64, darwin/amd64 cross compilations are the ones that fail. But it looks like these should not have been run at all. The problem appears to be in the Magefile not using (or passing along) environment variables correctly.

[michalpristas]

linking fix we have for

The mage package in the agent directory either seems to ignore the environment varialbe, or does not pass it along properly

#26313

[kuisathaverat]

workaround to not build x86 binaries on ARM hosts #26387, this also generates a Docker image only with the Elastic Agent binary.

[kuisathaverat]

finally, I've managed to build the arm64 dependencies at #26415

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[kuisathaverat]

merged, we will backport the fix to 7.x and 7.13