7.0.1 fails to build

[sjthespian]

I'm having trouble getting the 7.0.1 beats to compile using your image. It looks like it is installing mage, but then fails to find it.

> docker run -v $(pwd):/build -e BEATS_VERSION=7.0.1 -e GOARM=7 andig/beats4pi
Target version: 7.0.1
Target branch: 7.0
github.com/elastic/beats (download)
package github.com/elastic/beats: build constraints exclude all Go files in /go/src/github.com/elastic/beats
Switched to a new branch '7.0'
Branch 7.0 set up to track remote branch 7.0 from origin.
Installing mage v1.8.0 from vendor dir.
bash: mage: command not found
../dev-tools/make/mage.mk:9: recipe for target 'mage' failed
make: [mage] Error 127 (ignored)
cp: cannot stat 'filebeat': No such file or directory
2019-05-07 21:52:13 URL:https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.0.1-linux-x86.tar.gz [12899116/12899116] -> "filebeat-7.0.1-linux-x86.tar.gz" [1]
cp: cannot stat 'filebeat': No such file or directory
Installing mage v1.8.0 from vendor dir.
bash: mage: command not found
make: [mage] Error 127 (ignored)
../dev-tools/make/mage.mk:9: recipe for target 'mage' failed
cp: cannot stat 'metricbeat': No such file or directory
2019-05-07 21:52:18 URL:https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.0.1-linux-x86.tar.gz [25743741/25743741] -> "metricbeat-7.0.1-linux-x86.tar.gz" [1]
cp: cannot stat 'metricbeat': No such file or directory

I'll poke around a bit and see if this is just a path issue.

[sjthespian]

I think I know what the issue is, but not a good way to solve it. Since mage is built by the installer, it is being built for the arm architecture. However, since the container is running in x86_64, it can't run in the container.

root@806ceb5f8230:/go/src/github.com/elastic/beats# file $GOPATH/bin/linux_arm/mage
/go/bin/linux_arm/mage: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped
root@806ceb5f8230:/go/src/github.com/elastic/beats# uname -a
Linux 806ceb5f8230 4.4.0-145-generic #171-Ubuntu SMP Tue Mar 26 12:43:40 UTC 2019 x86_64 GNU/Linux
root@806ceb5f8230:/go/src/github.com/elastic/beats# $GOPATH/bin/linux_arm/mage
bash: /go/bin/linux_arm/mage: cannot execute binary file: Exec format error

[andig]

It seems that elastic has modified the build process, specifically to support multi-platform build. Unfortunately they have still not added ARM binaries. In order to get this to work you have various options:

1. use OSX as it contains qemu by default
2. modify the docker builder to include qemu (you're on your own with that)
3. hack the elastic build process to inject the host architecture when building the dev tools
4. (I think thats the preferred one) research how the updated elastic build process works and use that for generating the target build directly without use of this repo

I'm afraid you're on your own from here on...

[andig]

Adding to 3) after taking a quick look it might be possible to use make crosscompile directly from the individual beat's folder. Check the libbeat/scripts/Makefile for details.

Update

I've verified this is working:

~/htdocs/beats4pi/beats/filebeat 7.0* 2m 53s
❯ GOX_OS=linux make crosscompile
GOX_OSARCH:  !darwin/arm !darwin/arm64 !darwin/386
GOX_OS:  linux
GOX_ARCH:
GOX_FLAGS:  -arch=amd64 386 arm ppc64 ppc64le
go get github.com/mitchellh/gox
mkdir -p /Users/andig/htdocs/beats4pi/beats/filebeat/build/bin
gox -output="/Users/andig/htdocs/beats4pi/beats/filebeat/build/bin/{{.Dir}}-{{.OS}}-{{.Arch}}" -os="linux" -osarch="!darwin/arm !darwin/arm64 !darwin/386" -arch="amd64 386 arm ppc64 ppc64le"
Number of parallel builds: 3

-->       linux/arm: _/Users/andig/htdocs/beats4pi/beats/filebeat
-->   linux/ppc64le: _/Users/andig/htdocs/beats4pi/beats/filebeat
-->     linux/amd64: _/Users/andig/htdocs/beats4pi/beats/filebeat
-->     linux/ppc64: _/Users/andig/htdocs/beats4pi/beats/filebeat
-->       linux/386: _/Users/andig/htdocs/beats4pi/beats/filebeat

~/htdocs/beats4pi/beats/filebeat 7.0*
❯ ls build/bin
filebeat-linux-386     filebeat-linux-amd64   filebeat-linux-arm     filebeat-linux-ppc64   filebeat-linux-ppc64le filebeat-openbsd-386

So, for 7.x onwards the built-in compile process can be used. I've not checked if its even possible to build an entire ARM package of if there's still need to monkey-patch the release package.

Anyway, I think this has been answered?

[denizzzzp]

FIX
build ALL version ALL platform
https://github.com/denizzzzp/beats4piext

[andig]

Would be happy to accept a pr. As-is the fix is not mergeable.