xgo fails to build dgraph/badger on arm64

[darkn3rd]

Description

When compiling with either go-1.15.8 or go-1.16.0 and target of linux/arm64, xgo fails for an unknown reason.

When the target is linux/amd64, I get success. Also when I use GOARCH=arm64 go build, I also get success.

Reproducing

When compiling with xgo, I used a command something like this:

export GOVERSION=1.15.8
export GOARCH=arm64
cd $GOPATH/src/github.com/dgraph-io
git clone https://github.com/dgraph-io/badger.git
cd badger
git checkout 6c35ad6c28e0

xgo -x -go="go-$GOVERSION" \
  --targets=linux/$GOARCH --tags=jemalloc \
  -deps=https://github.com/jemalloc/jemalloc/releases/download/5.2.1/jemalloc-5.2.1.tar.bz2  \
  --depsargs='--with-jemalloc-prefix=je_ --with-malloc-conf=background_thread:true,metadata_thp:auto --enable-prof' .

Expected Results

Both linux/arm64 and linux/amd64 targets would successfully build.

Actual Results

On linux/arm64, this fails without much indication as to why it failed, other than the cp command. Below is the last several lines where it failed, and full gist of the build output.

cd /go/pkg/mod/github.com/!data!dog/zstd@v1.4.1
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x006.o -c debug.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x007.o -c divsufsort.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x008.o -c entropy_common.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x009.o -c error_private.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x010.o -c fastcover.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x011.o -c fse_compress.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x012.o -c fse_decompress.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x013.o -c hist.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x014.o -c huf_compress.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x015.o -c huf_decompress.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x016.o -c pool.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x017.o -c threading.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x018.o -c xxhash.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x019.o -c zbuff_common.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x020.o -c zbuff_compress.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x021.o -c zbuff_decompress.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x022.o -c zdict.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x023.o -c zstd_common.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x024.o -c zstd_compress.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x025.o -c zstd_ddict.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x026.o -c zstd_decompress.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x027.o -c zstd_decompress_block.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x028.o -c zstd_double_fast.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x029.o -c zstd_fast.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x030.o -c zstd_lazy.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x031.o -c zstd_ldm.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x032.o -c zstd_opt.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x033.o -c zstd_v01.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x034.o -c zstd_v02.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x035.o -c zstd_v03.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x036.o -c zstd_v04.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x037.o -c zstd_v05.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x038.o -c zstd_v06.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x039.o -c zstd_v07.c
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I $WORK/b115/ -g -O2 -o $WORK/b115/_x040.o -c zstdmt_compress.c
cd $WORK/b115
TERM='dumb' aarch64-linux-gnu-gcc-6 -I /go/pkg/mod/github.com/!data!dog/zstd@v1.4.1 -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_cgo_main.o -c _cgo_main.c
cd /go/pkg/mod/github.com/!data!dog/zstd@v1.4.1
TERM='dumb' aarch64-linux-gnu-gcc-6 -I . -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b115=/tmp/go-build -gno-record-gcc-switches -o $WORK/b115/_cgo_.o $WORK/b115/_cgo_main.o $WORK/b115/_x001.o $WORK/b115/_x002.o $WORK/b115/_x003.o $WORK/b115/_x004.o $WORK/b115/_x005.o $WORK/b115/_x006.o $WORK/b115/_x007.o $WORK/b115/_x008.o $WORK/b115/_x009.o $WORK/b115/_x010.o $WORK/b115/_x011.o $WORK/b115/_x012.o $WORK/b115/_x013.o $WORK/b115/_x014.o $WORK/b115/_x015.o $WORK/b115/_x016.o $WORK/b115/_x017.o $WORK/b115/_x018.o $WORK/b115/_x019.o $WORK/b115/_x020.o $WORK/b115/_x021.o $WORK/b115/_x022.o $WORK/b115/_x023.o $WORK/b115/_x024.o $WORK/b115/_x025.o $WORK/b115/_x026.o $WORK/b115/_x027.o $WORK/b115/_x028.o $WORK/b115/_x029.o $WORK/b115/_x030.o $WORK/b115/_x031.o $WORK/b115/_x032.o $WORK/b115/_x033.o $WORK/b115/_x034.o $WORK/b115/_x035.o $WORK/b115/_x036.o $WORK/b115/_x037.o $WORK/b115/_x038.o $WORK/b115/_x039.o $WORK/b115/_x040.o -g -O2
TERM='dumb' /usr/local/go/pkg/tool/linux_amd64/cgo -dynpackage zstd -dynimport $WORK/b115/_cgo_.o -dynout $WORK/b115/_cgo_import.go
cat >$WORK/b115/importcfg << 'EOF' # internal
# import config
packagefile bytes=/usr/local/go/pkg/linux_arm64/bytes.a
packagefile errors=/usr/local/go/pkg/linux_arm64/errors.a
packagefile fmt=/usr/local/go/pkg/linux_arm64/fmt.a
packagefile io=/usr/local/go/pkg/linux_arm64/io.a
packagefile io/ioutil=/usr/local/go/pkg/linux_arm64/io/ioutil.a
packagefile runtime=/usr/local/go/pkg/linux_arm64/runtime.a
packagefile runtime/cgo=/usr/local/go/pkg/linux_arm64/runtime/cgo.a
packagefile syscall=/usr/local/go/pkg/linux_arm64/syscall.a
EOF
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/b115/_pkg_.a -trimpath "$WORK/b115=>" -p github.com/DataDog/zstd -buildid JI4XFzGtRaO7UHQrFSf5/JI4XFzGtRaO7UHQrFSf5 -goversion go1.15.8 -D "" -importcfg $WORK/b115/importcfg -pack -c=4 $WORK/b115/_cgo_gotypes.go $WORK/b115/errors.cgo1.go $WORK/b115/zstd.cgo1.go $WORK/b115/zstd_stream.cgo1.go $WORK/b115/_cgo_import.go
/usr/local/go/pkg/tool/linux_amd64/pack r $WORK/b115/_pkg_.a $WORK/b115/_x001.o $WORK/b115/_x002.o $WORK/b115/_x003.o $WORK/b115/_x004.o $WORK/b115/_x005.o $WORK/b115/_x006.o $WORK/b115/_x007.o $WORK/b115/_x008.o $WORK/b115/_x009.o $WORK/b115/_x010.o $WORK/b115/_x011.o $WORK/b115/_x012.o $WORK/b115/_x013.o $WORK/b115/_x014.o $WORK/b115/_x015.o $WORK/b115/_x016.o $WORK/b115/_x017.o $WORK/b115/_x018.o $WORK/b115/_x019.o $WORK/b115/_x020.o $WORK/b115/_x021.o $WORK/b115/_x022.o $WORK/b115/_x023.o $WORK/b115/_x024.o $WORK/b115/_x025.o $WORK/b115/_x026.o $WORK/b115/_x027.o $WORK/b115/_x028.o $WORK/b115/_x029.o $WORK/b115/_x030.o $WORK/b115/_x031.o $WORK/b115/_x032.o $WORK/b115/_x033.o $WORK/b115/_x034.o $WORK/b115/_x035.o $WORK/b115/_x036.o $WORK/b115/_x037.o $WORK/b115/_x038.o $WORK/b115/_x039.o $WORK/b115/_x040.o # internal
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b115/_pkg_.a # internal
cp $WORK/b115/_pkg_.a /root/.cache/go-build/47/471dea8354ecb331e93244e8f87273d4804778c081e6f8431bfcebf56f1d726c-d # internal
2021/03/03 11:57:46 Failed to cross compile package: exit status 2.

• https://gist.github.com/darkn3rd/a2a77337e1d5101bafd0ac67824589bb

[techknowlogick]

Thank you for this detailed bug report. Sorry for taking some time to get back to you on this. I've been able to replicate this myself thanks to your instructions.

Would dgraph be open to sponsoring my work on this project? I've received private sponsorship from others to ensure ongoing compatibility of xgo with their projects, and with sponsorship from you I could setup a test harness for dgraph, catch errors like this and prioritize fixes for them as they show up. Without sponsorship I am only able to provide support on an as-available basis, and focus on my downstream projects first. If you'd like to discuss this more, my email is in my profile.

[darkn3rd]

Dgraph used xgo to build mac and windows builds, and I wanted to see if it it would work for arm64 using the same build process. Recently, macOS and Windows official support has dropped. Personally, I would like to have it work, for pure community OSS folks, especially with macOS going arm64.

[techknowlogick]

@darkn3rd after some investigation, the errors you are facing are due to https://github.com/dgraph-io/ristretto/blob/3836124cdc5a27eba5bf0a545e04906671ad3c41/z/calloc_jemalloc.go#L10 which specifies specific LDFLAGS which conflict with xgo's paths. I've been able to get it to build by modifying the LDFLAGS in the above link.

[darkn3rd]

@techknowlogick What were the changes you made?