"go install" command fails while running inside s390x docker container on x86_64 host using qemu

[nirmannarang]

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind bug

Description:

Steps to reproduce the issue:

1. Register x86_64 host with the latest qemu-user-static.
   docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

2. Build the following Docker Image using following Dockerfile.s390x using command docker build -t test/crossbuild:latest-s390x -f Dockerfile.s390x .

Dockerfile.s390x

FROM alpine:3.11 as qemu

ARG QEMU_VERSION=5.0.0-2
ARG QEMU_ARCHS="s390x"

RUN apk --update add curl

#Enable non-native runs on amd64 architecture hosts
RUN for i in ${QEMU_ARCHS}; do curl -L https://github.com/multiarch/qemu-user-static/releases/download/v${QEMU_VERSION}/qemu-${i}-static.tar.gz | tar zxvf - -C /usr/bin; done
RUN chmod +x /usr/bin/qemu-*

FROM s390x/golang:1.14.2-alpine3.11
MAINTAINER LoZ Open Source Ecosystem (https://www.ibm.com/developerworks/community/groups/community/lozopensource)

ARG MANIFEST_TOOL_VERSION=v1.0.2

#Enable non-native builds of this image on an amd64 hosts.
#This must be the first RUN command in this file!
COPY --from=qemu /usr/bin/qemu-*-static /usr/bin/

#Install su-exec for use in the entrypoint.sh (so processes run as the right user)
#Install bash for the entry script (and because it's generally useful)
#Install curl to download glide
#Install git for fetching Go dependencies
#Install ssh for fetching Go dependencies
#Install mercurial for fetching go dependencies
#Install wget since it's useful for fetching
#Install make for building things
#Install util-linux for column command (used for output formatting).
#Install grep and sed for use in some Makefiles (e.g. pulling versions out of glide.yaml)
#Install shadow for useradd (it allows to use big UID)
RUN apk update && apk add --no-cache su-exec curl bash git openssh mercurial make wget util-linux tini file grep sed shadow
RUN apk upgrade --no-cache

#Disable ssh host key checking
RUN echo 'Host *' >> /etc/ssh/ssh_config \
  && echo '    StrictHostKeyChecking no' >> /etc/ssh/ssh_config

#Disable cgo so that binaries we build will be fully static.
ENV CGO_ENABLED=0

#Recompile the standard library with cgo disabled.  This prevents the standard library from being
#marked stale, causing full rebuilds every time.
RUN go install -v std

#Install glide
RUN go get github.com/Masterminds/glide
ENV GLIDE_HOME /home/user/.glide

#Install dep
RUN go get github.com/golang/dep/cmd/dep

#Install ginkgo CLI tool for running tests
RUN go get github.com/onsi/ginkgo/ginkgo

#Install linting tools.
RUN wget -O - -q https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s v1.20.0
RUN golangci-lint --version

#Install license checking tool.
RUN go get github.com/pmezard/licenses

#Install tool to merge coverage reports.
RUN go get github.com/wadey/gocovmerge

#Install CLI tool for working with yaml files
RUN go get github.com/mikefarah/yaml

#Delete all the Go sources that were downloaded, we only rely on the binaries
RUN rm -rf /go/src/*

#Install vgo (should be removed once we take Go 1.11)
RUN go get -u golang.org/x/vgo

#Ensure that everything under the GOPATH is writable by everyone
RUN chmod -R 777 $GOPATH

RUN curl -sSL https://github.com/estesp/manifest-tool/releases/download/${MANIFEST_TOOL_VERSION}/manifest-tool-linux-s390x > manifest-tool && \
    chmod +x manifest-tool && \
    mv manifest-tool /usr/bin/

COPY entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/entrypoint.sh"]

The build just hangs at RUN go install -v std

3. Also, while running the same command inside s390x container on x86_64 host,, error Illegal instruction (core dumped) is thrown.

Register x86_64 host with the latest qemu-user-static.
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

docker run -it -v /home/test/qemu-s390x-static:/usr/bin/qemu-s390x-static s390x/golang:1.14.2-alpine3.11

Inside s390x container:

apk update && apk add --no-cache su-exec curl bash git openssh mercurial make wget util-linux tini file grep sed shadow
apk upgrade --no-cache

#Disable ssh host key checking
echo 'Host *' >> /etc/ssh/ssh_config
echo '    StrictHostKeyChecking no' >> /etc/ssh/ssh_config

#Disable cgo so that binaries we build will be fully static.
export CGO_ENABLED=0

#Recompile the standard library with cgo disabled.  This prevents the standard library from being
#marked stale, causing full rebuilds every time.
go install -v std

Describe the results you received:
Step 2 just halts while running step RUN go install -v std while building Dockerfile.s390x.
Step 3 gives the following error:
Illegal instruction (core dumped)

Describe the results you expected:
The command should have been successful without any output.

Environment:
x86_64 Ub18.04 4.15.0-101-generic Ubuntu SMP x86_64 GNU/Linux

• QEMU version: 5.0.0-2
• Container application: Docker

Output of docker version, podman version or singularity version

Client: Docker Engine - Community
 Version:           19.03.11
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        42e35e61f3
 Built:             Mon Jun  1 09:12:22 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.11
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       42e35e61f3
  Built:            Mon Jun  1 09:10:54 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Additional information optionally:
When I build the same Dockerfile.s390x on an s390x machine, it is built successfully.

[umarcor]

Hi @nirmannarang! golang seems te be tricky on qemu-user, and several bugs have been fixed in the last months:

• https://bugs.launchpad.net/qemu/+bug/1763536
• https://bugs.launchpad.net/qemu/+bug/1868055

It might be interesting trying this without Docker (using qemu-user on the host) and opening a bug report in QEMU.

[nirmannarang]

@umarcor Docker requirement is necessary actually. This is part of CI which builds this s390x go-build image on x86 and is then published on docker hub. And this fails only for s390x, other architectures like ppc64le, arm64, etc. are running successfully through this process.
Let me still give it a try without Docker(using qemu-user on the host).
Thanks.

[nirmannarang]

@umarcor GOARCH=s390x go install -v std command is successful if I run it directly on x86 host. The same command gives error if ran inside s390x docker container on x86 host.

[umarcor]

@umarcor Docker requirement is necessary actually. This is part of CI which builds this s390x go-build image on x86 and is then published on docker hub. And this fails only for s390x, other architectures like ppc64le, arm64, etc. are running successfully through this process.

Yes, I understand that. However, I think that this might be an issue with "qemu-user allowing s390x targets on x86_64 hosts". Because you seem to have access to an s390x host (you wrote "When I build the same Dockerfile.s390x on an s390x machine, it is built successfully"), I'm suggesting you to take Docker out of the equation FOR TESTING PURPOSES ONLY. Once you find/fix the issue, you can go back to using the container, as you expect.

GOARCH=s390x go install -v std command is successful if I run it directly on x86 host.

This is cross-compilation; qemu-user is not being used. I suggest to download a go installation for s390x into your host. If you try to execute it straightaway, it should fail. If you run qemu-user go install -v std, it should theoretically produce a packages/libraries for s390x. But, in this case, I'd expect it to fail with the same error that you are seeing in the container. This would mean that the issue is, indeed, with golang and qemu-user, but unrelated to docker.

[nirmannarang]

I installed qemu on x86 Ubuntu host. Extracted s390x go 1.14.2 binaries on the same. Ran the following commands:

root:~ wget -q https://storage.googleapis.com/golang/go1.14.2.linux-s390x.tar.gz
root@:~# chmod ugo+r go1.14.2.linux-s390x.tar.gz
root@:~# rm -rf /usr/local/go /usr/bin/go
root@:~#  tar -C /usr/local -xzf go1.14.2.linux-s390x.tar.gz
root@:~# ln -sf /usr/local/go/bin/go /usr/bin/
root@:~# ln -sf /usr/local/go/bin/gofmt /usr/bin/
root@:~# ln -sf /usr/bin/gcc /usr/bin/s390x-linux-gnu-gcc
root@:~# go version
/lib/ld64.so.1: No such file or directory
root@:~# qemu-s390x /usr/bin/go version
/lib/ld64.so.1: No such file or directory

[nirmannarang]

Another note, on x86 host, I ran the following commands:

docker run --rm --privileged multiarch/qemu-user-static:4.2.0-7 --reset -p yes

Ran an s390x golang 1.14.2 alpine container on x86 host

root@:~# docker run -it -v /usr/bin/qemu-s390x-static:/usr/bin/qemu-s390x-static s390x/golang:1.14.2-alpine3.11

Inside s390x container, ran following go commands:

/go # go version
go version go1.14.2 linux/s390x
/go # go install -v -x std
WORK=/tmp/go-build090332422
/go #

Strange that this is successful but docker build fails using same command inside Dockerfile.s390x.

[umarcor]

That seems to confirm that this is an issue upstream with allowing s390x targets on x86_64 hosts. I suggest to open a bug report as commented in #114 (comment).

[nirmannarang]

The issue has been fixed with the latest release 6.1.0-8. Hence closing the issue.