cmd/compile: CGO arm builds fail on Go 1.10beta1

[karalabe]

What version of Go are you using (go version)?

go1.10beta1

What operating system and processor architecture are you using (go env)?

Travis CI, Linux

What did you do?

Tried to cross compile go-ethereum to ARM5/6/7/64, which builds fine on all previous versions of Go.

What did you expect to see?

Successful build.

What did you see instead?

arm-linux-gnueabi-gcc: error: unrecognized command line option '-m64'

Full build logs at https://travis-ci.org/ethereum/go-ethereum/jobs/323227026

[bradfitz]

You assume we know how to build (or how you built) go-ethereum. How can we reproduce this to debug? I'm sure it's in that travis log somewhere, but a concise repro in the bug would be nice.

/cc @ianlancetaylor

[karalabe]

Fair enough, I'll try to cook up a repro for you.

[bradfitz]

Maybe your GOARM=5 CC=arm-linux-gnueabi-gcc go run build/ci.go install -arch arm should be CC_FOR_TARGET instead.

[karalabe]

Hmm, the issue seems to come from go run build/ci.go (not from the cross compilation invoked within), which for some reason wants to build runtime/cgo, so my CC env var makes that fail:

$ CC=aarch64-linux-gnu-gcc go run -x -v build/ci.go install -arch arm64
WORK=/tmp/go-build716618928
runtime/cgo
mkdir -p $WORK/b034/
cd /opt/google/go/src/runtime/cgo
CGO_LDFLAGS='"-g" "-O2" "-lpthread"' /opt/google/go/pkg/tool/linux_amd64/cgo -objdir $WORK/b034/ -importpath runtime/cgo -import_runtime_cgo=false -import_syscall=false -- -I $WORK/b034/ -g -O2 -Wall -Werror ./cgo.go
# runtime/cgo
aarch64-linux-gnu-gcc: error: unrecognized command line option '-m64'

Now, I could understand why this would fail if ci.go would depend on cgo, but it does not, rather it (and all its dependency tree) is pure Go, so go run shouldn't try to build runtime/cgo, yet it does.

---

Note, I can fix my issue by not setting CC, rather perhaps passing the compiler as a flag to ci.go and have that set it as an env var internally before invoking the cross compilation. However, it would be nice to know why go run wants to touch CGO, without a dependency for it.

Emphasis that this only happens with Go 1.10. None of the previous versions of Go touch runtime/cgo when building my build/ci.go file.

[bradfitz]

You can inspect dependencies with go list -json to see why there's a path to runtime/cgo.

[bradfitz]

(If ci.go uses os/user or net, it'll probably depend on cgo by default.)

[karalabe]

Hmm, indeed, runtime/cgo is indeed listed as some dependency:

$ go list -json ./build/ci.go 
{
	"Dir": "/work/src/github.com/ethereum/go-ethereum/build",
	"ImportPath": "command-line-arguments",
	"Name": "main",
	"Doc": "The ci command is called from Continuous Integration scripts.",
	"Root": "/work",
	"Stale": true,
	"StaleReason": "stale dependency: github.com/ethereum/go-ethereum/vendor/github.com/Azure/go-autorest/autorest",
	"GoFiles": [
		"ci.go"
	],
	"Imports": [
		"bufio",
		"bytes",
		"encoding/base64",
		"flag",
		"fmt",
		"github.com/ethereum/go-ethereum/internal/build",
		"go/parser",
		"go/token",
		"io/ioutil",
		"log",
		"os",
		"os/exec",
		"path/filepath",
		"regexp",
		"runtime",
		"strings",
		"time"
	],
	"Deps": [
		"archive/tar",
		"archive/zip",
		"bufio",
		"bytes",
		"compress/bzip2",
		"compress/flate",
		"compress/gzip",
		"compress/zlib",
		"container/list",
		"context",
		"crypto",
		"crypto/aes",
		"crypto/cipher",
		"crypto/des",
		"crypto/dsa",
		"crypto/ecdsa",
		"crypto/elliptic",
		"crypto/hmac",
		"crypto/internal/cipherhw",
		"crypto/md5",
		"crypto/rand",
		"crypto/rc4",
		"crypto/rsa",
		"crypto/sha1",
		"crypto/sha256",
		"crypto/sha512",
		"crypto/subtle",
		"crypto/tls",
		"crypto/x509",
		"crypto/x509/pkix",
		"encoding",
		"encoding/asn1",
		"encoding/base64",
		"encoding/binary",
		"encoding/hex",
		"encoding/json",
		"encoding/pem",
		"encoding/xml",
		"errors",
		"flag",
		"fmt",
		"github.com/ethereum/go-ethereum/internal/build",
		"github.com/ethereum/go-ethereum/vendor/github.com/Azure/azure-storage-go",
		"github.com/ethereum/go-ethereum/vendor/github.com/Azure/go-autorest/autorest",
		"github.com/ethereum/go-ethereum/vendor/github.com/Azure/go-autorest/autorest/azure",
		"github.com/ethereum/go-ethereum/vendor/github.com/Azure/go-autorest/autorest/date",
		"github.com/ethereum/go-ethereum/vendor/github.com/dgrijalva/jwt-go",
		"github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/cast5",
		"github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/openpgp",
		"github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/openpgp/armor",
		"github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/openpgp/elgamal",
		"github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/openpgp/errors",
		"github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/openpgp/packet",
		"github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/openpgp/s2k",
		"go/ast",
		"go/parser",
		"go/scanner",
		"go/token",
		"hash",
		"hash/adler32",
		"hash/crc32",
		"image",
		"image/color",
		"image/internal/imageutil",
		"image/jpeg",
		"internal/cpu",
		"internal/nettrace",
		"internal/poll",
		"internal/race",
		"internal/singleflight",
		"internal/syscall/unix",
		"io",
		"io/ioutil",
		"log",
		"math",
		"math/big",
		"math/bits",
		"math/rand",
		"mime",
		"mime/multipart",
		"mime/quotedprintable",
		"net",
		"net/http",
		"net/http/cookiejar",
		"net/http/httptrace",
		"net/http/internal",
		"net/textproto",
		"net/url",
		"os",
		"os/exec",
		"os/user",
		"path",
		"path/filepath",
		"reflect",
		"regexp",
		"regexp/syntax",
		"runtime",
		"runtime/cgo",
		"runtime/internal/atomic",
		"runtime/internal/sys",
		"sort",
		"strconv",
		"strings",
		"sync",
		"sync/atomic",
		"syscall",
		"text/template",
		"text/template/parse",
		"time",
		"unicode",
		"unicode/utf16",
		"unicode/utf8",
		"unsafe",
		"vendor/golang_org/x/crypto/chacha20poly1305",
		"vendor/golang_org/x/crypto/chacha20poly1305/internal/chacha20",
		"vendor/golang_org/x/crypto/cryptobyte",
		"vendor/golang_org/x/crypto/cryptobyte/asn1",
		"vendor/golang_org/x/crypto/curve25519",
		"vendor/golang_org/x/crypto/poly1305",
		"vendor/golang_org/x/net/http2/hpack",
		"vendor/golang_org/x/net/idna",
		"vendor/golang_org/x/net/lex/httplex",
		"vendor/golang_org/x/net/proxy",
		"vendor/golang_org/x/text/secure/bidirule",
		"vendor/golang_org/x/text/transform",
		"vendor/golang_org/x/text/unicode/bidi",
		"vendor/golang_org/x/text/unicode/norm"
	]
}

But why does Go 1.10beta1 want to rebuild runtime/cgo, but previous versions of Go didn't want to do that?

[bradfitz]

But why does Go 1.10beta1 want to rebuild runtime/cgo, but previous versions of Go didn't want to do that?

Because you got lucky before. Go 1.10 has proper build caching based on file contents and all relevant environment variables. Go 1.9 and earlier depended heavily on modtimes of files only (not contents) and ignored many important environment variables.

[bradfitz]

That is, in Go 1.9 and before, your runtime/cgo had already been built, so your CC=foo go run ci.go wasn't rebuilding it. But in Go 1.10, it assumes the new $CC value invalidates the cached results (whose cache key includes a different $CC) and rebuilds it.

[karalabe]

Sweet, that's what I was also guessing based on Florin's Advent post, just wanted to confirm. Thank you very much for your time on this, and sorry for the false alarm!