x/sys/unix: problem compiling yubihsm-connector

[mouse07410]

A follow-up to/continuation of the prematurely-closed #30777

What version of Go are you using (go version)?

$ go version
go version go1.12 darwin/amd64
$

Does this issue reproduce with the latest release?

The latest release seems to be the only version where this issue appears.

What operating system and processor architecture are you using (go env)?

macOS Mojave 10.14.3, Xcode-10.1, current Go package for Mac from https://golang.org, and current gb installed following https://getgb.io/docs/install/ instructions.

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/ur20980/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/ur20980/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/c6/lnc_0m093ys8w16md_fm1mnxhtfnj8/T/go-build615060564=/tmp/go-build -gno-record-gcc-switches -fno-common"
$ gb info
GB_PROJECT_DIR="/Users/ur20980/src/yubihsm-connector"
GB_SRC_PATH="/Users/ur20980/src/yubihsm-connector/src:/Users/ur20980/src/yubihsm-connector/vendor/src"
GB_PKG_DIR="/Users/ur20980/src/yubihsm-connector/pkg/darwin-amd64"
GB_BIN_SUFFIX="-darwin-amd64"
GB_GOROOT="/usr/local/go"
$ gb env
GB_PROJECT_DIR="/Users/ur20980/src/yubihsm-connector"
GB_SRC_PATH="/Users/ur20980/src/yubihsm-connector/src:/Users/ur20980/src/yubihsm-connector/vendor/src"
GB_PKG_DIR="/Users/ur20980/src/yubihsm-connector/pkg/darwin-amd64"
GB_BIN_SUFFIX="-darwin-amd64"
GB_GOROOT="/usr/local/go"
$ 

What did you do?

Updated golang.org/x/sys/unix package explicitly (after updating everything with --all), and tried building again with gb build

What did you expect to see?

Successful build.

What did you see instead?

$ gb vendor update golang.org/x/sys/unix
$ gb build
golang.org/x/sys/unix
github.com/fsnotify/fsnotify
golang.org/x/crypto/ssh/terminal
github.com/sirupsen/logrus
github.com/spf13/viper
github.com/sirupsen/logrus/hooks/syslog
golang.org/x/sys/unix.libc_close_trampoline·f: relocation target golang.org/x/sys/unix.libc_close_trampoline not defined for ABIInternal (but is defined for ABI0)
golang.org/x/sys/unix.libc_ioctl_trampoline·f: relocation target golang.org/x/sys/unix.libc_ioctl_trampoline not defined for ABIInternal (but is defined for ABI0)
golang.org/x/sys/unix.libc_kevent_trampoline·f: relocation target golang.org/x/sys/unix.libc_kevent_trampoline not defined for ABIInternal (but is defined for ABI0)
golang.org/x/sys/unix.libc_kqueue_trampoline·f: relocation target golang.org/x/sys/unix.libc_kqueue_trampoline not defined for ABIInternal (but is defined for ABI0)
golang.org/x/sys/unix.libc_mmap_trampoline·f: relocation target golang.org/x/sys/unix.libc_mmap_trampoline not defined for ABIInternal (but is defined for ABI0)
golang.org/x/sys/unix.libc_munmap_trampoline·f: relocation target golang.org/x/sys/unix.libc_munmap_trampoline not defined for ABIInternal (but is defined for ABI0)
golang.org/x/sys/unix.libc_open_trampoline·f: relocation target golang.org/x/sys/unix.libc_open_trampoline not defined for ABIInternal (but is defined for ABI0)
# yubihsm-connector
yubihsm-connector
FATAL: command "build" failed: exit status 2
$ 

In #30777 you said

You need to update to the current version of the golang.org/x/sys/unix package...

Would you mind telling how you suggest doing that, and why in your opinion my update (as shown above) is not sufficient?

[agnivade]

Could you please show us the repro steps without gb ? We would want to eliminate as many variables as possible to home in on the target issue. If it is an issue with go build, then it should fail using normal go commands also.

[ianlancetaylor]

I'm sorry, I don't know what gb vendor update does.

[mouse07410]

Could you please show us the repro steps without gb ?

I would be happy to - but, alas, I'm not a go programmer, and, frankly, wouldn't have touched this whole stuff - except that the package I need is (misguidedly, IMHO) written in Go. So I've no choice.

The good part of it is - my installation is very close to the standard because I don't know enough to mess with it. The bad part is - I don't even know how to do what you asked for, i.e., how to translate what gb does into go commands.

Here's what the Makefile does:

build:
	@gb generate ${GB_GEN_FLAGS}
	@gb build ${GB_BUILD_FLAGS}

Both of the env vars referred above are empty. Here's the entire Makefile:
Makefile.txt (those @ are just to make the command silent - no echo from make)

If you can tell me what to do/try to help debugging this problem - I'll be happy to do it.

[mouse07410]

I'm sorry, I don't know what gb vendor update does.

It does (at least according to its documentation - I probably have even less experience with it than you do) exactly what you suggested is needed - updates the given package, or if --all is specified in place of the package name - all the packages in src/vendor.

So there's no reason that I can see to assume that golang.org/x/sys/unix in src/vendor directory of this package is any older than current.

[mouse07410]

Also, in case it matters - I successfully built this package end of last November, with whatever Go compiler was stable release back then. It doesn't look like the relevant parts of it changed since.

[ianlancetaylor]

The problematic changes to golang.org/x/sys/unix were made in December, and were designed to work with Go 1.12.

I have used a Darwin system, downloaded yubihsm-connector, installed the gb build tool, run gb vendor update --all, which does change the working directory vendor/src/golang.org/x/sys/unix among other directories, and ran gb build. It builds golang.org/x/sys/unix successfully, but then gives the error

# github.com/spf13/afero
util.go:28:2: can't find import: "golang.org/x/text/transform"
FATAL: command "build" failed: exit status 2

Building and testing golang.org/x/sys/unix in the usual way, with the Go tool, works fine.

OK, I think I may know what is happening. In the Go 1.12 cycle the compiler, assembler, and linker were changed to track the ABI version of assembler symbols; see #27539. As part of making this work, the go tool itself was changed to build an ABI file; see https://golang.org/cl/147257 (some later CLs were also involved).

It is likely that gb build has not been updated for this. Until that happens, it may not be possible to use gb build with Go 1.12 with the golang.org/x/sys/unix package.

[mouse07410]

@ianlancetaylor first, thank you for your effort!

In your opinion, is is possible to substitute go <something> for gb build? That might resolve my immediate problem, at least - and since you managed to build some packages, perhaps you could advise me on how to build the rest of them?

[ianlancetaylor]

I'm sorry, I don't know the answer to that question. I haven't used gb. I suggest that you ask on the golang-nuts Google Group or in some other forum; see https://golang.org/wiki/Questions.

[mouse07410]

@ianlancetaylor thank you. This probably isn't something that go can address/fix.

One workaround is to adjust the GOPATH for the project, making it point to the dependencies dir. E.g., if the project is in $HOME/dir/test, the project code itself is in $HOME/dir/test/src/test and the deps are in $HOME/dir/test/vendor/src (as per gb requirements), the correct setting for the build would be

$ cd $HOME/dir/test/src/dir
$ GOPATH=$GOPATH:$HOME/dir/test/vendor go build

Then the build would succeed.

Perhaps the above would be useful for others.