Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go tries to recompile already installed packages: missing headers #27027

Closed
pebbe opened this issue Aug 16, 2018 · 5 comments

Comments

Projects
None yet
5 participants
@pebbe
Copy link
Contributor

commented Aug 16, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11rc1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/tmp/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/peter/go"
GOPROXY=""
GORACE=""
GOROOT="/my/opt/go"
GOTMPDIR="/tmp"
GOTOOLDIR="/my/opt/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build728326710=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I tried to build this program:

package main

import (
	"fmt"
	"github.com/pebbe/dbxml"
)

func main() {
	fmt.Println(dbxml.Version())
}

What did you expect to see?

I expected to get no errors. The package github.com/pebbe/dbxml is installed in GOPATH.

What did you see instead?

# github.com/pebbe/dbxml
c_dbxml.cc:2:27: fatal error: dbxml/DbXml.hpp: Bestand of map bestaat niet
 #include <dbxml/DbXml.hpp>
                           ^
compilation terminated.

Go tries to recompile the package that is already installed, but it fails because the header files are in a non-standard location. The package itself was installed with a modified environment, like this:

DBXML=/path/to/dbxml # location of Oracle Berkeley DB XML
export CGO_LDFLAGS="-L$DBXML/lib -Wl,-rpath=$DBXML/lib"
export LIBRARY_PATH=$DBXML/lib
export CPATH=$DBXML/include
go install github.com/pebbe/dbxml

When I try to compile the example program, these environment variables are unavailable.

This all used to work fine in older versions of Go. I don't when this changed. Go 1.10? Go 1.11?

@mvdan

This comment has been minimized.

Copy link
Member

commented Aug 16, 2018

This is likely because of GOCACHE. If I remember correctly, the go env used to build each package is recorded too, so that changing any settings (such as CGO_LDFLAGS) will properly trigger a rebuild.

You might have better luck using this package as a binary package. I assume the other option is to not change go env.

@andybons andybons changed the title Go tries to recompile already installed packages: missing headers cmd/go: go tries to recompile already installed packages: missing headers Aug 16, 2018

@andybons andybons added this to the Unplanned milestone Aug 16, 2018

@andybons

This comment has been minimized.

Copy link
Member

commented Aug 16, 2018

@pebbe

This comment has been minimized.

Copy link
Contributor Author

commented Aug 17, 2018

Compiling with GOCACHE=off doesn't solve the problem.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Aug 18, 2018

The package itself was installed with a modified environment

GOPATH/pkg is not something you can micromanage anymore. It's just a set of 1-entry caches. If they're up-to-date for the requested build, they'll be used; otherwise not. The go command can now tell that relevant environment variables have changed since the package was built (in particular, $CGO_LDFLAGS changed), so it does not reuse the (stale) installed package. In general, a change to CGO_LDFLAGS should trigger a rebuild, so sorry but this is working as intended.

I'm not sure what to say except that you now need to keep those environment variables set the same way from build to build.

@rsc rsc closed this Aug 18, 2018

@pwaller

This comment has been minimized.

Copy link
Contributor

commented Aug 30, 2018

@rsc has a blog post somewhere (which I have failed to find) which introduces the notion of content-based staleness - the closest I could find was this golang-nuts post where he introduces it. But I recall an 'aha' moment when I read that. go install is now just a function of its inputs, giving you a reproducible build, and the cache is "just a cache", not something you can poke. This means that the binaries you get from a build should not depend on what commands happened to run previously.

If I recall correctly, in the post I can't find, @rsc mentioned that initially he thought - and you might think - that 'depending on what happened to run before' was a feature which enabled you to do various things (as in this issue). He came around (and in his writing, I came around) to the idea that actually those problems can be solved through other means, giving us the very useful property that go install is just a function of the inputs you present it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.