go/build and go/types: import path bug in go1.5

[netbrain]

What version of Go are you using (go version)?
go version go1.5rc1 linux/amd64

What operating system and processor architecture are you using?
debian jessie / amd64

What did you do?
Im trying to read package information with the go/build package, then parsing the package using the go/types package. The package i'm trying to parse has a dependency on another package, and this is where the issue lies.

What did you expect to see?
Successfull import of package "bar"

What did you see instead?
Failed to import package "bar"

I have created a code sample which reproduces this issue: https://github.com/netbrain/importbug

netbrain@netbox:~/dev/go/src/github.com/netbrain/importbug$ echo $GOPATH
/home/netbrain/dev/go
netbrain@netbox:~/dev/go/src/github.com/netbrain/importbug$ tree
.
├── bar
│   └── bar.go
├── foo
│   └── foo.go
└── importbug_test.go

2 directories, 4 files
netbrain@netbox:~/dev/go/src/github.com/netbrain/importbug$ go test
2015/08/06 16:39:59 /home/netbrain/dev/go/src/github.com/netbrain/importbug/foo/foo.go:3:8: could not import github.com/netbrain/importbug/bar (can't find import: github.com/netbrain/importbug/bar)
exit status 1
FAIL    github.com/netbrain/importbug   0.002s

I tried to debug the issue with gdb, and a discovered a couple of things, when build.Import is invoked on the package "github.com/netbrain/importbug/bar" then on line 603 in build.go this gets resolved to "/home/netbrain/dev/go/pkg/linux_amd64/github.com/netbrain/importbug/bar.a", which doesn't exist.

furthermore, the for loop @ gcimporter.go:66 loops through the following filenames
"/home/netbrain/dev/go/pkg/linux_amd64/github.com/netbrain/importbug/bar.a"
"/home/netbrain/dev/go/pkg/linux_amd64/github.com/netbrain/importbug/bar.o"

as these files doesn't exist, it simply returns a blank string. which means the package was not found. Resulting in the error displayed in the test.

The following is a backtrace of the test, showing the involved packages and function calls.

#0  go/build.(*Context).Import (ctxt=0x8b2c20 <go/build.Default>, path="", srcDir=".", 
    mode=3, ~r3=0x0, ~r4=...) at /home/netbrain/tools/go/src/go/build/build.go:460
#1  0x000000000050b74b in go/build.Import (path="github.com/netbrain/importbug/bar", 
    srcDir=".", mode=3, ~r3=0x40f446 <runtime.mallocgc+1766>, ~r4=...)
    at /home/netbrain/tools/go/src/go/build/build.go:1002
#2  0x00000000005c4084 in go/internal/gcimporter.FindPkg (
    path="github.com/netbrain/importbug/bar", srcDir=".", filename="", 
    id="github.com/netbrain/importbug/bar")
    at /home/netbrain/tools/go/src/go/internal/gcimporter/gcimporter.go:47
#3  0x00000000005c46f9 in go/internal/gcimporter.Import (
    packages=map[string]*go/types.Package, path="github.com/netbrain/importbug/bar", 
    pkg=0x0, err=...) at /home/netbrain/tools/go/src/go/internal/gcimporter/gcimporter.go:128
#4  0x0000000000511a51 in go/importer.gcimports.Import (m=0xc820011320, 
    path="github.com/netbrain/importbug/bar", ~r1=0x21, ~r2=...)
    at /home/netbrain/tools/go/src/go/importer/importer.go:57
#5  0x0000000000561777 in go/types.(*Checker).collectObjects (check=0xc82005b750)
    at /home/netbrain/tools/go/src/go/types/resolver.go:178
#6  0x000000000053b6f3 in go/types.(*Checker).Files (check=0xc8200a21c0, 
    files= []*go/ast.File = {...}, err=...)
    at /home/netbrain/tools/go/src/go/types/check.go:223
#7  0x000000000052b222 in go/types.(*Config).Check (conf=0xc820018f40, path="foo", 
    fset=0xc820018d00, files= []*go/ast.File = {...}, info=0xc820075b30, ~r4=0x0, ~r5=...)
    at /home/netbrain/tools/go/src/go/types/api.go:311
#8  0x0000000000482098 in github.com/netbrain/importbug.TestImportBug (t=0xc82009c1b0)
    at /home/netbrain/dev/go/src/github.com/netbrain/importbug/importbug_test.go:40

in gdb I had the following breakpoints set

break resolver.go:178
break gcimporter.go:116
break gcimporter.go:37
break gcimporter.go:47
break build.go:460
break build.go:1002

[netbrain]

As far as I can see, the issue is related to the following code.

347cc981 src/pkg/go/build/build.go (Russ Cox             2012-03-06 00:36:24 -0500 479)         case "gc":
5f418195 src/go/build/build.go     (Michael Hudson-Doyle 2015-04-14 10:20:18 +0200 480)                 pkgtargetroot = "pkg/" + ctxt.GOOS + "_" + ctxt.GOARCH + suffix
5f418195 src/go/build/build.go     (Michael Hudson-Doyle 2015-04-14 10:20:18 +0200 481)                 pkga = pkgtargetroot + "/" + p.ImportPath + ".a"

Where the variable "pkga" is given a value of "/home/netbrain/dev/go/pkg/linux_amd64/github.com/netbrain/importbug/bar.a", which is a file that doesn't exist.

@mwhudson, @rsc any idea whats going on with this? any help is greatly appreciated!

[davecheney]

I believe this is not a bug

importbug/imporbug_test.go does not import "
github.com/netbrain/importbug/foo", or "github.com/netbrain/importbug/bar"
so they will not be compiled and stored in $GOPATH/pkg.

If I go install those packages then run the test it passes as expected.

On Thu, Aug 6, 2015 at 8:59 PM, Kim Eik notifications@github.com wrote:

As far as I can see, the issue is related to the following code.

347cc98 src/pkg/go/build/build.go (Russ Cox 2012-03-06 00:36:24 -0500 479) case "gc":
5f41819 src/go/build/build.go (Michael Hudson-Doyle 2015-04-14 10:20:18 +0200 480) pkgtargetroot = "pkg/" + ctxt.GOOS + "_" + ctxt.GOARCH + suffix
5f41819 src/go/build/build.go (Michael Hudson-Doyle 2015-04-14 10:20:18 +0200 481) pkga = pkgtargetroot + "/" + p.ImportPath + ".a"

Where the variable "pkga" is given a value of
"/home/netbrain/dev/go/pkg/linux_amd64/github.com/netbrain/importbug/bar.a",
which is a file that doesn't exist.

@mwhudson https://github.com/mwhudson, @rsc https://github.com/rsc
any idea whats going on with this? any help is greatly appreciated!

—
Reply to this email directly or view it on GitHub
#12050 (comment).

[netbrain]

I guess that makes a lot of sense. however github.com/netbrain/importbug/foo is correctly parsed with buildPkg, err := build.Import("github.com/netbrain/importbug/foo", "", build.ImportComment) so why shouldn't bar be aswell? As far as i can tell both build.Import() and conf.Check() (line 40 in the test) both uses the build.Import() which points to gcimports.Import().

[davecheney]

Can you try after running rm -rf $GOPATH/pkg ?

On Thu, Aug 6, 2015 at 9:10 PM, Kim Eik notifications@github.com wrote:

I guess that makes a lot of sense. however
github.com/netbrain/importbug/foo is correctly parsed parsed with buildPkg,
err := build.Import("github.com/netbrain/importbug/foo", "",
build.ImportComment) so why shouldn't bar be aswell? As far as i can tell
both build.Import() and conf.Check() (line 40 in the test) both uses the
build.Import() which points to gcimports.Import().

—
Reply to this email directly or view it on GitHub
#12050 (comment).

[netbrain]

Still the same result

2015/08/06 13:14:14 /home/netbrain/dev/go/src/github.com/netbrain/importbug/foo/foo.go:3:8: could not import github.com/netbrain/importbug/bar (can't find import: github.com/netbrain/importbug/bar)

[netbrain]

i can confirm running go install in github.com/netbrain/importbug/bar will fix the test. However, this would mean that all packages i would want to inspect with go/types would have to be installed?

[davecheney]

However, this would mean that all packages i would want to inspect with go/types would have to be installed?

This looks to be the case. The only implementation of types.Importer I could find was go/importer which operated on the compiled object file. I don't know if this is a hard restriction, or just the most expedient.

[netbrain]

I find this quite weird, one would think that if go/build.build.Import() succeeds in parsing a package which isn't previously installed. Then go/types.Config.Check() surely should have an identical behavior as they both use go/importer.Import() which in turn, atleast in my case uses go/internal/gcimporter.

I guess I will look some more at this issue. Anyways thank you for your assistance thus far 👍

[ianlancetaylor]

I think this is working as expected, but Robert can decide. In any case it may require better documentation.

[griesemer]

@netbrain This is working as intended.

go/types type-checks a single package, the one for which the source (== AST) is provided. Any imports are delegated to the importer which looks at compiled packages (which must be installed).

It is possible to provide an importer that looks at the source: such an importer would have to first parse and type-check the imported package from source, and possibly do the same with its own imports, recursively so. There are also questions about how much to cache (we don't want to re-parse and re-type-check the same package over and over) and when to invalidate the cache, which files to consider, etc. It's not entirely trivial to get this right on all platforms and taking into account build flags etc. which is why importing is separated out from go/types as an independent mechanism.

The x/tools/go/loader package provides much of the functionality but it's not in the std lib yet, exactly because it's complicated and the API is not yet fixed.

If you need this working for you now, you can implement your own importer and provide it to go/types. You could look at (and copy from) src/cmd/api/goapi.go:421 for an outline of an importer that does about what you might want, and then customize it as needed.

Eventually go/importer may provide a source importer.

[netbrain]

@griesemer thank you for the detailed explanation. For the performance reasons you describe it is probably better to require installation of all packages instead of creating my own source importer. But i guess the documentation or atleast the error message should indicate that imported packages through go/types need to be compiled in order for the library to discover them. Anyways, ill leave that up to you guys. Thanks again!

[griesemer]

go/types doesn't care or even know how the package data structure of imported packages came into being, it just cares about the data which is returned by an importer. go/types' access to imported packages is via the importer function which is implemented externally.

The go/importer package provides (at the moment) two importers, one for packages compiled by gc and one for packages compiled by gccgo. The documentation in that package explicitly mentions the compilers. But it is correct that it doesn't say that the importers read from the installed binary files.

@alandonovan has been working on a longer and detailed explanation of how to use go/types and how it all interacts with the source code, installed code, etc. It will be available with the 1.5 release (or shortly thereafter).