x/tools/cmd/goimports: can't find proto packages

[danielchatfield]

Please answer these questions before submitting your issue. Thanks!

1. What version of Go are you using (go version)?

go version go1.6.2 darwin/amd64

1. What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/danielchatfield"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.6.2/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.6.2/libexec/pkg/tool/darwin_amd64"
GO15VENDOREXPERIMENT="1"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common"
CXX="clang++"
CGO_ENABLED="1"

Package fooproto in github.com/danielchatfield/repo/foo/proto

Running goimports on the following fails to find the import.

package main

func main() {
    _ := &fooproto.SomeStruct{}
}

The first commit that broke this is https://github.com/golang/tools/commit/e047ae774b027ae958a270598c5ac2591e457afc

[bradfitz]

This is a side-effect of the pruning of the search space we now do. We have a function which determines whether a package like "fooproto" might appear in a directory as a function of its directory name, considering mostly its final two path components.

It ignores case, and drops hyphens and periods, and then looks for substrings. But "fooproto" doesn't appear in "foo/proto". I suppose I could also try concatenating the last two path components and searching that too.

But does that solve the general problem?

How do protobuf package names & directory names look like in the wild? I haven't been following what the conventions are outside of Google or at least outside of Google open source projects.

[akalin-keybase]

Is this one instance of the general problem that a package whose name differs from its containing directory won't be found by goimports anymore?

i.e. if I have $GOPATH/src/github.com/akalin/mymain/main.go with:

package main

import (
    "github.com/akalin/mypkg"
    "github.com/akalin/mypkg2"
)

func main() {
    mypkg.Foo()
    myotherpkg.Bar()
}

where the mypkg package is in github.com/akalin/mypkg, but the myotherpkg package is in github.com/akalin/mypkg2, then goimports will automatically insert the first import if omitted, but won't insert the second import if omitted? (This is the current behavior, according to my tests.)

Sounds like this is a casualty of the new speed improvements. If so, it's unfortunate but understandable. I didn't run into this with protobuf packages, but with similar auto-generated code, namely https://github.com/keybase/client/tree/master/go/protocol , which has package name keybase1 but directory name protocol.

It's probably impossible to solve this in general without slowing down goimports again, but I wonder if we could have some sort of convention for auto-generated code, such that the directory name is some function of the package name, e.g. pkgName + "Autogen".

[bradfitz]

Sounds like this is a casualty of the new speed improvements.

Yup.

If so, it's unfortunate but understandable.

It's at least fortunate in one way: it gives yet another reason to name your packages the conventional way, making it a bit tedious if you want to violate the norms.

I definitely wouldn't penalize people like that on purpose, but if it's necessary to make goimports fast, I think it's quite fine.

I wouldn't support any proposal (like pkgName + "Autogen") to make violating the norm easier. There's no reason for autogen packages to declare that they're auto-generated in their package name. If it really needs to be in the import path, it can be in the second to last path component or something.

[bradfitz]

@dsymonds, can you answer my question in #16600 (comment)? Are there conventions for how people name their proto packages vs. how they end up on disk in their import path?

[dsymonds]

I'm not sure there's a well-established convention. We encourage people to always rename proto imports for this kind of reason.

If you're going to apply any hack, I'd suggest bypassing the pruning if the import name ends in "pb", which should rarely happen except for protos.

[cespare]

At $COMPANY our protobuf build scripts move our generated code to a directory that matches the package name specifically to avoid violating the Go convention and to make sure that tools like goimports work.

(As far as the special names thing goes, we also name our packages specially using option go_package = "foopb"; to make it easier to identify which packages are generated code; this seems orthogonal to the package name != path issue, though.)

[bradfitz]

If you're going to apply any hack, I'd suggest bypassing the pruning if the import name ends in "pb", which should rarely happen except for protos.

Oh, that's an interesting idea, to selectively prune. Or I can at least filter off "pb", and then prune.

Does some tool default to the "pb" suffix, or is that just convention?

[dsymonds]

It's a strong convention. The part before the "pb" (and sometimes there's nothing before the pb) is human chosen, and isn't always consistent for a given imported package. From memory, golint and other similar tools know the "pb" suffix convention and modify their behaviour a bit.