Generate .pb.go files with absolute import paths.

[jeremyje]

We use go modules and we have proto files that can live in 1 of the 2 directories, pkg/pb or internal/pb. We want the internal/pb proto files to reference the pkg/pb files that are emitted from protoc-gen-go and protoc.

Is it possible to do this? The problem right now is the internal/pb files are generated under the assumption that the pkg/pb files live in the same directory which causes compile issues.

We fixed this by sed replacing the pkg/pb with open-match.dev/open-match/pkg/pb in the files, https://github.com/googleforgames/open-match/blob/master/Makefile#L727.

Example pkg/pb
https://github.com/googleforgames/open-match/blob/master/api/backend.proto
https://github.com/googleforgames/open-match/blob/master/pkg/pb/backend.pb.go

Example internal/pb
https://github.com/googleforgames/open-match/blob/master/api/synchronizer.proto
https://github.com/googleforgames/open-match/blob/master/internal/pb/synchronizer.pb.go

[dsnet]

Actually, protoc-gen-go is only ever supposed to be used with absolute paths, not relative paths. The problem seems to be that your .proto file specifies a go_package that looks more like a relative path than absolute Go package path.

[dsnet]

proto files that can live in 2 directories, pkg/pb and internal/pb

I'm not sure what this means. Do you intend for a single .proto file to be generated into two different Go packages? What's the use case for this?

[jeremyje]

If we use absolute paths in the go_package the files are written with the git tree as $(REPOSITORY_ROOT)/open-match.dev/open-match/internal/pb so we use relative paths so the proto files are written to the proper spot. Is there a better way to do this? I tried using the ; in the go_package ex: option go_package = "open-match.dev/open-match/pkg/pb;pkg/pb" and option go_package = "pkg/pb;open-match.dev/open-match/pkg/pb". Neither of these have desirable effects.

We use go modules so we aren't using the Go workspace layout for our project.

[dsnet]

Is #748 perhaps something that more accurately resolves your issue?

[jeremyje]

Is there any documentation on these flags? I tried to grok what source_relative does but I couldn't figure it out.

[dsnet]

https://github.com/golang/protobuf#packages-and-input-paths

EDIT: Given that pkg/pb/backend.pb.go is generated in a different directory from where backend.proto is located, it is unlikely that source_relative will help you. However, the mod_root flag I mentioned in #748 might.

[neild]

To start with, set option go_package to the import path of the package that contains that .proto file. For the specific examples you give:

// In api/backend.proto:
option go_package = "open-match.dev/open-match/pkg/pb";

// In api/synchronizer.proto:
option go_package = "open-match.dev/open-match/internal/pb";

Setting the go_package option like this will let protoc-gen-go generate the right code when one .proto file imports another.

The next question is what protoc command to run to get the output files you want. There are two options:

1. protoc -I.:third_party --go_out=$OUTDIR api/messages.proto
   This will produce $OUTDIR/open-match.dev/open-match/pkg/pb/messages.pb.go. The name is derived from the import path of the Go package.
2. protoc -I.:third_party --go_out=paths=source_relative:/tmp/out api/messages.proto
   This will produce $OUTDIR/api/messages.pb.go. The name is derived from the name of the input file.

Neither of these is exactly what you want, which is pkg/pb/messages.pb.go. The mod_root flag from #748 would be useful here (--go_out=mod_root=open-match.dev/open-match:$OUTDIR), but it doesn't exist yet.

The simplest thing for you to do is probably to pick one of the above options, generate the file into a temporary directory, and copy it into the desired location.

[neild]

(going to close this, but please reopen if this doesn't answer the question)

[maxlandon]

Hello,
I came across this problem as well: I have many Protobuf packages in my project, with many imports.
I have found a solution to this, that is temporary though.

Pre-requisites

1. I use uber/prototool for managing my Protobuf directory. Really good tool, eases your life a lot.
2. Once you have discovered how to use it (or if you know already), open the prototool.yaml file, and in the generate section, there is a extra_modifiers subsection.
   You might see something like:

google/api/annotations.proto: google.golang.org/genproto/googleapis/api/annotations

How this solves your problem

This extra_modifiers line allows you to specify the import path that you want in the Go generated files, with two benefits:

1. It fixes the problem of relative imports.
2. It is totally independant of the directory where protoc will write the generated code.

Example

Assuming you have a foo/bar/baz.proto file that imports a foo/blob/test.proto file,
Assuming the Go path to these files is github.com/user/project/proto/foo/bar/baz,
You will need to add the following lign:
foo/blob/test.proto: github.com/user/project/proto/foo/blob/test. Protoc will detect, for each file importing the blob/test.proto file, that it needs to replace the relative Go import string with this one.

Thus, the proto/gen/foo/bar/baz.pb.go file will have an import string like import "github.com/user/project/proto/gen/foo/blob".

Caveat

The caveat here is that you cannot use wildcards to specify an entire directory package, like foo/bar/*.proto, because yaml (or prototool) doesn't understand them. This means that you will have to add a lign for each proto file that you wish to import in another.

However

Despite these, the prototool project is, IMHO, a really really useful tool when you get to manage complex, structured sets of Protobuf definitions. The option I am talking about in this post is only one of many that can dealt with and automated thanks to prototool.

I hope this answer will be useful to many !