You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have need to interop with some tensorflow protobufs. However, the ones in question expect to be able to import each other in their current tensorflow/core/example layout. See example.proto
I've tried various things to get a copy of these files working in my own little monorepo, but protoc-gen-go doesn't quite help me out.
I have a monolithic repository that is kept at the path foobar in my GOPATH. The Go code inside it is kept at foobar/go.
At first, I tried just including the proto files in the package I was using them, foobar/go/tfrec, since I just need a way to write TFRecord files. But that causes protoc to crash saying that the imported file tensorflow/core/example/feature.proto couldn't be found.
Okay, so, I went and tried to add a foobar/proto directory since protobufs are cross-language, anyway, so maybe they deserve their own directory. But I'd rather not compile the Go code in there means that they will be outside my foobar/go code where my repo's tools are expecting them.
That seemed to work out, but I discovered it only works because those proto files expect to be generated inside one package. If you have a proto file that depends on a proto file that generates code in another package, and attempt to generate code in a directory other than where the .proto file was, the generated code won't build because the import path to the other generated package is wrong.
So, you'd say "oh, well, use import_prefix to solve that". Well, that doesn't work, because import_prefix also adjusts the import path of the github.com/google/protobuf import! And that doesn't work because I've vendored github.com/google/protobuf and its import path needs to remain the same. Using a sed to clean this up is a fragile hack.
Ah, then there's the M config you could use. The problem there is that means you have to track your proto imports in two places: once in the actual proto files, and a second time where ever you're running protoc. This is especially annoying when working with third-party proto files.
I have one proposed solution: I have code that adds a proto_prefix setting to protoc-gen-go that only adjusts the import paths to other generated proto packages. While annoying to remember the first time, it means only having to solve this once for all of my repo's packages and not having to update my build tooling every time a new import is added to a proto file.
I'm not entirely sure how this is working out in other people's repositories, though, so perhaps I've missed something obvious.
The text was updated successfully, but these errors were encountered:
It looks like tensorflow builds using bazel (https://bazel.io/), which has its own system for building proto packages. It's building a "tensorflow" package from protos in many different directories. There's no easy way to do this directly with protoc, so far as I know.
If the tensorflow source files each included an option go_package line (as detailed in #139), you could build the Go package with something like:
I have need to interop with some tensorflow protobufs. However, the ones in question expect to be able to import each other in their current
tensorflow/core/example
layout. See example.protoI've tried various things to get a copy of these files working in my own little monorepo, but protoc-gen-go doesn't quite help me out.
I have a monolithic repository that is kept at the path
foobar
in myGOPATH
. The Go code inside it is kept atfoobar/go
.At first, I tried just including the proto files in the package I was using them,
foobar/go/tfrec
, since I just need a way to write TFRecord files. But that causes protoc to crash saying that the imported filetensorflow/core/example/feature.proto
couldn't be found.Okay, so, I went and tried to add a
foobar/proto
directory since protobufs are cross-language, anyway, so maybe they deserve their own directory. But I'd rather not compile the Go code in there means that they will be outside myfoobar/go
code where my repo's tools are expecting them.That seemed to work out, but I discovered it only works because those proto files expect to be generated inside one package. If you have a proto file that depends on a proto file that generates code in another package, and attempt to generate code in a directory other than where the .proto file was, the generated code won't build because the import path to the other generated package is wrong.
So, you'd say "oh, well, use
import_prefix
to solve that". Well, that doesn't work, becauseimport_prefix
also adjusts the import path of thegithub.com/google/protobuf
import! And that doesn't work because I've vendoredgithub.com/google/protobuf
and its import path needs to remain the same. Using a sed to clean this up is a fragile hack.Ah, then there's the
M
config you could use. The problem there is that means you have to track your proto imports in two places: once in the actual proto files, and a second time where ever you're running protoc. This is especially annoying when working with third-party proto files.I have one proposed solution: I have code that adds a
proto_prefix
setting to protoc-gen-go that only adjusts the import paths to other generated proto packages. While annoying to remember the first time, it means only having to solve this once for all of my repo's packages and not having to update my build tooling every time a new import is added to a proto file.I'm not entirely sure how this is working out in other people's repositories, though, so perhaps I've missed something obvious.
The text was updated successfully, but these errors were encountered: