Duplicate Functions and Variables in Generated Package

[michael-james-holloway]

First, just want to say thank you for working on and releasing vtprotobuf. We're working on transitioning out of gogo/protobuf and read your great blog post announcing this alternative.

We've found a slight issue with our use case. We have a few proto packages that have multiple files in them. The generated _vtproto.pb.go files redeclare some utility functions/variables (e.g. sov, skip, ErrInvalidLength, etc.). As an example:

hellopb/service.proto:

syntax = "proto3";

package hellopb;

...

message HelloRequest {
  string q = 1;
}

message HelloResponse {
  string response = 2;
}

service HelloService {
  rpc Hello(HelloRequest) returns (HelloResponse) {}
}

hellopb/db.proto:

syntax = "proto3";

package hellopb;

...

message MessageEntry {
  string q = 1;
  ...
}

We then run:

protoc --proto_path=. --proto_path=../../../  \
  --go_out=../../../ --plugin protoc-gen-go=/go/bin/protoc-gen-go \
  --go-grpc_out=../../../ --plugin protoc-gen-go-grpc=/go/bin/protoc-gen-go-grpc \
  --grpc-gateway_out=../../../ \
  --go-vtproto_out=../../../ --plugin protoc-gen-go-vtproto=/go/bin/protoc-gen-go-vtproto \
  --go-vtproto_opt=features=marshal+unmarshal+size hellopb/service.proto

and

protoc --proto_path=. --proto_path=../../../  \
  --go_out=../../../ --plugin protoc-gen-go=/go/bin/protoc-gen-go \
  --go-grpc_out=../../../ --plugin protoc-gen-go-grpc=/go/bin/protoc-gen-go-grpc \
  --grpc-gateway_out=../../../ \
  --go-vtproto_out=../../../ --plugin protoc-gen-go-vtproto=/go/bin/protoc-gen-go-vtproto \
  --go-vtproto_opt=features=marshal+unmarshal+size hellopb/db.proto

If we run go vet we get:

service_vtproto.pb.go:214:6: encodeVarint redeclared in this block
    db_vtproto.pb.go:125:54: previous declaration
service_vtproto.pb.go:308:6: sov redeclared in this block
    db_vtproto.pb.go:188:23: previous declaration
service_vtproto.pb.go:311:6: soz redeclared in this block
    db_vtproto.pb.go:191:23: previous declaration
service_vtproto.pb.go:694:6: skip redeclared in this block
    db_vtproto.pb.go:545:36: previous declaration
service_vtproto.pb.go:774:2: ErrInvalidLength redeclared in this block
    db_vtproto.pb.go:625:2: previous declaration
service_vtproto.pb.go:775:2: ErrIntOverflow redeclared in this block
    db_vtproto.pb.go:626:2: previous declaration
service_vtproto.pb.go:776:2: ErrUnexpectedEndOfGroup redeclared in this block
    db_vtproto.pb.go:627:2: previous declaration

Is there anyway to avoid this? Post protoc cleanup on this is pretty tough. Is there anyway those functions/variables could just be imported from vtprotobuf?

[coufalja]

We had the same issue, and a way to avoid this is to merge the protoc call into a single command, that way the vtprotobuf avoids the duplication. Try passing both protofiles to a single command instead:

protoc --proto_path=. --proto_path=../../../  \
  --go_out=../../../ --plugin protoc-gen-go=/go/bin/protoc-gen-go \
  --go-grpc_out=../../../ --plugin protoc-gen-go-grpc=/go/bin/protoc-gen-go-grpc \
  --grpc-gateway_out=../../../ \
  --go-vtproto_out=../../../ --plugin protoc-gen-go-vtproto=/go/bin/protoc-gen-go-vtproto \
  --go-vtproto_opt=features=marshal+unmarshal+size hellopb/service.proto hellopb/db.proto

[michael-james-holloway]

Works perfectly, thanks @coufalja.

[kevhong]

Would it be possible to support a flag to configure referencing the functions or generating unique names? (e.g. --go-vtproto_opt=include-functions=false) In my generation script, I had to write a hack to rename all the functions as it's not easy to pass all the protos.