Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proto: panic: duplicate enum registered: google.protobuf.FieldDescriptorProto_Type #178

Closed
Globegitter opened this issue Apr 27, 2016 · 18 comments

Comments

@Globegitter
Copy link

Globegitter commented Apr 27, 2016

I am trying to use custom options in our codebase, as mentioned here: https://developers.google.com/protocol-buffers/docs/proto#customoptions

Our codebase actively supports python, go and java. So protos are built against these three environments. It seems while custom options work fine for java and python there seems to be an issue with go leading to following error message when running code coverage on one of our tests:

2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FieldMask
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.Timestamp
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.Any
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FileDescriptorSet
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FileDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.DescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.DescriptorProto.ExtensionRange
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FieldDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.EnumDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.EnumValueDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.ServiceDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.MethodDescriptorProto
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FileOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.MessageOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.FieldOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.EnumOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.EnumValueOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.ServiceOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.MethodOptions
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.UninterpretedOption
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.UninterpretedOption.NamePart
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.SourceCodeInfo
2016/04/27 15:33:58 proto: duplicate proto type registered: google.protobuf.SourceCodeInfo.Location
panic: proto: duplicate enum registered: google.protobuf.FieldDescriptorProto_Type

goroutine 1 [running]:
panic(0x8fa960, 0xc8201552d0)
    /usr/lib/go/src/runtime/panic.go:464 +0x3e6
github.com/golang/protobuf/proto.RegisterEnum(0xbbca60, 0x29, 0xc820199980, 0xc8201999b0)
    /path/to/third_party/go/ptypes/src/github.com/golang/protobuf/proto/properties.go:811 +0xe2
third_party/proto/google/protobuf.init.4()
    third_party/proto/google/protobuf/descriptor.pb.go:1913 +0x706
third_party/proto/google/protobuf.init()
    third_party/proto/google/protobuf/descriptor.pb.go:2798 +0x5cd
main.init()
    /path/to/lobby_test_main.go:149 +0x7e

The issue being here we are importing the module third_party/proto/google/protobuf multiple times like so:

import ( 
...
    _cover9 "third_party/proto/google/protobuf"

    _cover10 "third_party/proto/google/protobuf"

    _cover11 "third_party/proto/google/protobuf"

    _cover12 "third_party/proto/google/protobuf"
)

these are being used to set up the coverage before running the actual tests. If I remove these imports and the coverage instrumentation for them all works fine.

So it seems to me that the init function of the descriptor.pb.go is being called multiple times, which in turn registers the enum multiple times and then fails. I don't quite understand how importing the module multiple times can have any effect. If I import our own protos multiple times (that also register enums) everything seems fine.

Is that an issue with the generated code? And is there any way to fix this? (possibly via #131 ?)

@dsymonds
Copy link
Contributor

Importing the same package multiple times shouldn't cause multiple runs of its init functions. Is it possible you have the same package being imported under more than one import path? That would cause the observed behaviour.

@Globegitter
Copy link
Author

@dsymonds Thanks for the response; that should not be happening but I will look into that.

Also just tested #131 and it would indeed fix it given that we would add the right go_package statement upstream. Is there any possibility of this happening?

@Globegitter
Copy link
Author

There is nowhere that I can find it being imported under more than one import path right now, but given that a solution more or less already exists in two other existing PRs it would be much preferable to wait for that.

Then we don't have to do our custom imports and this would also be on par with how the python library do it and simplify things quite a bit for cross-langauge support.

@erikreppel
Copy link

Did this ever get any kind of resolution? I'm hitting this issue when trying to use the gcp datastore library and protobufs in the same project.

@douglas-reid
Copy link

I, too, am seeing this issue when trying to use any of the google cloud client libraries and protobufs in the same project.

@zombiezen
Copy link
Contributor

/cc @jba

I would echo dsymond's question: are you sure you haven't put the same generated proto under multiple import paths?

@jba
Copy link

jba commented Sep 23, 2016

@okdave is working on the go-package option in some files.

We did recently (a month ago?) put a lot of protos in a common place. For example, FieldMask is now in https://github.com/google/go-genproto/blob/master/protobuf/field_mask.pb.go, which has the import path "google.golang.org/genproto/protobuf". Does this also live under another import path?

@okdave
Copy link
Contributor

okdave commented Sep 26, 2016

If you're seeing this, it would be good to understand your project's layout. It's possible that you'll see this issue with vendoring – for example, having the same proto .pb.go file vendored into two different places in your source tree.

@yohcop
Copy link

yohcop commented Oct 27, 2016

I have the same problem, and I'll try to describe what is happening to me. I've tried many different things/layouts, and had 3 problems - and 0 solutions so far. The enum being registered multiple times was the starting point.

Problem 1

  • I use this package, and my code uses descriptor.pb.go from this repo (because I implement a protoc plugin)
  • since this repo doesn't include descriptor.proto, I needed to get it from the original repo to use it in my own .proto files
  • my.proto file uses the descriptor.proto of course, to add field and message extensions
  • I also use Bazel... so my go_proto_library rule for my.proto, depends on the go_proto_library rule I hand-created for descriptor.proto.
  • that last rule for descriptor.proto, creates a package, google/protobuf (later I'll explain what happens if that package is set to github.com/golang/....)
  • I now have 2 libraries, google/protobuf and github.com/golang/.../descriptor defining those enums. They are both linked because I used this package, as well as my proto which is linked against descriptor.proto.

Problem 2

My code was importing github.com/golang/.../descriptor, but the generated proto from the go_proto_library was importing google/protobuf. So I decided to change my code's imports from github.com/golang/.../descriptor to google/protobuf instead.

The problem next was that the package github.com/golang/../plugin has a plugin.CodeGeneratorRequest. That proto contains a ProtoFile field, which are of type github.com/golang/.../descriptor.FileDescriptorProto. And this type is then incompatible with google/protobuf.FileDescriptorProto.

Problem 3

I then decided to move the google/protobuf package to github.com/golang/.../descriptor. But because I use bazel and that the github.com repository is in an external/ directory I'm not supposed to modify it. So I tried to recreate the directory structure github.com/golang/..../descriptor, and move descriptor.proto and the BUILD file there.

But now, I have two different packages both producing a github.com/golang/..../descriptor.a object file. And Bazel complains about it, of course.

Solution?

I think adding descriptor.proto to this repo would actually solve this. I would then have 2 options:

  • create a go_proto_library that uses both the go_library for descriptor.pb.go and a filegroup for descriptor.proto.
  • forget about the pre-built descriptor.pb.go and only add a go_proto_library for descriptor.proto.

Any other ideas?

This may be a little contrived, and I hope my explanations are clear enough. It's relatively unlikely @Globegitter's problem is the same exact thing, but who knows...

Thanks!

@yohcop
Copy link

yohcop commented Oct 27, 2016

So to summarize, yes this was due in my case to having descriptor.pb.go in two packages, linked together, but the problem was created by this repo not having both descriptor.pb.go and descriptor.proto.

https://github.com/google/go-genproto may be a solution, but then again, for this to work this repo should not have a descriptor.pb.go either, or the output of two libraries will clash (problem 3).

I also found #131 like @Globegitter mentioned. I hope this could be resolved soon :)

@junghoahnsc
Copy link

I have that same issue. Two other third-party packages are using Descriptor in different package:

org_golang_google_genproto/protobuf/descriptor.pb.go
com_github_golang_protobuf/protoc-gen-go/descriptor/descriptor.pb.go

I think Descriptor should exist in one package. What is the best workaround for now?

@songtianyi
Copy link

Any solution here?

@jba
Copy link

jba commented Nov 14, 2016

We are working on de-duplicating the descriptor proto.

bcmills referenced this issue Dec 12, 2016
This provides a more reasonable API for obtaining a FileDescriptorProto and
DescriptorProto for a given proto.Message — a process that is currently possible
(but undocumented) using the public proto API.

The major use case for obtaining a DescriptorProto is to decode custom message
options, which are encoded as extensions on the MessageOptions submessage.
(See https://developers.google.com/protocol-buffers/docs/proto#customoptions.)

Fixes #179.
PiperOrigin-RevId: 139356528
@mickeyreiss
Copy link

@jba @bcmills Any update on this issue? I'm still seeing this occur after cf10ca0.

@jba
Copy link

jba commented Dec 15, 2016

@pongad is actively working on it.

@pongad
Copy link

pongad commented Dec 15, 2016

This problem should be resolved from genproto's side now. By agreement in here and here, descriptor is removed from genproto. The official home for it is now github.com/golang/protobuf/protoc-gen-go/descriptor.

@dsnet dsnet changed the title panic: proto: duplicate enum registered: google.protobuf.FieldDescriptorProto_Type proto: panic: duplicate enum registered: google.protobuf.FieldDescriptorProto_Type Feb 14, 2018
@dsnet
Copy link
Member

dsnet commented Mar 14, 2018

Closing this as fixed per @pongad's comment.

There are multiple ways that a registration conflict can occur:

  • Using vendor can cause the same generated proto message to be in multiple packages. These are distinct packages from Go's perspective so the init functions are run for each, even if the code is semantically identical. When init runs, it panics about duplicate registrations. The movement of go tool to versioning should alleviate this issue. See cmd/go: add package version support to Go toolchain go#24301.
  • Cloning the .proto file and generating Go messages from each can also result in this issue. It's not clear what should be done in this case since I don't fully understand all the cases why someone clones a proto file. Whether we should do something in the protobuf repo itself to alleviate this problem is a discussion for proto: remove or improve registration #268.

If you're still experiencing problems, feel free to open a new issue.

@dsnet dsnet closed this as completed Mar 14, 2018
benma added a commit to benma/go-ethereum that referenced this issue Oct 25, 2018
When some of the same messages are redefined anywhere in a Go project,
the protobuf package panics (see
golang/protobuf#178).

Since this package is internal, there is no way to work around it, as
one cannot use it directly, but also cannot define the same messages.

There is no downside in making the package accessible.
@WindGreen
Copy link

WindGreen commented Nov 13, 2018

my case:
dirA/a.proto package a
dirB/a.proto package a (should be b)
cased this problem

benma added a commit to benma/go-ethereum that referenced this issue Nov 18, 2018
When some of the same messages are redefined anywhere in a Go project,
the protobuf package panics (see
golang/protobuf#178).

Since this package is internal, there is no way to work around it, as
one cannot use it directly, but also cannot define the same messages.

There is no downside in making the package accessible.
benma added a commit to benma/go-ethereum that referenced this issue Jan 24, 2019
When some of the same messages are redefined anywhere in a Go project,
the protobuf package panics (see
golang/protobuf#178).

Since this package is internal, there is no way to work around it, as
one cannot use it directly, but also cannot define the same messages.

There is no downside in making the package accessible.
fjl pushed a commit to ethereum/go-ethereum that referenced this issue Jan 26, 2019
When some of the same messages are redefined anywhere in a Go project,
the protobuf package panics (see
golang/protobuf#178).

Since this package is internal, there is no way to work around it, as
one cannot use it directly, but also cannot define the same messages.

There is no downside in making the package accessible.
denis-papyrus pushed a commit to papyrusglobal/go-ethereum that referenced this issue Mar 12, 2019
When some of the same messages are redefined anywhere in a Go project,
the protobuf package panics (see
golang/protobuf#178).

Since this package is internal, there is no way to work around it, as
one cannot use it directly, but also cannot define the same messages.

There is no downside in making the package accessible.
@golang golang locked as resolved and limited conversation to collaborators Jun 25, 2020
@golang golang unlocked this conversation Jun 25, 2020
@golang golang locked as resolved and limited conversation to collaborators Jun 25, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests