Failing to parse a valid descriptor generated for message with a few enums

[jkff]

Encountered this issue while working on apache/beam#4672 . I haven't had time to create a minimal self-contained reproducible example, but the gist of the issue is as follows.

I have a message that looks like this:

message Foo {
  enum Bar1 { ... }
  enum Bar2 { ... }
}

Accessing Foo from code results in an error parsing the Foo generated descriptor.

Adding a bogus field to Foo solves the problem:

message Foo {
  ...
  int32 ignored = 1;
}

Below is a paste of the error I got with the actual message (StandardPTransforms in the PR linked above, in model/pipeline/src/main/proto/beam_runner_api.proto)

java.lang.RuntimeException: Failed to construct instance from factory method DirectRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
	at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
	at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
	at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:305)
	at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:346)
	at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:328)
	at org.apache.beam.sdk.extensions.sorter.SortValuesTest.testSecondaryKeySorting(SortValuesTest.java:73)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy1.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:108)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:146)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:128)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
	... 54 more
Caused by: java.lang.ExceptionInInitializerError
	at org.apache.beam.repackaged.beam_runners_direct_java.model.pipeline.v1.RunnerApi$StandardPTransforms.getDescriptor(RunnerApi.java:8878)
	at org.apache.beam.repackaged.beam_runners_direct_java.model.pipeline.v1.RunnerApi$StandardPTransforms$Primitives.getDescriptor(RunnerApi.java:9075)
	at org.apache.beam.repackaged.beam_runners_direct_java.model.pipeline.v1.RunnerApi$StandardPTransforms$Primitives.getValueDescriptor(RunnerApi.java:9067)
	at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.BeamUrns.getUrn(BeamUrns.java:27)
	at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.PTransformTranslation.<clinit>(PTransformTranslation.java:57)
	at org.apache.beam.runners.direct.DirectRunner$Enforcement.<clinit>(DirectRunner.java:82)
	at org.apache.beam.runners.direct.DirectRunner.<init>(DirectRunner.java:142)
	at org.apache.beam.runners.direct.DirectRunner.fromOptions(DirectRunner.java:137)
	... 59 more
Caused by: java.lang.IllegalArgumentException: Failed to parse protocol buffer descriptor for generated code.
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom(Descriptors.java:340)
	at org.apache.beam.repackaged.beam_runners_direct_java.model.pipeline.v1.RunnerApi.<clinit>(RunnerApi.java:56396)
	... 67 more
Caused by: org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:106)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.CodedInputStream$ArrayDecoder.checkLastTagWas(CodedInputStream.java:559)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:817)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.DescriptorProtos$DescriptorProto.<init>(DescriptorProtos.java:4966)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.DescriptorProtos$DescriptorProto.<init>(DescriptorProtos.java:4897)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.DescriptorProtos$DescriptorProto$1.parsePartialFrom(DescriptorProtos.java:9474)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.DescriptorProtos$DescriptorProto$1.parsePartialFrom(DescriptorProtos.java:9469)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:816)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.DescriptorProtos$FileDescriptorProto.<init>(DescriptorProtos.java:1200)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.DescriptorProtos$FileDescriptorProto.<init>(DescriptorProtos.java:1124)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.DescriptorProtos$FileDescriptorProto$1.parsePartialFrom(DescriptorProtos.java:4633)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.DescriptorProtos$FileDescriptorProto$1.parsePartialFrom(DescriptorProtos.java:4628)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:139)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:173)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:185)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:190)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.DescriptorProtos$FileDescriptorProto.parseFrom(DescriptorProtos.java:2132)
	at org.apache.beam.repackaged.beam_runners_direct_java.com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom(Descriptors.java:338)
	... 68 more

[jkff]

Now I added another enum to that same proto, and now I'm getting the same InvalidProtocolBufferException but it disappears if I remove the "ignored" variable.

What's more weird, I took the descriptor proto literal from the generated code and dumped it to disk, and protoc --decode_raw is able to decode it. Programmatically decoding it from other places in my project also works. I wonder if this has something to do with shading - is there much reflection magic going on?

At the least, more informative error messages would help ("Protocol message end-group tag did not match expected tag." gives no information as to where this is happening in the stream, or even what is the expected and actual tag).

[acozzette]

@jkff Sorry for taking such a long time to reply to this--are you still running into this problem? I wonder if it's potentially caused by a mismatch between the protoc version and the runtime, because the runtime has to be at least as new as the protoc version used for generating code.

[jkff]

I've run into this problem both times when I tried to modify this proto, but both times was able to work around it (in first case adding a bogus field, second removing it back). I haven't run into it since then, but something is clearly quite broken here.

If there's currently too little information to provide a real fix, a good first step would be to improve the proto parsing error messages (like "Protocol message end-group tag did not match expected tag.") so that next time someone hits this they are able to provide more information.

[elharo]

Doesn't seem to have bothered anyone for multiple releases and I haven't been able to reproduce this so I'm going to close this. Not sure what was happening here but it seems to have been fixed sometime in the last several years. If it reoccurs or there's a full reproduction available, please comment and I'll reopen and take another look.