APIv2: types/dynamicpb: add support for dynamic messages

[mwitkow]

Since gRPC now has support for server-side reflection (https://github.com/grpc/grpc-go/tree/master/reflection, basically stubby ls) it would be able to process dynamic messages in Go.

I'd be interested in adding this support either in protobuf or in an outside, but would probably need some restructuring in protobuf itself: e.g. allowing to use the same encoding/decoding methods but with the schema driven from a FileDescriptorProto instead of Golang tags.

[awalterschulze]

I like using https://github.com/katydid/katydid/blob/master/parser/proto/proto.go#L121
Here are some usage docs http://katydid.github.io/parser/parserusage.html

[awalterschulze]

I like using https://github.com/katydid/katydid/blob/master/parser/proto/proto.go#L121
Here are some usage docs http://katydid.github.io/parser/parserusage.html

[abe-winter]

I have a proof-of-concept of this working in a fork
https://github.com/abe-winter/dynproto

[abe-winter]

I have a proof-of-concept of this working in a fork
https://github.com/abe-winter/dynproto

[Merovius]

@abe-winter Awesome :) IIt would be great if it wouldn't depend on cgo, though. Is there a reason you didn't use descriptors in proto-encoding?

[Merovius]

@abe-winter Awesome :) IIt would be great if it wouldn't depend on cgo, though. Is there a reason you didn't use descriptors in proto-encoding?

[abe-winter]

@Merovius the output of ParseSchema is a FileDescriptorProto, and all the downstream code uses the proto. My use case requires schema parsing. An alternative is to shell out to the protoc program, but then protoc needs to be installed everywhere.

Would it make sense to set a // +build cgo on the parsing code? I'm not super-familiar with golang, but I think that makes cgo a soft dependency.

[abe-winter]

@Merovius the output of ParseSchema is a FileDescriptorProto, and all the downstream code uses the proto. My use case requires schema parsing. An alternative is to shell out to the protoc program, but then protoc needs to be installed everywhere.

Would it make sense to set a // +build cgo on the parsing code? I'm not super-familiar with golang, but I think that makes cgo a soft dependency.

[abe-winter]

☝️ I was wrong above -- I'm already relying on libprotobuf being installed, might as well rely on protoc.

I'll move schema parsing to its own package so it's truly optional.

[abe-winter]

☝️ I was wrong above -- I'm already relying on libprotobuf being installed, might as well rely on protoc.

I'll move schema parsing to its own package so it's truly optional.

[Merovius]

Great, I think it's a good idea to have the schema-parsing in a separate package.

Another change I would suggest, is to add a DynamicMessage (or similar) struct which implements proto.Message statically and embeds/contains a *reflect.Type. DynStruct could then return a []*DynamicMessage, instead of a []*reflect.Type. The advantage would be, that a) Marshal/Unmarshal could remain type-safe and take a proto.Message and b) packages which wrap proto in a message-type agnostic way (grpc reflection comes to mind), could also use the dynamic message without breaking their type-safety.

[Merovius]

Great, I think it's a good idea to have the schema-parsing in a separate package.

Another change I would suggest, is to add a DynamicMessage (or similar) struct which implements proto.Message statically and embeds/contains a *reflect.Type. DynStruct could then return a []*DynamicMessage, instead of a []*reflect.Type. The advantage would be, that a) Marshal/Unmarshal could remain type-safe and take a proto.Message and b) packages which wrap proto in a message-type agnostic way (grpc reflection comes to mind), could also use the dynamic message without breaking their type-safety.

[abe-winter]

Hmm. You have two points here:

• grpc and other codebases use proto.Message internally and so DynStruct should support the Message interface.
  • This sounds right but I'm worried about making this change without a testing plan. I don't know what grpc would do with dynamic messages and what the requirements would be in practice.
• Buffer.Marshal/Unmarshal gain a safety advantage from taking Message instead of empty interface{}.
  • This sounds wrong to me. Nothing inside those functions calls the Message methods. We can implement Message and Marshaler for your DynamicMessage struct, but this would just be copying around parts of Marshal/Unmarshal. Not sure there's any advantage.

[abe-winter]

Hmm. You have two points here:

• grpc and other codebases use proto.Message internally and so DynStruct should support the Message interface.
  • This sounds right but I'm worried about making this change without a testing plan. I don't know what grpc would do with dynamic messages and what the requirements would be in practice.
• Buffer.Marshal/Unmarshal gain a safety advantage from taking Message instead of empty interface{}.
  • This sounds wrong to me. Nothing inside those functions calls the Message methods. We can implement Message and Marshaler for your DynamicMessage struct, but this would just be copying around parts of Marshal/Unmarshal. Not sure there's any advantage.

[jhump]

I've been working on something related to this, to be able to use the service reflection to build clients. I haven't gotten to a point of implementing the equivalent of DynamicMessage, but I do have better descriptors (like protobuf provides in Java and C++ runtimes) as well as a better client interface for the reflection service. I think the descriptors would be a suitable base for pure-Go implementation of a dynamic message.
https://github.com/jhump/protoreflect

[jhump]

I've been working on something related to this, to be able to use the service reflection to build clients. I haven't gotten to a point of implementing the equivalent of DynamicMessage, but I do have better descriptors (like protobuf provides in Java and C++ runtimes) as well as a better client interface for the reflection service. I think the descriptors would be a suitable base for pure-Go implementation of a dynamic message.
https://github.com/jhump/protoreflect

[jhump]

FWIW, I have an all-Go implementation of dynamic messages:
https://godoc.org/github.com/jhump/protoreflect/dynamic
And there's also a dynamic RPC stub, that invokes RPCs using method descriptors and dynamic messages:
https://godoc.org/github.com/jhump/protoreflect/dynamic/grpcdynamic

It's not 100% yet -- mainly missing useful support for proto3 well-known-types and knobs for better controlling JSON output (for example). I'd say it's currently beta-quality right now -- not a lot of miles on it other than the tests, and there are several TODOs for more tests (though a lot of the code does get exercised by the tests that are present).

The *dynamic.Message type does implement proto.Message and it also has a Descriptor method that is compatible with that of generated messages. It implements proto.Marshaler, proto.Unmarshaler, encoding.TextMarshaler, and encoding.TextUnmarshaler so it can also be used with functions in the proto package for serializing and de-serializing.

It can't yet be used with the jsonpb package. I've opened a pull request that, if merged, would remedy that (#325).

Of course, APIs that accept a proto.Message which in turn use proto.GetProperties, proto.MessageName, or any of the extension-related messages in the proto package will not work with this new type. Those methods all assume the object not only implements the interface, but that it also is a registered generated type.

[jhump]

FWIW, I have an all-Go implementation of dynamic messages:
https://godoc.org/github.com/jhump/protoreflect/dynamic
And there's also a dynamic RPC stub, that invokes RPCs using method descriptors and dynamic messages:
https://godoc.org/github.com/jhump/protoreflect/dynamic/grpcdynamic

It's not 100% yet -- mainly missing useful support for proto3 well-known-types and knobs for better controlling JSON output (for example). I'd say it's currently beta-quality right now -- not a lot of miles on it other than the tests, and there are several TODOs for more tests (though a lot of the code does get exercised by the tests that are present).

The *dynamic.Message type does implement proto.Message and it also has a Descriptor method that is compatible with that of generated messages. It implements proto.Marshaler, proto.Unmarshaler, encoding.TextMarshaler, and encoding.TextUnmarshaler so it can also be used with functions in the proto package for serializing and de-serializing.

It can't yet be used with the jsonpb package. I've opened a pull request that, if merged, would remedy that (#325).

Of course, APIs that accept a proto.Message which in turn use proto.GetProperties, proto.MessageName, or any of the extension-related messages in the proto package will not work with this new type. Those methods all assume the object not only implements the interface, but that it also is a registered generated type.

[dsnet]

Placing on hold until #364 is complete.

[dsnet]

Placing on hold until #364 is complete.

[dsnet]

Work on #364 is in full swing and support for dynamic messages is planned.

[dsnet]

Work on #364 is in full swing and support for dynamic messages is planned.