jsonpb: capture raw JSON of unknown google.protobuf.Any messages

[creachadair]

When decoding protobuf messages from JSON, the decoder currently requires the generated code for all of the message types referenced from any field of type [repeated] google.protobuf.Any be linked into the binary.

We (Kythe) have tools that receive JSON-encoded protobuf messages as input, decode them, examine some of their fields, and then do other work based on what those fields contain. Schematically, we have a message like this:

message M {
   Config info = 1;
   repeated google.protobuf.Any details = 2;
}

This arrives as JSON, is decoded, and then passed along based on the contents of info.

The tool links M and Config already, since those are common to all uses. Unfortunately, if the provider of the message adds an unknown message to the details field whose decoder isn't linked, the decoding step currently fails. This makes it difficult to use Any messages forward-compatibly.

Ideally, we'd like the ability to treat Any messages similarly to how the encoding/json package handles json.RawMessage, viz., that they could be decoded to an intermediate type that preserves the encoded format until explicitly decoded.

One sensible implementation could be to add an option to jsonpb.Unmarshaler, that when set would make the decoding of google.protobuf.Any messages return an instance of a library type (perhaps: jsonpb.RawAny?) that

• implements proto.Message,
• encapsulates the un-decoded Any message (as JSON), and
• completes the decoding when explicitly unmarshalled (requiring the type to be linked)

There are probably other solutions, but this would I think solve our issues without requiring breaking changes to the API.

We could probably work around this by introducing our own work-alike equivalent of "Any". However, given that Any is designed as an extension point, it would be nice if the decoder could have control over this without changing the message structure explicitly.

[dsnet]

One sensible implementation could be to add an option to jsonpb.Unmarshaler, that when set would make the decoding of google.protobuf.Any messages return an instance of a library type (perhaps: jsonpb.RawAny?) that

Where would jsonpb.RawAny be stored? Are you assuming that it is part of the generated data structure for the Any message? Keep in mind that generated messages contain a concrete pointer to the Any struct, so you can't just replace the underlying type.

Also, implements proto.Message

We are changing the proto.Message interface to be more behaviorally complete (see #364). jsonpb.RawAny would need to be able to implement the interface with sensible protobuf reflection information.

Lastly, this general issue isn't specific to Any. There are similar issues when decoding enum value names that we don't know about or extensions that we don't know about. It seems to me that a solution for Any should be generalizable to the overall issue of what to do when protobuf type information is missing during unmarshal. Also, such a solution should apply to the protobuf text format as well.

[dsnet]

Also, this is a problem that exists in other languages. Do other language implementations have an API for capturing or dealing with raw text or JSON when lacking protobuf type information?

[creachadair]

Also, this is a problem that exists in other languages. Do other language implementations have an API for capturing or dealing with raw text or JSON when lacking protobuf type information?

This problem does exist in other languages as well. The only other languages I use regularly are C++ and Java, both of which have the same problem (at least the plugins I've used—there are multiple plugins for both Java and C++). None of the generators I have seen so far support the requested functionality I described above.

I think it has to be dealt with separately by each code generator language anyway, as it's not part of the protobuf compiler API. For our purposes, Go is the most important case, since most of Kythe's key plumbing is written in Go.

[creachadair]

Where would jsonpb.RawAny be stored? Are you assuming that it is part of the generated data structure for the Any message?

Right now, assuming the target message is linked, the decoder converts the JSON into an Any message, whose content (value) is the wire format of the target message. I was imagining that in this scenario, that message would be the "raw" placeholder, still in wire format. I don't claim that's necessarily the best option, but it seems like it would fit the paradigm reasonably well. I assumed the RawAny message would look something like this:

message RawAny {
   string type_url = 1;  // the type URL of the original unknown message whose JSON is below
   string json = 2;  // JSON requires UTF-8 encoding
}

When the decoder gets an unrecognized message, the Any message it produces would be something like

type_url: "type.googleapis.com/jsonpb.RawAny"
content: <wire-format-of-the-corresponding-RawAny-message>

or words to that effect.

[dsnet]

That seems like a reasonable way to preserve raw JSON, and the idea is extensible to the text format as well. I think this should be a proposal filed against the mainline protobuf project as that solution is not specific to Go.

[creachadair]

I think this should be a proposal filed against the mainline protobuf project as that solution is not specific to Go.

I filed protocolbuffers/protobuf#5226 upstream. If it's adopted, there will still be Go-specific changes, though.

[dsnet]

Great. I'll put this issue on hold for now.

If it's adopted, there will still be Go-specific changes, though.

That's fine. This repository's goal is more so to implement the feature set of the mainline protobuf project, rather than to trail-blaze its own features. If the protobuf team deems this an important feature to provide, we'll implement it.

[dsnet]

I'm going to close this. If and only if protocolbuffers/protobuf#5226 occurs, then we'll implement this.