Handling protobuf 'well-known types'

[zacharyvoase]

I wanted to open a discussion about how to deal with 'well-known types' in protobuf, especially as they relate to JSON encoding. There's a couple of other pre-requisites that have to be figured out first.

Context

The protobuf definition includes some types, referred to in the official docs as 'well-known types', used for representing things like generic values and objects. A good example is google.protobuf.Struct, which is something like a Record<string, any>:

message SomeMessage {
  google.protobuf.Struct attrs = 1;
}

Naïvely, this would actually be represented as a protobuf object that looks like this, using google.protobuf.Value (which is itself a big oneof able to represent a number of different scalar types):

SomeMessage {
  attrs {
    fields {
      "someKey": Value { numberValue: 123 }
      "anotherKey": Value { stringValue: "Hello, World!" }
    }
  }
}

But the spec (and the first-party implementations) say this should serialize to JSON as:

{
  "someKey": 123,
  "anotherKey": "Hello, World!"
}

I worked on the --ts_opt=json_names feature, which aims to bring the output of .toObject() (and input to .fromObject()) calls in line with the native JSON serialization of protobufs, and this is relevant to me because I think interoperability with the official JSON representation is important (especially for a language like JS/TS!)

What I've Tried So Far

I wanted to start by creating a conformance test that uses just google.protobuf.Struct and google.protobuf.Value at first, but ran into some issues with the Bazel rules (issues that don't happen when just invoking protoc simply on the command-line). You can replicate the issues I'm seeing by just writing a simple proto file like the above SomeMessage example, with build rules like this:

proto_library(
    name = "some_message",
    srcs = ["some_message.proto"],
    deps = ["@com_google_protobuf//:struct_proto"],
)

ts_proto_library(
    name = "some_message_ts",
    deps = [
        ":some_message",
        "@com_google_protobuf//:struct_proto"
    ],
)

I'm not super familiar with Bazel rule implementation, so I wouldn't mind some help figuring this out. The next step would be adding some special-casing in the compiler for google.protobuf.Struct and google.protobuf.Value, maybe just in the fromObject()/toObject() method implementations, guarded behind a flag. But due to the Bazel issues I haven't even gotten to that yet.

[jackpunt]

Zack, have you made progress getting bazel to work for you?
(I also consider contributing to this project, but have zero experience with bazel, so have not much tried to clone/build)
If you have pointers to how to get things up & running lmk. (& apologies for being off-topic)

[tomasweigenast]

Don't forget about google.protobuf.Any!
It should allow .pack() and .unpack() methods!
Here is my related issue: #156

Another thing that should be implemented is metadata of generated files, for example, we need to know the type's name or its TypeUrl

[chrisplusplus]

Did you ever figure this one out or was it ever implemented?

[tomasweigenast]

Any update on this?

[thesayyn]

This is fixed with the rust rewrite. Now we support well known types!