types/known: functions for working with Struct

[ptone]

The helper files for some of the ptype proto types such as Duration and Timestamp are useful.

It would be great to have a ptypes.Struct <-> map[string]interface{} converter helper - as this would seem to me to be the closest analog native go representation.

[ptone]

Something along the lines of this as a start:

ptone@7ee10a8

would this be useful?

[cybrcodr]

Just my opinion. With map[string]interface{}, user will still need to type assert on the map value. That's pretty similar to accessing ptypes.Struct.Fields which is of type map[string]*Value, i.e. need to type assert to one of *Value_X. There is also the advantage of having the *Value getter methods if using ptypes.Struct.Fields.

[ptone]

There are plenty of 3rd party libraries that work with a type of map[string]interface{} and handle that reflection and assertion in their implementation. These will not work with the struct ptype, and so this adds more burden on users to always handle that, hence the suggestion for an optional utility function.

[cybrcodr]

Fair enough. I'm not convinced though that this should belong here, partly being cautious about expanding the API surface. I'm going to close this for now, but if you have a really good reason and feel strongly that this should be part of this package, feel free to reopen. Thanks.

[ptone]

Just to leave some more notes on this issue:

Python (with weaker typing) supports this type of convenience:
https://developers.google.com/protocol-buffers/docs/reference/python-generated#struct

And you can achieve the intended result (very inefficiently) by marshaling through jsonpb to json, and then unmarshal back to map[string]interface{}

[jsmouret]

Leaving more notes :)

A helper to convert map[string]interface{} to google.protobuf.Struct:
https://gist.github.com/jsmouret/2bc876e8def6c63410556350eca3e43d

[xuyang2]

I found a ptypes.Struct <-> map[string]interface{} converter in GoogleCloudPlatform/google-cloud-go:

func DecodeToMap(s *pb.Struct) map[string]interface{}

[catkins]

It's a shame this has been closed - having some helpers to/from pb.Struct would make these types much easier to work with.

[awfm9]

It makes no sense not to have these helpers.

[hiromis]

We recently came across a need for something similar and this is what we came up with:
https://play.golang.org/p/-_o7hLUSWhV

Not the most performant way but we chose simplicity over performance.

[neild]

Reopening to take a second look when we consider ptypes in google.golang.org/protobuf.

[puellanivis]

Directing people towards or expecting people to use marshal/unmarshal through encoding/json is probably a bad idea, because package json will sorts the keys on a map[string]interface{} before emitting them. I’ve seen a server spin on max CPU for a couple minutes because it kept going through marshal/unmarshal rather than porting values directly.

I think we could easily do a better job in this package. And if we make it convenient enough, then people are far less likely to fall into one of the many gaping pitfalls on performance or memory just trying to get a solution out the door.

[cstrahan]

Directing people towards or expecting people to use marshal/unmarshal through encoding/json is probably a bad idea, because package json will sorts the keys on a map[string]interface{} before emitting them.

In principle, I agree... but in practice, one must currently decide between maintaining a (intricate, and possibly buggy) implementation via reflection, or going the sure-fire way of round tripping through JSON (de)serialization.

As you suggest, I think this would be best solved by adding direct support in the protobuf lib, that way everyone achieves better performance, and the logic has more eyeballs on it (to prevent subtle edge-case bugs).

As a final note, I'll mention that even a project as large and successful as Envoy still uses JSON round-tripping:

https://github.com/envoyproxy/go-control-plane/blob/b0ba75c9c5574133fecebe557aa8d126e69f5ca7/pkg/conversion/struct.go#L27-L60

[puellanivis]

This thread is not about providing an arbitrary proto.Message to map[string]interface{}, but rather strictly a structpb.Struct to map[string]interface{}, and back. This should not require anything more than a type switch.

“But what if they’re using a named type of concrete type XY? That won’t work in a type switch.” We should not support that. We should solely and narrowly support only what structpb.Struct supports, and no more.

[dsnet]

Fixed by https://golang.org/cl/225298. It will be available in the next release of the google.golang.org/protobuf module, to be released within a week or so.