proposal: encoding/json: add "inline" struct tag

[gopherbot]

by bjruth:

Discussion was conceived on golang-nuts:
https://groups.google.com/forum/#!topic/golang-nuts/bAgNll-EMkI to add support for a
flag that supports unmarshaling arbitary JSON into structs. A tag was announced for the
mgo/bson package here:
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/ZeP7PaXVDQo that
transparently stores data into a separate struct field denoted by "inline"
during an unmarshal and then merges that data back into the output during marshaling.

I believe this would be a very useful feature to add into the encoding/json package and
does not introduce any breaking changes in the Go 1.x series.

Edited by @dsnet on 2020-11-10 to fix formatting.

[adg]

Comment 1:

I think this is worthwhile. I have an unfinished proposal in my email drafts that is
basically exactly what Gustavo describes in his bson release announcement mail.
It should be simple enough that it could be done before the 1.2 cutoff. Any disagreement?

Labels changed: added priority-soon, go1.2, feature, removed priority-triage.

Owner changed to @adg.

Status changed to Accepted.

[adg]

Comment 2:

https://golang.org/cl/13180043

Status changed to Started.

[adg]

Comment 3:

This issue was closed by revision 466001d.

Status changed to Fixed.

[adg]

Comment 4:

Submitted the CL by mistake.

Status changed to Started.

[adg]

Comment 5:

This issue was closed by revision 27f4166.

Status changed to Fixed.

[adg]

Comment 6:

And that was the undo CL closing it again :/
New CL https://golang.org/cl/13324045/

Status changed to Started.

[robpike]

Comment 7:

Not going into 1.2.

Labels changed: removed go1.2.

[rsc]

Comment 8:

Not even sure this is worthwhile.

Status changed to Thinking.

[gopherbot]

Comment 9 by bjruth:

@rsc It's not that a manual approach doesn't work, but since the unmarshaler already
handles distinguishing known vs. unknown fields relative to the target interface, the
extra data can thrown into a "catch-all" field.

[rsc]

Comment 10:

Labels changed: added go1.3maybe.

[rsc]

Comment 11:

Labels changed: removed feature.

[rsc]

Comment 12:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 13:

Labels changed: added repo-main.

[lukescott]

Comment 14:

I really need this. I have a struct that has static and dynamic fields. The dynamic
fields (in a map) are chosen by the user and need to be inline with the static ones.

[extemporalgenome]

Comment 15:

If this is going to be similar to <labix.org/v2/mgo/bson>, ideally an "inline" map
or struct pointer would not be allocated (the field value would remain nil) until there
is something valid to stuff in it.

[gopherbot]

Comment 16 by relistan:

+1. I was pondering writing a similar thing and came across this. Essentially we query a
3rd party API that is not versioned and changes. Catching the leftovers would let us
better get a sense of what was skipped. At a minimum it's great for debugging.

[gopherbot]

Comment 17 by andrew@happyinspector.com:

+1 I have something where I have a some common fields (id, path, revision, etc) and I
use an anonymous struct that is embedded in other structs. However, I can't implement a
custom json marshal on the common struct as then all the other structs will only return
the output from the marshalled common fields.

[gopherbot]

Comment 18 by andrew@happyinspector.com:

+1 I have something where I have some common fields (id, path, revision, etc) and I use
an anonymous struct that is embedded in other structs. However, I can't implement a
custom json marshal on the common struct as then all the other structs will only return
the output from the marshalled common fields.

[benji-bou]

+1

[tooolbox]

I have something where I have a some common fields (id, path, revision, etc) and I
use an anonymous struct that is embedded in other structs. However, I can't implement a
custom json marshal on the common struct as then all the other structs will only return
the output from the marshalled common fields.

Just ran into this.

The use of anonymous structs for inlining is excellent up until your anonymous struct has a MarshalJSON method.

[mitar]

In #42417 I proposed to have the following syntax:

type Message struct {
  Name  string                 `json:"name"`
  Type  string                 `json:"type"`
  Skip  string                 `json:"-"`
  Extra map[string]interface{} `json:"+"`
}

So + would mark a field into which all extra/unexpected fields would go.

[dsnet]

+ is an interesting idea, but the + character is already a valid name, so we cannot use that in a backwards compatible way.

[smoyer64]

The - character is also a valid name but the encoding/json package lets you use it by adding the json:"-," value. See https://pkg.go.dev/encoding/json#Marshal.

[dsnet]

@smoyer64, that's doesn't fix the issue I mentioned:

• json:- means this field is ignored
• json:-, means this field is named "-"

However, today:

• json:+ means this field is named "+"; and we can't redefine the meaning of this
• json:+, means this field is named "+"

[fdcds]

encoding/xml supports this via the xml:",any" tag:

• If the XML element contains a sub-element that hasn't matched any
  of the above rules and the struct has a field with tag ",any",
  unmarshal maps the sub-element to that struct field.

(https://golang.org/pkg/encoding/xml/#Unmarshal)

Should the same mechanism be used for encoding/json?