proposal: encoding/json: add "inline" struct tag #6213

gopherbot · 2013-08-22T02:57:29Z

by bjruth:

Discussion was conceived on golang-nuts:
https://groups.google.com/forum/#!topic/golang-nuts/bAgNll-EMkI to add support for a
flag that supports unmarshaling arbitary JSON into structs. A tag was announced for the
mgo/bson package here:
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/ZeP7PaXVDQo that
transparently stores data into a separate struct field denoted by "inline"
during an unmarshal and then merges that data back into the output during marshaling.

I believe this would be a very useful feature to add into the encoding/json package and
does not introduce any breaking changes in the Go 1.x series.

adg · 2013-08-22T03:00:48Z

Comment 1:

I think this is worthwhile. I have an unfinished proposal in my email drafts that is
basically exactly what Gustavo describes in his bson release announcement mail.
It should be simple enough that it could be done before the 1.2 cutoff. Any disagreement?

Labels changed: added priority-soon, go1.2, feature, removed priority-triage.

Owner changed to @adg.

Status changed to Accepted.

adg · 2013-08-28T03:33:59Z

Comment 2:

https://golang.org/cl/13180043

Status changed to Started.

adg · 2013-08-29T04:40:12Z

Comment 3:

This issue was closed by revision 466001d.

Status changed to Fixed.

adg · 2013-08-29T04:43:35Z

Comment 4:

Submitted the CL by mistake.

Status changed to Started.

adg · 2013-08-29T04:46:17Z

Comment 5:

This issue was closed by revision 27f4166.

Status changed to Fixed.

adg · 2013-08-29T04:47:16Z

Comment 6:

And that was the undo CL closing it again :/
New CL https://golang.org/cl/13324045/

Status changed to Started.

robpike · 2013-09-03T23:29:39Z

Comment 7:

Not going into 1.2.

Labels changed: removed go1.2.

rsc · 2013-09-05T15:59:23Z

Comment 8:

Not even sure this is worthwhile.

Status changed to Thinking.

gopherbot · 2013-09-05T16:25:13Z

Comment 9 by bjruth:

@rsc It's not that a manual approach doesn't work, but since the unmarshaler already
handles distinguishing known vs. unknown fields relative to the target interface, the
extra data can thrown into a "catch-all" field.

rsc · 2013-11-27T18:44:56Z

Comment 10:

Labels changed: added go1.3maybe.

rsc · 2013-11-27T20:29:16Z

Comment 11:

Labels changed: removed feature.

rsc · 2013-12-04T01:31:31Z

Comment 12:

Labels changed: added release-none, removed go1.3maybe.

rsc · 2013-12-04T01:50:32Z

Comment 13:

Labels changed: added repo-main.

lukescott · 2014-03-01T00:10:12Z

Comment 14:

I really need this. I have a struct that has static and dynamic fields. The dynamic
fields (in a map) are chosen by the user and need to be inline with the static ones.

extemporalgenome · 2014-04-25T17:32:17Z

Comment 15:

If this is going to be similar to <labix.org/v2/mgo/bson>, ideally an "inline" map
or struct pointer would not be allocated (the field value would remain nil) until there
is something valid to stuff in it.

gopherbot · 2014-05-07T03:56:03Z

Comment 16 by relistan:

+1. I was pondering writing a similar thing and came across this. Essentially we query a
3rd party API that is not versioned and changes. Catching the leftovers would let us
better get a sense of what was skipped. At a minimum it's great for debugging.

gopherbot · 2014-07-11T05:55:16Z

Comment 17 by andrew@happyinspector.com:

+1 I have something where I have a some common fields (id, path, revision, etc) and I
use an anonymous struct that is embedded in other structs. However, I can't implement a
custom json marshal on the common struct as then all the other structs will only return
the output from the marshalled common fields.

gopherbot · 2014-07-11T05:55:49Z

Comment 18 by andrew@happyinspector.com:

+1 I have something where I have some common fields (id, path, revision, etc) and I use
an anonymous struct that is embedded in other structs. However, I can't implement a
custom json marshal on the common struct as then all the other structs will only return
the output from the marshalled common fields.

benji-bou · 2016-06-18T14:19:31Z

+1


        object: add Inliner struct

The Inliner supports merging two objects into single one during JSON or BSON encoding/decoding. The merge is needed in order to store Credential and Bootstrap information as single object for Sneaker storage. The current Sneaker / Mongo implementation assumes they're merged into single object. Currently JSON does not support inlining: golang/go#6213 And BSON inlining does not work for out use-case: go-yaml/yaml#55 Thus this helper struct for inlining two values into one. The values are assumed to be objects - map[string]interface{}.


        object: add Inliner struct

The Inliner supports merging two objects into single one during JSON or BSON encoding/decoding. The merge is needed in order to store Credential and Bootstrap information as single object for Sneaker storage. The current Sneaker / Mongo implementation assumes they're merged into single object. Currently JSON does not support inlining: golang/go#6213 And BSON inlining does not work for out use-case: go-yaml/yaml#55 Thus this helper struct for inlining two values into one. The values are assumed to be objects - map[string]interface{}.


        object: add Inliner struct

The Inliner supports merging two objects into single one during JSON or BSON encoding/decoding. The merge is needed in order to store Credential and Bootstrap information as a single object for Sneaker storage. The current Sneaker / Mongo implementation assumes they're merged into single object. Currently JSON does not support inlining: golang/go#6213 And BSON inlining does not work for our use-case: go-yaml/yaml#55 Thus this helper struct for inlining two values into one. The values are assumed to be objects - map[string]interface{}.


        Retain unrecognized service properties

During unmarshaling, the default behavior of the stock Golang JSON marshaler, and consequently the YAML one we used which mimics its behavior, is to toss away unrecognized properties. This isn't what we want for two reasons: First, we want to issue errors/warnings on unrecognized fields to aid in diagnostics; we will set aside some extensible section for 3rd parties to use. This is not addressed in this change, however. Second, and more pertinent, is that we need to retain unrecognized fields for certain types like services, which are extensible by default. Until golang/go#6213 is addressed -- imminent, it seems -- we will have to do a somewhat hacky workaround to this problem. This change contains what I consider to be the "least bad" in that we won't introduce a lot of performance overhead, and just have to deal with the slight annoyance of the ast.Services node type containing both Public/Private *and* PublicUntyped/PrivateUntyped fields alongside one another. The marshaler dumps property bags into the *Untyped fields, and the parsetree analyzer expands them out into a structured ast.Service type. Subsequent passes can then ignore the *Untyped fields altogether. Note that this would cause some marshaling funkiness if we ever wanted to remarshal the mutated ASTs back into JSON/YAML. Since we don't do that right now, however, I've not made any attempt to keep the two pairs in synch. Post-parsetree analyzer, we literally just forget about the *Untyped guys.

yefim · 2017-03-28T18:20:35Z

Are there any updates on this?

bradfitz · 2017-03-28T19:00:16Z

No, and I think the JSON package is pretty frozen for new features.

But I'll promote this to a proposal. This bug predates the proposal process.

gitblit · 2017-03-28T19:06:18Z

Ouch, that is too bad if true.

JakeWarner · 2017-03-31T00:33:34Z

After using mgo, I'd love to see ",inline" imported into the standard json library.

dolmen · 2017-04-15T13:34:20Z

This feature would allow to catch what is called additionalProperties in JSON Schema.

I don't think however that inline is an appropriate name.

faguirre1 · 2017-06-09T23:40:45Z

ping

rsc · 2017-06-14T16:24:30Z

Not for Go 1.9. I hope to take a look at all the open json proposals this summer and try to see if a coherent picture emerges.

gelraen · 2019-04-10T12:04:29Z

ping. I'd love to see this implemented.


        Adds custom marshaling for SCIM resources

Needed until the ancient Go issue is fixed: golang/go#6213

gopherbot added Thinking priority-soon labels Jul 11, 2014

gopherbot assigned adg Jul 11, 2014

rsc added this to the Unplanned milestone Apr 10, 2015

rsc removed priority-soon labels Apr 10, 2015

rjeczalik referenced this issue Aug 23, 2015

Add support for inlining struct fields. #40

Merged

cespare referenced this issue Apr 16, 2016

proposal: encoding/json: reject unknown fields in Decoder #15314

Closed

joeduffy referenced this issue Nov 19, 2016

Strict JSON/YAML unmarshaling #4

Closed

bradfitz changed the title ~~encoding/json: add "inline" struct tag~~ proposal: encoding/json: add "inline" struct tag Mar 28, 2017

bradfitz modified the milestones: Proposal, Unplanned Mar 28, 2017

bradfitz added the Proposal label Mar 28, 2017

bradfitz unassigned adg Mar 28, 2017

bradfitz removed the Thinking label Mar 28, 2017

BenTheElder referenced this issue Nov 3, 2017

add EmtpyDir Volumes to prow kube types #5343

Merged

morlay referenced this issue Jan 4, 2018

Support inline flag #160

Open

kargakis referenced this issue Jan 18, 2018

Use canonical meta types in prowjobs #6199

Merged

dsnet referenced this issue Jan 19, 2018

jsonpb: Marshaler don't respect json.Marshal's specification #487

Closed

dsnet referenced this issue Mar 21, 2018

encoding/json: Marshaler interface not producing "correct"/desired results #24466

Closed

ianlancetaylor added the Proposal-Hold label Apr 30, 2018

atombender referenced this issue Jan 31, 2019

Catch-all for unknown fields #281

Closed

markvincze referenced this issue Mar 15, 2019

Cloud sql proxy sidecar #5

Merged

smoyer64 added a commit to PennState/scim-client that referenced this issue Apr 23, 2019

Adds custom marshaling for SCIM resources

22553d1

Needed until the ancient Go issue is fixed: golang/go#6213

smoyer64 referenced this issue Jun 10, 2019

Explicitly allow extra keys in "component details" #60

Merged

edwardstudy referenced this issue Jun 30, 2019

Support Kubernetes Affinity #364

Closed

golang/go

Join GitHub today

proposal: encoding/json: add "inline" struct tag #6213

Comments

gopherbot commented Aug 22, 2013

This comment has been minimized.

adg commented Aug 22, 2013

This comment has been minimized.

adg commented Aug 28, 2013

This comment has been minimized.

adg commented Aug 29, 2013

This comment has been minimized.

adg commented Aug 29, 2013

This comment has been minimized.

adg commented Aug 29, 2013

This comment has been minimized.

adg commented Aug 29, 2013

This comment has been minimized.

robpike commented Sep 3, 2013

This comment has been minimized.

rsc commented Sep 5, 2013

This comment has been minimized.

gopherbot commented Sep 5, 2013

This comment has been minimized.

rsc commented Nov 27, 2013

This comment has been minimized.

rsc commented Nov 27, 2013

This comment has been minimized.

rsc commented Dec 4, 2013

This comment has been minimized.

rsc commented Dec 4, 2013

This comment has been minimized.

lukescott commented Mar 1, 2014

This comment has been minimized.

extemporalgenome commented Apr 25, 2014

This comment has been minimized.

gopherbot commented May 7, 2014

This comment has been minimized.

gopherbot commented Jul 11, 2014

This comment has been minimized.

gopherbot commented Jul 11, 2014

This comment has been minimized.

benji-bou commented Jun 18, 2016

This comment has been minimized.

yefim commented Mar 28, 2017

This comment has been minimized.

bradfitz commented Mar 28, 2017

This comment has been minimized.

gitblit commented Mar 28, 2017

This comment has been minimized.

JakeWarner commented Mar 31, 2017

This comment has been minimized.

dolmen commented Apr 15, 2017

This comment was marked as off-topic.

faguirre1 commented Jun 9, 2017

This comment has been minimized.

rsc commented Jun 14, 2017

This comment was marked as off-topic.

gelraen commented Apr 10, 2019