/go

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: encoding/json: add "inline" struct tag #6213

Open
gopherbot opened this Issue Aug 22, 2013 · 27 comments

Comments

Assignees
No one assigned
Labels
Projects
None yet
Milestone
  Proposal
@gopherbot

gopherbot commented Aug 22, 2013

by bjruth:

Discussion was conceived on golang-nuts:
https://groups.google.com/forum/#!topic/golang-nuts/bAgNll-EMkI to add support for a
flag that supports unmarshaling arbitary JSON into structs. A tag was announced for the
mgo/bson package here:
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/ZeP7PaXVDQo that
transparently stores data into a separate struct field denoted by "inline"
during an unmarshal and then merges that data back into the output during marshaling.

I believe this would be a very useful feature to add into the encoding/json package and
does not introduce any breaking changes in the Go 1.x series.
@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Aug 22, 2013

Contributor

Comment 1:

I think this is worthwhile. I have an unfinished proposal in my email drafts that is
basically exactly what Gustavo describes in his bson release announcement mail.
It should be simple enough that it could be done before the 1.2 cutoff. Any disagreement?

Labels changed: added priority-soon, go1.2, feature, removed priority-triage.

Owner changed to @adg.

Status changed to Accepted.
Contributor

adg commented Aug 22, 2013

Comment 1:

I think this is worthwhile. I have an unfinished proposal in my email drafts that is
basically exactly what Gustavo describes in his bson release announcement mail.
It should be simple enough that it could be done before the 1.2 cutoff. Any disagreement?

Labels changed: added priority-soon, go1.2, feature, removed priority-triage.

Owner changed to @adg.

Status changed to Accepted.
@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Aug 28, 2013

Contributor

Comment 2:

https://golang.org/cl/13180043

Status changed to Started.
Contributor

adg commented Aug 28, 2013

Comment 2:

https://golang.org/cl/13180043

Status changed to Started.
@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Aug 29, 2013

Contributor

Comment 3:

This issue was closed by revision 466001d.

Status changed to Fixed.
Contributor

adg commented Aug 29, 2013

Comment 3:

This issue was closed by revision 466001d.

Status changed to Fixed.
@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Aug 29, 2013

Contributor

Comment 4:

Submitted the CL by mistake.

Status changed to Started.
Contributor

adg commented Aug 29, 2013

Comment 4:

Submitted the CL by mistake.

Status changed to Started.
@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Aug 29, 2013

Contributor

Comment 5:

This issue was closed by revision 27f4166.

Status changed to Fixed.
Contributor

adg commented Aug 29, 2013

Comment 5:

This issue was closed by revision 27f4166.

Status changed to Fixed.
@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Aug 29, 2013

Contributor

Comment 6:

And that was the undo CL closing it again :/
New CL https://golang.org/cl/13324045/

Status changed to Started.
Contributor

adg commented Aug 29, 2013

Comment 6:

And that was the undo CL closing it again :/
New CL https://golang.org/cl/13324045/

Status changed to Started.
@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Sep 3, 2013

Contributor

Comment 7:

Not going into 1.2.

Labels changed: removed go1.2.
Contributor

robpike commented Sep 3, 2013

Comment 7:

Not going into 1.2.

Labels changed: removed go1.2.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Sep 5, 2013

Contributor

Comment 8:

Not even sure this is worthwhile.

Status changed to Thinking.
Contributor

rsc commented Sep 5, 2013

Comment 8:

Not even sure this is worthwhile.

Status changed to Thinking.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Sep 5, 2013

Comment 9 by bjruth:

@rsc It's not that a manual approach doesn't work, but since the unmarshaler already
handles distinguishing known vs. unknown fields relative to the target interface, the
extra data can thrown into a "catch-all" field.

gopherbot commented Sep 5, 2013

Comment 9 by bjruth:

@rsc It's not that a manual approach doesn't work, but since the unmarshaler already
handles distinguishing known vs. unknown fields relative to the target interface, the
extra data can thrown into a "catch-all" field.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Nov 27, 2013

Contributor

Comment 10:

Labels changed: added go1.3maybe.
Contributor

rsc commented Nov 27, 2013

Comment 10:

Labels changed: added go1.3maybe.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Nov 27, 2013

Contributor

Comment 11:

Labels changed: removed feature.
Contributor

rsc commented Nov 27, 2013

Comment 11:

Labels changed: removed feature.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 4, 2013

Contributor

Comment 12:

Labels changed: added release-none, removed go1.3maybe.
Contributor

rsc commented Dec 4, 2013

Comment 12:

Labels changed: added release-none, removed go1.3maybe.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 4, 2013

Contributor

Comment 13:

Labels changed: added repo-main.
Contributor

rsc commented Dec 4, 2013

Comment 13:

Labels changed: added repo-main.
@lukescott

This comment has been minimized.

Show comment
Hide comment
@lukescott

lukescott Mar 1, 2014

Comment 14:

I really need this. I have a struct that has static and dynamic fields. The dynamic
fields (in a map) are chosen by the user and need to be inline with the static ones.

lukescott commented Mar 1, 2014

Comment 14:

I really need this. I have a struct that has static and dynamic fields. The dynamic
fields (in a map) are chosen by the user and need to be inline with the static ones.
@extemporalgenome

This comment has been minimized.

Show comment
Hide comment
@extemporalgenome

extemporalgenome Apr 25, 2014

Comment 15:

If this is going to be similar to <labix.org/v2/mgo/bson>, ideally an "inline" map
or struct pointer would not be allocated (the field value would remain nil) until there
is something valid to stuff in it.

extemporalgenome commented Apr 25, 2014

Comment 15:

If this is going to be similar to <labix.org/v2/mgo/bson>, ideally an "inline" map
or struct pointer would not be allocated (the field value would remain nil) until there
is something valid to stuff in it.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot May 7, 2014

Comment 16 by relistan:

+1. I was pondering writing a similar thing and came across this. Essentially we query a
3rd party API that is not versioned and changes. Catching the leftovers would let us
better get a sense of what was skipped. At a minimum it's great for debugging.

gopherbot commented May 7, 2014

Comment 16 by relistan:

+1. I was pondering writing a similar thing and came across this. Essentially we query a
3rd party API that is not versioned and changes. Catching the leftovers would let us
better get a sense of what was skipped. At a minimum it's great for debugging.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Jul 11, 2014

Comment 17 by andrew@happyinspector.com:

+1 I have something where I have a some common fields (id, path, revision, etc) and I
use an anonymous struct that is embedded in other structs. However, I can't implement a
custom json marshal on the common struct as then all the other structs will only return
the output from the marshalled common fields.

gopherbot commented Jul 11, 2014

Comment 17 by andrew@happyinspector.com:

+1 I have something where I have a some common fields (id, path, revision, etc) and I
use an anonymous struct that is embedded in other structs. However, I can't implement a
custom json marshal on the common struct as then all the other structs will only return
the output from the marshalled common fields.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Jul 11, 2014

Comment 18 by andrew@happyinspector.com:

+1 I have something where I have some common fields (id, path, revision, etc) and I use
an anonymous struct that is embedded in other structs. However, I can't implement a
custom json marshal on the common struct as then all the other structs will only return
the output from the marshalled common fields.

gopherbot commented Jul 11, 2014

Comment 18 by andrew@happyinspector.com:

+1 I have something where I have some common fields (id, path, revision, etc) and I use
an anonymous struct that is embedded in other structs. However, I can't implement a
custom json marshal on the common struct as then all the other structs will only return
the output from the marshalled common fields.

@gopherbot gopherbot added Thinking priority-soon labels Jul 11, 2014

@gopherbot gopherbot assigned adg Jul 11, 2014

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@rsc rsc removed priority-soon labels Apr 10, 2015

@rjeczalik rjeczalik referenced this issue Aug 23, 2015

Merged

Add support for inlining struct fields. #40

@cespare cespare referenced this issue Apr 16, 2016

Closed

proposal: encoding/json: reject unknown fields in Decoder #15314

@benji-bou

This comment has been minimized.

Show comment
Hide comment
@benji-bou

benji-bou Jun 18, 2016

+1

benji-bou commented Jun 18, 2016

+1
@eraclitux

This comment has been minimized.

Show comment
Hide comment
@eraclitux

eraclitux Jul 5, 2016

+1
Example use case is retrieving documents stored in MongoDB with a field like this:

Extra    map[string]float32 `bson:",inline"`

to account data that could be missing.
This enables reusing same struct defined for inserting also for reading data.

eraclitux commented Jul 5, 2016 •
edited

+1
Example use case is retrieving documents stored in MongoDB with a field like this:

Extra    map[string]float32 `bson:",inline"`

to account data that could be missing.
This enables reusing same struct defined for inserting also for reading data.

@cespare cespare referenced this issue Sep 12, 2016

Closed

encoding/json: Go's json decoder should support '*' spill over #17079

rjeczalik added a commit to rjeczalik/koding that referenced this issue Sep 18, 2016

@rjeczalik
object: add Inliner struct 
The Inliner supports merging two objects into
single one during JSON or BSON encoding/decoding.

The merge is needed in order to store Credential
and Bootstrap information as single object for
Sneaker storage. The current Sneaker / Mongo
implementation assumes they're merged into
single object.

Currently JSON does not support inlining:

  golang/go#6213

And BSON inlining does not work for out use-case:

  go-yaml/yaml#55

Thus this helper struct for inlining two values
into one.

The values are assumed to be objects - map[string]interface{}.
Verified
This commit was signed with a verified signature.
@rjeczalik rjeczalik Rafal Jeczalik
GPG key ID: 0A7BE01AB9EFB7F8 Learn about signing commits
2ce89c6

rjeczalik added a commit to rjeczalik/koding that referenced this issue Sep 18, 2016

@rjeczalik
object: add Inliner struct 
The Inliner supports merging two objects into
single one during JSON or BSON encoding/decoding.

The merge is needed in order to store Credential
and Bootstrap information as single object for
Sneaker storage. The current Sneaker / Mongo
implementation assumes they're merged into
single object.

Currently JSON does not support inlining:

  golang/go#6213

And BSON inlining does not work for out use-case:

  go-yaml/yaml#55

Thus this helper struct for inlining two values
into one.

The values are assumed to be objects - map[string]interface{}.
Verified
This commit was signed with a verified signature.
@rjeczalik rjeczalik Rafal Jeczalik
GPG key ID: 0A7BE01AB9EFB7F8 Learn about signing commits
99e79f1

rjeczalik added a commit to rjeczalik/koding that referenced this issue Sep 19, 2016

@rjeczalik
object: add Inliner struct 
The Inliner supports merging two objects into
single one during JSON or BSON encoding/decoding.

The merge is needed in order to store Credential
and Bootstrap information as a single object for
Sneaker storage. The current Sneaker / Mongo
implementation assumes they're merged into
single object.

Currently JSON does not support inlining:

  golang/go#6213

And BSON inlining does not work for our use-case:

  go-yaml/yaml#55

Thus this helper struct for inlining two values
into one.

The values are assumed to be objects - map[string]interface{}.
Verified
This commit was signed with a verified signature.
@rjeczalik rjeczalik Rafal Jeczalik
GPG key ID: 0A7BE01AB9EFB7F8 Learn about signing commits
4e32732

joeduffy added a commit to pulumi/pulumi that referenced this issue Nov 19, 2016

@joeduffy
Retain unrecognized service properties 
During unmarshaling, the default behavior of the stock Golang JSON marshaler,
and consequently the YAML one we used which mimics its behavior, is to toss away
unrecognized properties.  This isn't what we want for two reasons:

First, we want to issue errors/warnings on unrecognized fields to aid in diagnostics;
we will set aside some extensible section for 3rd parties to use.  This is not
addressed in this change, however.

Second, and more pertinent, is that we need to retain unrecognized fields for certain
types like services, which are extensible by default.

Until golang/go#6213 is addressed -- imminent, it seems -- we will have to do a
somewhat hacky workaround to this problem.  This change contains what I consider to
be the "least bad" in that we won't introduce a lot of performance overhead, and
just have to deal with the slight annoyance of the ast.Services node type containing
both Public/Private *and* PublicUntyped/PrivateUntyped fields alongside one another.
The marshaler dumps property bags into the *Untyped fields, and the parsetree analyzer
expands them out into a structured ast.Service type.  Subsequent passes can then
ignore the *Untyped fields altogether.

Note that this would cause some marshaling funkiness if we ever wanted to remarshal
the mutated ASTs back into JSON/YAML.  Since we don't do that right now, however, I've
not made any attempt to keep the two pairs in synch.  Post-parsetree analyzer, we
literally just forget about the *Untyped guys.
d9631f6

@joeduffy joeduffy referenced this issue Nov 19, 2016

Closed

Strict JSON/YAML unmarshaling #4

@yefim

This comment has been minimized.

Show comment
Hide comment
@yefim

yefim Mar 28, 2017

Are there any updates on this?

yefim commented Mar 28, 2017

Are there any updates on this?
@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Mar 28, 2017

Member

No, and I think the JSON package is pretty frozen for new features.

But I'll promote this to a proposal. This bug predates the proposal process.
Member

bradfitz commented Mar 28, 2017

No, and I think the JSON package is pretty frozen for new features.

But I'll promote this to a proposal. This bug predates the proposal process.

@bradfitz bradfitz changed the title from encoding/json: add "inline" struct tag to proposal: encoding/json: add "inline" struct tag Mar 28, 2017

@bradfitz bradfitz modified the milestones: Proposal, Unplanned Mar 28, 2017

@bradfitz bradfitz added the Proposal label Mar 28, 2017

@bradfitz bradfitz unassigned adg Mar 28, 2017

@bradfitz bradfitz removed the Thinking label Mar 28, 2017

@gitblit

This comment has been minimized.

Show comment
Hide comment
@gitblit

gitblit Mar 28, 2017

gitblit commented Mar 28, 2017

@JakeWarner

This comment has been minimized.

Show comment
Hide comment
@JakeWarner

JakeWarner Mar 31, 2017

After using mgo, I'd love to see ",inline" imported into the standard json library.

JakeWarner commented Mar 31, 2017

After using mgo, I'd love to see ",inline" imported into the standard json library.
@dolmen

This comment has been minimized.

Show comment
Hide comment
@dolmen

dolmen Apr 15, 2017

This feature would allow to catch what is called additionalProperties in JSON Schema.

I don't think however that inline is an appropriate name.

dolmen commented Apr 15, 2017

This feature would allow to catch what is called additionalProperties in JSON Schema.

I don't think however that inline is an appropriate name.
@faguirre1

This comment has been minimized.

Show comment
Hide comment
@faguirre1

faguirre1 Jun 9, 2017

ping

faguirre1 commented Jun 9, 2017

ping
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Jun 14, 2017

Contributor

Not for Go 1.9. I hope to take a look at all the open json proposals this summer and try to see if a coherent picture emerges.
Contributor

rsc commented Jun 14, 2017

Not for Go 1.9. I hope to take a look at all the open json proposals this summer and try to see if a coherent picture emerges.

@BenTheElder BenTheElder referenced this issue Nov 3, 2017

Merged

add EmtpyDir Volumes to prow kube types #5343

@morlay morlay referenced this issue Jan 4, 2018

Open

Support inline flag #160

@kargakis kargakis referenced this issue Jan 18, 2018

Merged

Use canonical meta types in prowjobs #6199

@dsnet dsnet referenced this issue Jan 19, 2018

Closed

jsonpb: Marshaler don't respect json.Marshal's specification #487

@dsnet dsnet referenced this issue Mar 21, 2018

Closed

encoding/json: Marshaler interface not producing "correct"/desired results #24466

@ianlancetaylor ianlancetaylor added the Proposal-Hold label Apr 30, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment