Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: encoding/json, encoding/xml: support zero values of structs with omitempty #11939

Open
joeshaw opened this issue Jul 30, 2015 · 86 comments · Fixed by kubernetes/test-infra#12414

Comments

@joeshaw
Copy link
Contributor

joeshaw commented Jul 30, 2015

Support zero values of structs with omitempty in encoding/json and encoding/xml.

This bites people a lot, especially with time.Time. Open bugs include #4357 (which has many dups) and #10648. There may be others.

Proposal

Check for zero struct values by adding an additional case to the isEmptyValue function:

case reflect.Struct:
        return reflect.Zero(v.Type()).Interface() == v.Interface()

This will solve the vast majority of cases.

(Optional) Introduce a new encoding.IsZeroer interface, and use this to check for emptiness:

Update: I am dropping this part of the proposal, see below.

type IsZeroer interface {
        IsZero() bool
}

Visit this playground link and note that the unmarshaled time.Time value does not have a nil Location field. This prevents the reflection-based emptiness check from working. IsZero() already exists on time.Time, has the correct semantics, and has been adopted as a convention by Go code outside the standard library.

An additional check can be added to the isEmptyValue() functions before checking the value's Kind:

if z, ok := v.Interface().(encoding.IsZeroer); ok {
        return z.IsZero()
}

Compatibility

The encoding.IsZeroer interface could introduce issues with existing non-struct types that may have implemented IsZero() without consideration of omitempty. If this is undesirable, the encoding.IsZeroer interface check could be moved only within the struct case:

case reflect.Struct:
        val := v.Interface()
        if z, ok := val.(encoding.IsZeroer); ok {
                return z.IsZero()
        }
        return reflect.Zero(v.Type()).Interface() == val

Otherwise, this change is backward-compatible with existing valid uses of omitempty. Users who have applied omitempty to struct fields incorrectly will get their originally intended behavior for free.

Implementation

I (@joeshaw) have implemented and tested this change locally, and will send the CL when the Go 1.6 tree opens.

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Jul 30, 2015
@gopherbot
Copy link

gopherbot commented Aug 25, 2015

CL https://golang.org/cl/13914 mentions this issue.

@gopherbot
Copy link

gopherbot commented Aug 28, 2015

CL https://golang.org/cl/13977 mentions this issue.

@joeshaw
Copy link
Contributor Author

joeshaw commented Sep 18, 2015

The empty struct approach is implemented in CL 13914 and the IsZeroer interface is implemented in CL 13977.

In order for them to be reviewable separately they conflict a bit -- mostly in the documentation -- but I will fix for one if the other is merged.

@adg adg added Proposal and removed Proposal labels Sep 25, 2015
@joeshaw
Copy link
Contributor Author

joeshaw commented Oct 19, 2015

In the CLs @rsc said,

I'd really like to stop adding to these packages. I think we need to leave well enough alone at some point.

I see what he's getting at. CL 13977, which implements the IsZeroer interface is clearly an enhancement and adds API to the standard library that needs to be maintained forever. So, I am abandoning that CL and that part of the proposal.

However, I still feel strongly about omitempty with empty structs, and I want to push for CL 13914 to land for Go 1.6.

I use the JSON encoding in Go a lot, as my work is mostly writing services that communicate with other services, in multiple languages, over HTTP. The fact that structs don't obey omitempty is a frequent source of confusion (see #4357 and its many dups and references, and #10648) and working around it is really annoying. Other programming languages do not conform to Go's ideal "zero value" idea, and as a result encoding a zero value is semantically different in JSON than omitting it or encoding it as null. People run into this most commonly with time.Time. (There is also the issue that decoding a zero time.Time does not result in an empty struct, see #4357 (comment) for background on that.)

I think it should be considered a bug that Go does not support omitempty for these types, and although it adds a small amount of additional code, it fixes a bug.

@rsc rsc modified the milestones: Proposal, Unplanned Oct 24, 2015
@rsc rsc changed the title proposal: encoding: Support zero values of structs with omitempty in encoding/json and encoding/xml proposal: encoding/json, encoding/xml: support zero values of structs with omitempty Oct 24, 2015
@jeromenerf
Copy link

jeromenerf commented Mar 27, 2016

This proposal is marked as unplanned, yet the related bug report #10648 is marked as go1.7.
Is it still being worked /thought on?

@rsc
Copy link
Contributor

rsc commented Mar 28, 2016

To my knowledge, it is not being worked on. Honestly this seems fairly low
priority and will likely miss Go 1.7.

On Sun, Mar 27, 2016 at 12:22 PM Jérôme Andrieux notifications@github.com
wrote:

This proposal is marked as unplanned, yet the related bug report #10648
#10648 is marked as go1.7.
Is it still being worked /thought on?


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#11939 (comment)

@jeromenerf
Copy link

jeromenerf commented Mar 28, 2016

OK.

This is more of a convenience than a priority indeed.

It can be a pain point when dealing with libs that don't support "embedded structs" as pointer though.

@Perelandric
Copy link

Perelandric commented Mar 29, 2016

I wonder if a low-impact alternative to the IsZeroer interface would be to allow one to return an error called json.CanOmit (or similar) from an implementation of the Marshaler interface. That way the dev is in control of determining what constitutes a zero value, and it doesn't impact other code.

It's not a perfect solution, since one can't add methods to types defined in another package, but this can be worked around to a degree.

Taking the time.Time example:

type MyTime struct {
  time.Time
}

// Implement the Marshaler interface
func (mt MyTime) MarshalJSON() ([]byte, error) {
  res, err := json.Marshal(mt.Time)

  if err == nil && mt.IsZero() {
    return res, json.CanOmit // Exclude zero value from fields with `omitempty`
  }
  return res, err
}

I haven't looked into implementation, but on the surface it would seem like a low-overhead solution, assuming the only work would be to check if an error returned equals json.CanOmit on fields where omitempty was included.

Using errors as a flag is not without precedent in the standard library, e.g. filepath#WalkFunc allows one to return filepath.SkipDir to skip recursion into a directory.

@joeshaw
Copy link
Contributor Author

joeshaw commented May 12, 2016

@Perelandric I mentioned a possible sentinel error value in https://golang.org/cl/13914 but I didn't get feedback on the idea or an opportunity to implement it before the Go 1.7 freeze. After Russ's comments on my original CL (and showing the unexpected difficulty in implementing this) I think that's the better way to go.

@gopherbot
Copy link

gopherbot commented May 15, 2016

CL https://golang.org/cl/23088 mentions this issue.

@adg
Copy link
Contributor

adg commented Jul 19, 2016

While it's clear that we can do this, it's not clear that we want to. I'd like to see a formal proposal document that weighs the advantages and drawbacks of this feature addition. In particular, I am concerned about compatibility and maintenance burden.

@joeshaw
Copy link
Contributor Author

joeshaw commented Jul 19, 2016

thanks Andrew. I worked on this a little bit at GopherCon. I will look into putting together a formal proposal.

@albrow
Copy link
Contributor

albrow commented Aug 2, 2016

@joeshaw we ran into this issue at my place of work and I'm eagerly awaiting your proposal. Feel free to contact me if you would like any help. Email is on my profile.

@Perelandric
Copy link

Perelandric commented Sep 8, 2016

@joeshaw Is the proposal you're considering based on the sentinel object idea, or are you considering a different approach? Do you think you'll have time for this before the next release?

@joeshaw
Copy link
Contributor Author

joeshaw commented Sep 9, 2016

@Perelandric Yes, I think the sentinel object idea is the most straightforward way to go.

Other options include:

  • The IsZeroer interface (but I think this has potential backward compatibility issues)
  • "Backspacing" over objects that serialize to {} (but I think this requires too big a change to the JSON encoder code, and doesn't handle json.Marshaler implementers like time.Time)

I don't think I will be able to do this (proposal + implementation) before Go 1.8. If someone else wants to take it on for 1.8, I will gladly pass along my knowledge and partial implementation.

@Perelandric
Copy link

Perelandric commented Sep 10, 2016

Thanks @joeshaw. I created an implementation using the sentinel error for the encoding/json package and will start to work on the proposal in a bit. I think I'll focus primarily on this approach.

The Marshaler interface in encoding/xml is different from that in encoding/json, and seems as though a custom zero-value can already be established without needing to return anything special. Did you find that to be true?

After I make a little more progress, I'll post a link to a branch in case you, @albrow or anyone else wishes to review and contribute.

If you have any additional thoughts or info in the meantime, please let me know. Thank you!

@Perelandric
Copy link

Perelandric commented Sep 19, 2016

Change of heart on this. If there's resistance to adding to packages, then this won't fly. Maybe someone else wishes to advocate for this.

@pschultz
Copy link

pschultz commented Dec 14, 2016

Mentioned in one of the duplicate issues was the idea to let MarshalJSON return (nil, nil) to skip the field. Borrowing your earlier example:

type MyTime struct { time.Time }

func (mt MyTime) MarshalJSON() ([]byte, error) {
    if mt.IsZero() {
        return nil, nil // Exclude zero value from fields with `omitempty`
    }

    return json.Marshal(mt.Time)
}

In Go 1.7, returning nil is not a valid implementation for MarshalJSON and leads to "unexpected end of JSON input" errors. This approach doesn't require any visible change to the encoding package (not even adding an error value).

For what it's worth, I just intuitively wrote a MarshalJSON method like that, expecting a field to be omitted from the JSON output.

@joeshaw
Copy link
Contributor Author

joeshaw commented Dec 14, 2016

@pschultz That approach seems reasonable to me, but it can't be used with omitempty.

The reason you get that error is because the JSON encoder checks for the validity of the JSON coming out of MarshalJSON and the result of returning a nil byte slice is (something like) "key":,. If returning a nil byte slice indicated that it should be omitted that'd be a different way to omit something from being encoded in the JSON than omitempty. (That might be fine, it seems ok to me.)

The benefit of the error value is that it could fit in with the existing omitempty because you'd return something like []byte(""), ErrOmitEmpty and it'd obey omitempty yet still return a valid value ("") if not set.

@Perelandric
Copy link

Perelandric commented Dec 14, 2016

@pschultz: A nil return wouldn't be able to be used as a flag for omitempty without causing undesired errors when omitempty is not present, since the implementation of MarshalJSON doesn't know when omitempty is actually there.

I don't know if having nil as an alternative to omitempty would be the best either. Seems like the user of a type should be the one deciding when it is omitted. IMO, the implementation of MarshalJSON should always return some useful representation, or an error when that's impossible.

oneumyvakin added a commit to oneumyvakin/jirardeau that referenced this issue Jan 17, 2017
@PumpkinSeed
Copy link

PumpkinSeed commented Sep 20, 2021

@joeshaw thanks for this, Our company really needed this feature so I just decided to implement it. https://github.com/PumpkinSeed/json

csrwng added a commit to csrwng/hypershift-1 that referenced this issue Nov 10, 2021
This serializer is needed for ignition configuration serialization. It
has support for zero values of structs with `omitempty`. Using the
stdlib json serializer results in invalid ignition configurations.
See https://github.com/clarketm/json and
golang/go#11939
csrwng added a commit to csrwng/hypershift-1 that referenced this issue Nov 11, 2021
This serializer is needed for ignition configuration serialization. It
has support for zero values of structs with `omitempty`. Using the
stdlib json serializer results in invalid ignition configurations.
See https://github.com/clarketm/json and
golang/go#11939
@mitar
Copy link
Contributor

mitar commented Jan 6, 2022

I made a stand-alone proposal which could address partially also concerns here: that returning nil from MarshalJSON would omit the field: #50480

It allows some other use cases (like dynamic decision based on data itself) but it can help here, too, I think (for some use cases, not all though).

@eli-darkly
Copy link

eli-darkly commented Apr 13, 2022

I agree with everyone who thinks it's really unfortunate that all attempts to address this so far have been abandoned with the general reason of "let's not add to these packages." My only extra comment is this:

Now that Go has generics, there are strong reasons for people to implement a type like Optional[T] for representation of optional types (the main reasons are immutability and nil-safety). Such a type will likely (or necessarily?) be implemented as a struct like { defined bool; value T }. But currently there is no way to get omitempty behavior for such a thing— so anyone who wants unset properties to be dropped must continue using pointers. And that's true even if T is a simple type; that is, you can get omitempty behavior with a plain bool, but that means false values will be dropped; if the semantics you want are "true, false, or undefined" and you can't use a type like Optional[bool], then your only option is to use *bool which has the disadvantages I mentioned. To me that's a not-insignificant usability problem for the package.

I think any of the approaches mentioned above would solve this for a type like Optional[T]. Using the interface approach, it would as simple as func (o Optional[T]) IsZero() bool { return !o.defined }. Using the other approaches, there would be a func (o Optional[T]) MarshalJSON which would return nil/a sentinel error if defined was false and otherwise it would call json.Marshal on the wrapped value. Etc. Being able to implement such wrapper types is a major use case for generics, and it's unfortunate that only a fixed set of value types can have this particular behavior.

xenoscopic added a commit to mutagen-io/mutagen that referenced this issue May 26, 2022
While it would be nice to exclude empty configuration structures from
the resulting JSON, it's simply too tedious and makes the structures
used to represent sessions more error-prone since nil pointer checks are
required when traversing them. The better option would be something like
golang/go#11939, so we'll just see how that space evolves.

Signed-off-by: Jacob Howard <jacob@mutagen.io>
2uasimojo added a commit to 2uasimojo/installer that referenced this issue Jun 21, 2022
install-configs generated with versions of installer libs since
fdb10d9 / openshift#5793 would fail to install on 4.10 due to the addition of
an optional-but-not-omitempty field combined with strict unmarshalling
that punts on unrecognized fields.
- Add `omitempty` so the field isn't generated during marshaling if it's
missing in the go object.
- Make it a pointer field, because `omitempty` isn't effective otherwise
(cf. golang/go#11939).

closes: 2098299
2uasimojo added a commit to 2uasimojo/installer that referenced this issue Jun 22, 2022
install-configs generated with versions of installer libs since
fdb10d9 / openshift#5793 would fail to install on 4.10 due to the addition of
an optional-but-not-omitempty field combined with strict unmarshalling
that punts on unrecognized fields.
- Add `omitempty` so the field isn't generated during marshaling if it's
missing in the go object.
- Make it a pointer field, because `omitempty` isn't effective otherwise
(cf. golang/go#11939).

closes: 2098299
2uasimojo added a commit to 2uasimojo/installer that referenced this issue Jun 22, 2022
OSImage was added recently with `omitempty`, but this doesn't stop json
emitting contents for it when marshaled unless the field is also a
pointer type -- see golang/go#11939

partial-bug: 2098299
@johan-lejdung
Copy link

johan-lejdung commented Oct 12, 2022

@eli-darkly

I agree with everyone who thinks it's really unfortunate that all attempts to address this so far have been abandoned with the general reason of "let's not add to these packages." My only extra comment is this:

Now that Go has generics, there are strong reasons for people to implement a type like Optional[T] for representation of optional types...

Exactly our use-case, and it's infuriating that there is no way to omit fields using std library when you have an implementation such as this.

All above solutions work, but I'd personally find it most clean if return nil, ErrOmitEmpty were to be implemented. I'd love to help drive this forward, this has been an open issue for ~7 years now, what are the next steps?

@sagikazarmark
Copy link

sagikazarmark commented Oct 12, 2022

what are the next steps?

Convincing the Go team to reconsider, I guess. :/

stevej pushed a commit to linkerd/linkerd2-proxy-init that referenced this issue Nov 17, 2022
Fixes
- linkerd/linkerd2#2962
- linkerd/linkerd2#2545

### Problem
Field omissions for workload objects are not respected while marshaling to JSON.

### Solution
After digging a bit into the code, I came to realize that while marshaling, workload objects have empty structs as values for various fields which would rather be omitted. As of now, the standard library`encoding/json` does not support zero values of structs with the `omitemty` tag. The relevant issue can be found [here](golang/go#11939). To tackle this problem, the object declaration should have _pointer-to-struct_ as a field type instead of _struct_ itself. However, this approach would be out of scope as the workload object declaration is handled by the k8s library.

I was able to find a drop-in replacement for the `encoding/json` library which supports zero value of structs with the `omitempty` tag. It can be found [here](https://github.com/clarketm/json). I have made use of this library to implement a simple filter like functionality to remove empty tags once a YAML with empty tags is generated, hence leaving the previously existing methods unaffected

Signed-off-by: Mayank Shah <mayankshah1614@gmail.com>
stevej pushed a commit to linkerd/linkerd2-proxy-init that referenced this issue Nov 17, 2022
Fixes
- linkerd/linkerd2#2962
- linkerd/linkerd2#2545

### Problem
Field omissions for workload objects are not respected while marshaling to JSON.

### Solution
After digging a bit into the code, I came to realize that while marshaling, workload objects have empty structs as values for various fields which would rather be omitted. As of now, the standard library`encoding/json` does not support zero values of structs with the `omitemty` tag. The relevant issue can be found [here](golang/go#11939). To tackle this problem, the object declaration should have _pointer-to-struct_ as a field type instead of _struct_ itself. However, this approach would be out of scope as the workload object declaration is handled by the k8s library.

I was able to find a drop-in replacement for the `encoding/json` library which supports zero value of structs with the `omitempty` tag. It can be found [here](https://github.com/clarketm/json). I have made use of this library to implement a simple filter like functionality to remove empty tags once a YAML with empty tags is generated, hence leaving the previously existing methods unaffected

Signed-off-by: Mayank Shah <mayankshah1614@gmail.com>
stevej pushed a commit to linkerd/linkerd2-proxy-init that referenced this issue Nov 22, 2022
Fixes
- linkerd/linkerd2#2962
- linkerd/linkerd2#2545

### Problem
Field omissions for workload objects are not respected while marshaling to JSON.

### Solution
After digging a bit into the code, I came to realize that while marshaling, workload objects have empty structs as values for various fields which would rather be omitted. As of now, the standard library`encoding/json` does not support zero values of structs with the `omitemty` tag. The relevant issue can be found [here](golang/go#11939). To tackle this problem, the object declaration should have _pointer-to-struct_ as a field type instead of _struct_ itself. However, this approach would be out of scope as the workload object declaration is handled by the k8s library.

I was able to find a drop-in replacement for the `encoding/json` library which supports zero value of structs with the `omitempty` tag. It can be found [here](https://github.com/clarketm/json). I have made use of this library to implement a simple filter like functionality to remove empty tags once a YAML with empty tags is generated, hence leaving the previously existing methods unaffected

Signed-off-by: Mayank Shah <mayankshah1614@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.