New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: encoding/json: preserve unknown fields #22533

Open
ibrt opened this Issue Nov 1, 2017 · 3 comments

Comments

Projects
None yet
5 participants
@ibrt
Contributor

ibrt commented Nov 1, 2017

Yesterday I've implemented #15314, which allows to optionally fail JSON decoding if an object has a key which cannot be mapped to a field in the destination struct.

In the discussion of that proposal, a few people floated the idea of having a mechanism to collect such keys/values instead of silently ignoring them or failing to parse.

The main use case I can think of is allowing for JSON to be decoded into structs, modified, and serialized back while preserving unknown keys (modulo the order in which they appeared, and potentially "duplicate" keys that are dropped due to uppercase/lowercase collisions, etc.). This behavior is supported by many languages / libraries and other serialization systems such as protocol buffers.

I propose to add this type to the JSON package:

type UnknownFields map[string]interface{}

Users of the JSON package can then embed this type in structs for which they'd like to use the feature:

type Data struct {
  json.UnknownFields
  FirstField int
  SecondField string
}

On decoding, any object key/value which cannot be mapped to a field in the destination struct would be decoded and stored in UnknownFields. On encoding, any key present UnknownFields would be added to the serialized object.

I can think of a couple edge cases which are tricky, and I propose to resolve them as follows:

Nested structs

It's possible for nested structs to also declare UnknownFields. In such cases any UnknownFields in nested structs should be ignored, both when decoding and encoding. Pros: it is consistent with how we already flatten fields, and it's the only way to ensure decoding is unambiguous. Cons: keys that somehow were set to UnknownFields in a child struct would be ignored on encoding.

Key collisions

When encoding it's possible that a key in UnknownFields would collide with another field on the struct. In such cases the key in UnknownFields should be ignored. Pros: it is consistent with the behavior in absence of UnknownFields, seems generally less error prone, it cannot happen in a plain decode/edit/encode cycle, it's unambiguous. Cons: it can possibly lead to silently dropping some values.

PS: I'm happy to do the implementation should the proposal or some variation of it be approved.

@gopherbot gopherbot added this to the Proposal milestone Nov 1, 2017

@gopherbot gopherbot added the Proposal label Nov 1, 2017

@ibrt ibrt changed the title from proposal: encoding/json: collect unknown fields to proposal: encoding/json: preserve unknown fields Nov 1, 2017

@dsnet

This comment has been minimized.

Member

dsnet commented Nov 1, 2017

  • When encoding, what happens when UnknownFields contains a key that is a duplicate of a field in the struct?
  • What if I want the unknown values as json.RawMessage instead of interface{}, which is fairly nasty to work with.

Stepping back a moment. What is the use case you have in mind? Proposals are far more effective if they start with concrete problems they are trying to solve.

@ibrt

This comment has been minimized.

Contributor

ibrt commented Nov 1, 2017

Hi!

As for the first bullet point it is already described in the proposal (last paragraph: key collisions).

Regarding json.RawMessage it seems to me that if one wants to manipulate these fields they would have to deserialize them anyways, if one just wants to pass them through it doesn't make a practical difference, except that deserialization would be cheaper. If desired we could simply add a type RawUnknownFields map[string]json.RawMessage, or use a different method for specifying where to put the data (no strong opinion, I just want the feature).

Use case is also somewhat described in the proposal: decoding into a struct, editing, and then encoding back while preserving fields. It is a relatively common need in distributed systems where JSON is used for RPC. For context it can be interesting to read this issue protocolbuffers/protobuf#272 about protobuf removing support for preserving unknown fields and then adding it back.

@rsc

This comment has been minimized.

Contributor

rsc commented Nov 6, 2017

I'd like to save this issue for a future rethink of all of encoding/json. It's important we don't keep adding piecemeal each new feature that seems useful by itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment