proposal: encoding/json/v2: escape hatch to omit & override field

[jacobzim-stl]

Proposal Details

When dealing with malfunctioning external systems, escape hatches in the JSON system can be valuable to unblock progress.

I propose the following types are added to encoding/json/v2

package json

type OmitField     struct{}
type OverrideField jsontext.Value

Let's say a misbehaving server rejects the presence of the field "valid", and wants a JSON number for "id". It would be valuable to explicitly omit, or override the field. User may be part of a larger structure, and it would be inconvenient to rewrite the type.

type User struct {
	Name      string         `json:"name'`
	Id        string         `json:"id"`
	Valid     bool           `json:"valid"`
	ExtraJSON map[string]any `json:",unknown"`
}

user := User{
	Name:      "John Doe",
	ExtraJSON: map[string]any{
		"valid": json.OmitField,
		"id":    json.OverrideField(`1234567890123456`),
	},
}
// Marshals to `{"name":"John Doe","id":1234567890123456}`

In the current implementation, trying to do this would result in an error, since duplicate fields can't be marshaled.

I'd suggest that this only apply for inline fallback map[~string]T fields, perhaps only with json:",unknown". The alternative of writing a custom MarshalFunc adds complexity, and doesn't provide per-instance flexibility.

Thanks @dsnet for your hard work and designs on json/v2

[gabyhelp]

Related Issues

• proposal: encoding/json: dynamic field omission #52803 (closed)
• proposal: encoding/json: allow skipping field by returning an error in `MarshalJSON() ([]byte, error)` #57554 (closed)
• proposal: encoding/json/v2: MarshallJSONTo: Skip fields when marshalling to json #74322
• proposal: encoding/json: allow struct encoder to handle undefined fields #64515
• proposal: encoding/json: allow returning nil from MarshalJSON to omit the field #50480 (closed)
• proposal: encoding/json: add options to allow marshalling of ignored JSON fields #29426 (closed)
• encoding/json: support omitempty and omitzero at API level #72972 (closed)
• proposal: encoding/json: allow defining which fields will be marshalled #45812
• proposal: encoding/json: Struct omitempty #58489 (closed)
• proposal: encoding/json: add an option `omitempty` to json.Encoder #19763 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

I think this should just use WithMarshalers / WithUnmarshalers to do a type specific override, rather than push more complexity into the default marshaling rules.

[jacobzim-stl]

@seankhliao Thanks for the suggestion. That costs the ability to make per-instance omissions, rather than per-type, such as in an array.

As this is for unblocking communication with misbehaving external systems, I think the additional complexity may be welcome — it shouldn't be needed in a happy path.

[dsnet]

The alternative of writing a custom MarshalFunc adds complexity, and doesn't provide per-instance flexibility.

If MarshalFuncs were able to support the ability to apply a filter (e.g., this MarshalFunc is only applicable under /valid), I think this could be more flexible than what this is proposing. Combined with #74322, I think that gets you everything you want?

[jacobzim-stl]

I tried writing out some sample code to do this, I'm likely missing something, but I struggled to get working code here.

	newUsers := NewUsers{
		Users: []User{
			{Name: "Bob", Id: "1231182309", Valid: true},
			{Name: "Sam", ExtraJSON: map[string]any{
				// "age": jsonv2.OmitMember,
				// "id":  jsonv2.OverrideMember(2938123),
			}},
			{Name: "Alice", Id: "10293133412", Valid: false},
		},
	}

	bytes, err := jsonv2.Marshal(newUsers,
		jsonv2.WithMarshalers(
			jsonv2.MarshalToFunc(func(enc *jsontext.Encoder, u *User) error {

				// Filter by JSON pointer, using temporary hack until API is added.
				if enc.StackPointer() == "/users/0" {

					// All existing fields still need to be marshaled by hand.
					// In practice, I'd probably marshal to a map[string]any, 
					// then delete/replace unwanted keys and re-marshal.
					enc.WriteToken(jsontext.BeginObject)

					enc.WriteToken(jsontext.String("name"))
					enc.WriteToken(jsontext.String(u.Name))

					enc.WriteToken(jsontext.String("id"))
					enc.WriteToken(jsontext.Int(123456789))

					// elide "valid" field

					enc.WriteToken(jsontext.EndObject)

					return nil
				}

				return jsonv2.SkipFunc
			}),
		))