encoding/json: unmarshal option to treat omitted fields as null

[Qhesz]

Json objects have two types of null-like fields, either a null value for foo such as {"foo":null}, or an omitted foo field such as {}. When unmarshalling into a Golang struct, the field can't really be absent, but I can choose whether the field's type is nullable or not(*1). I don't want to preserve the distinction between json absent and null (one null-type is plenty), so I'd like the option to treat the absent field as if it were null, and map both null-like json types into the same Golang behaviour.

(*1) See #33835 for proper null unmarshalling support, where null into a non-nullable type returns an error, and user-defined types determine whether they can unmarshal null or not. This issue makes most sense in combination with that. The current behaviour is to treat both absent and null like absent when unmarshalling, and in combination with #33835, the (opt-in) behaviour I'm proposing is to treat both absent and null like null instead.

The behaviour that I want to avoid is having the Golang field skipped over during unmarshalling when the json field is absent. If my Golang field is optional / some nullable type, it doesn't make a huge difference - the field will be skipped, stay at its zero value, which I can define as the same value I'd get unmarshalling null into it. The issue is when I have a non-nullable field - I've gone to the effort of saying my input has a defined structure with a defined non-nullable type, I want the unmarshaller to return an error if the input did not contain the structure I said it should. My proposal would return an error in this case, by treating the absent as null, then unmarshalling null into the non-nullable golang type, and in combination with (*1) that would return an error.

The implementation I prefer is an opt-in json.Decoder flag along the lines of OmittedAsNull(), so I can set it once for my whole project. I'm less interested in an omittedasnull struct tag, because I'd end up writing it on every struct field I define.

Either way the code would be the same when they're used: after parsing an object into a struct, determine which struct fields were not visited, and unmarshal null into them.

[julieqiu]

/cc @bradfitz @mvdan

[mvdan]

The decoder already has a number of options, and there are proposals to add many more. In general, the bar for new API is high, because we have to be careful about what every possible combination of options will do.

Have you thought about ways to accomplish this without new API, or to instead contribute to one of the many existing API proposals?

[Qhesz]

Sure, it could be implemented as the default behaviour, but then compatibility would break.

I mentioned struct tags above, but would end up writing them on every field.

My three proposals OmittedAsNull, OmitNulls and StrictNulls could be rolled into one SuperNull setting. While I think StrictNulls is a no-brainer, and should become the default eventually, it is not obvious it should always be used with the other two. The other two could probably be bundled together.

There is no workaround to achieve the behaviour I'm talking about - encoding/json completely controls how structs are traversed, and I'm proposing an opt-in change to its behaviour. encoding/json does not expose any methods of configuration other than the Decoder API and struct tags, so I'm not sure what other alternative you're suggesting.