Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: encoding/json: struct tag to marshal nil slices and maps as non-null #27589

Open
pjebs opened this Issue Sep 10, 2018 · 22 comments

Comments

Projects
None yet
8 participants
@pjebs
Copy link

pjebs commented Sep 10, 2018

There have been many reports about encoding a nil slice or map as [ ] or { } respectively.

See: #2278

Currently it is encoded as null. This is a source of bugs in many non-theoretical cases (eg consumers of JSON APIs). There are many issues on Google about others falling into this pitfall so it is at least not uncommon.

It would be nice if a struct tag can be used to signify that we intend the nil slice or map to be treated as an empty slice or map respectively for the purposes of marshalling.

Current solutions include: https://github.com/helloeave/json. This is not a nice solution because it requires the assistance of a third-party package.

This is how it's implemented in third party lib: helloeave/json@7b6e442

@pjebs

This comment has been minimized.

Copy link
Author

pjebs commented Sep 10, 2018

@mvdan mvdan changed the title Feature request: Struct tag for json to marshall null slices/maps as non-nil proposal: encoding/json: struct tag to marshal nil slices and maps as non-null Sep 10, 2018

@gopherbot gopherbot added this to the Proposal milestone Sep 10, 2018

@gopherbot gopherbot added the Proposal label Sep 10, 2018

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Sep 10, 2018

I've retitled this to be a proposal.

I'm not sure if it's a good idea, but it's certainly feasible and wouldn't add much new API. /cc @bradfitz @dsnet

@pjebs

This comment has been minimized.

Copy link
Author

pjebs commented Sep 12, 2018

For slices and maps, it should be noted that the proposed struct tag is mutually exclusive to omitempty. For backwards compatibility, if both omitempty and the new struct tag is present, then omitempty takes precedence.

@pjebs pjebs closed this Sep 13, 2018

@pjebs pjebs reopened this Sep 13, 2018

@pjebs

This comment has been minimized.

Copy link
Author

pjebs commented Sep 22, 2018

Here is my pull request. I hope it gets accepted because I believe it adds value to the "Go" programming language.

#27813

@pjebs

This comment has been minimized.

Copy link
Author

pjebs commented Sep 22, 2018

I called the struct tag: heednull

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Sep 24, 2018

"heed" is a unique word, at least in terms of Go's existing API.

If this is accepted (once it's reviewed together with all the existing JSON proposals), I suspect we'd want to change "heed" to something else.

@pjebs

This comment has been minimized.

Copy link
Author

pjebs commented Sep 24, 2018

I scoured the dictionary and thesaurus for appropriate words that were short and exemplified the intention but couldn't find anything. Hopefully a better word can be found because I was initially hesitant of the word too - although admittedly it is growing on me.

@pjebs

This comment has been minimized.

Copy link
Author

pjebs commented Sep 24, 2018

@bradfitz Any idea of the timeframe to when all the JSON proposals will be examined? Some of them are now 2+ years. The pivotal proposal is 7+ years.

@QtRoS

This comment has been minimized.

Copy link
Contributor

QtRoS commented Sep 25, 2018

@bradfitz @pjebs what about nullsafe or nullfree or nonull?
+1 that heednull is unacceptable.

@pjebs

This comment has been minimized.

Copy link
Author

pjebs commented Sep 26, 2018

For Go2, omitempty should become omitzero. The zero value and what is colloquially called "empty" are different for maps and slices.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Oct 3, 2018

Will add this to my JSON scan for this cycle.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Oct 3, 2018

Maybe emptynil.

@pjebs

This comment has been minimized.

Copy link
Author

pjebs commented Oct 3, 2018

HEED
verb
1.
pay attention to; take notice of.

noun
1.
careful attention.
@pjebs

This comment has been minimized.

Copy link
Author

pjebs commented Oct 3, 2018

Seems like precisely the right word. It's not part of the Go vocabulary, but it seems like a great addition in various future contexts for programming languages in general.

eg. heed memory, heed security etc

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Oct 10, 2018

heednull means "pay attention to null" by not respecting that the value is null and instead printing something other than "null"? That seems backward to me.

@pjebs

This comment has been minimized.

Copy link
Author

pjebs commented Oct 10, 2018

If it was called heednil instead, then one can argue that the slice or map was nil, and you are instructing the encoder to "pay attention to it".

I then went from that step to heednull because it flowed off the tongue better. The null in heednull was not really a reference to the eventual json output null.

Perhaps due to the confusion, and ambiguity, heednil is closer to being "Go-like".

@rsc rsc added the Proposal-Hold label Oct 17, 2018

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Oct 17, 2018

On hold for JSON sweep.

@pjebs

This comment has been minimized.

Copy link
Author

pjebs commented Oct 17, 2018

What does that mean?

@dsnet

This comment has been minimized.

Copy link
Member

dsnet commented Oct 17, 2018

The json package already has a bloated API surface. Any additions need to carefully considered together with all the other proposals. I alluded to the need for this: #27813 (comment)

The "JSON sweep" is a sweep through all the JSON proposal trying to come up with a comprehensive yay/nay to each proposal in an effort to balance features desired with not adding too much technical debt.

@tommie

This comment has been minimized.

Copy link

tommie commented Dec 31, 2018

Current solutions include: https://github.com/helloeave/json

This fork (which sadly I can't use because AppEngine doesn't have sync.Map, it seems) sets an encoder-wide option instead of using a struct tag. It feels to me that either an entire system is fine with having null, or there is an expectation that all slices are non-null. Why is the proposal to do this as a struct tag instead of an encoder option?

@QtRoS

This comment has been minimized.

Copy link
Contributor

QtRoS commented Dec 31, 2018

@tommie in my opinion the problem is that any system doesn't work in isolation. Complex software has complex integrations with different technologies and different expectations. Global flag is not versatile enough.

@tommie

This comment has been minimized.

Copy link

tommie commented Dec 31, 2018

Complex software has complex integrations with different technologies and different expectations. Global flag is not versatile enough.

That is a non-answer, IMHO (and somewhat belittling). I can see different complex software requiring different options, and AFAIK, this thread has not discussed the already existing solution yet.

  • Say you have a frontend that needs non-null results, and a backend that wants null results. It's not per-field, but per encoder invocation.
  • Say most fields are "arrays", but one field is an optional array. Then it's per-field.

Defining what the default behavior is could still be an encoder option, while allowing per-field overrides. The two solutions are not mutually exclusive.

The problem I'm trying to solve is interoperability with Javascript. Passing null means my browser-side code needs to handle an empty array in a non-idiomatic way, leading to more fragile code. It seems to me @pjebs is trying to solve a different problem with the struct tag solution, and I'm curious what that is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.