encoding/json: tag json:"-" doesn't hide an embedded field

[ainar-g]

What version of Go are you using (go version)?

$ go version
go version go1.13.3 linux/amd64
$ gotip version
go version devel +696c41488a Mon Nov 11 15:37:55 2019 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes, see gotip version.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ainar/.cache/go-build"
GOENV="/home/ainar/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ainar/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ainar/go/go1.13"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/ainar/go/go1.13/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/ainar/dev/tmp/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build199093238=/tmp/go-build -gno-record-gcc-switches"

What did you do?

type A struct {
	Name string `json:"name"`
}

type B struct {
	A
	Name string `json:"-"`
}

https://play.golang.org/p/uKY3umGYEp3

What did you expect to see?

Either:

<nil> {}

Or:

some error about tag conflict

What did you see instead?

<nil> {"name":"1234"}

[mvdan]

Hmm. While I agree that intuitively this is what one would expect, I don't think the docs give a definitive answer. The pieces of the docs that explain "-" and anonymous fields are separate, so it's not clear which takes precedence.

[mvdan]

Here's another tricky question. The docs say:

The Go visibility rules for struct fields are amended for JSON when deciding which field to marshal or unmarshal.

However, does this mean just the original Go field's name, or does it also include the JSON field name one can add via a struct field tag? Right now, the "shadowing" of field names follows what's in the struct tags. That is, your Name string `json:"-"` doesn't affect your Name string `json:"name"`, becaues they don't even share a name. The names in question are - and name.

[ainar-g]

The names in question are - and name.

Well, - is technically a “directive” and not a name, is it? But I agree, it's a lot of undefined behaviour.

[mvdan]

Here's an example of my reasoning above: https://play.golang.org/p/lEDqVGEVxAR

The shadowing happens based on what's in the struct field tags. If we completely swap that and use the original struct names instead, we'd break valid use cases like this one.

Well, - is technically a “directive” and not a name, is it?

You're right there. But even if you say "in that case the name remains the original Name", you still have two different names - Name and name.

[mvdan]

I'll leave this open for a week if anyone has more thoughts, or a specific way in which this could be changed without breaking existing valid use cases.

The only idea that comes to mind that would be truly backwards-compatible would be to extend the json tag syntax to allow omitting a field name, not just a specific field only. For example, `json:"name,-"` to omit any embedded field with `json:"name"`. "-" already takes the place of a name, so it taking the place of an option doesn't seem like a terrible idea.

The question then would be if it's worth adding this feature. How often does one want to omit fields from anonymous structs? Are there any current workarounds? Do any external json libraries support this feature?

[ainar-g]

If the behaviour isn't changed, can it at least be documented better, so that I could bother @dominikh the community could build static analysis tools to detect such idiosyncrasies?

[mvdan]

Yes, whatever we decide to do, I think it would be good to clarify that the visibility rules apply to the JSON names alone.

[breml]

I looked into this issue myself and I feel like the observation of @ainar-g is valid.
The reason for this is the following:

In the documentation of encoding/json the section explaining "-" is directly following the section about ",omitempty" and there it says, that a field is omitted, if it contains the zero value:

The "omitempty" option specifies that the field should be omitted from the encoding if the field has an empty value, defined as false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string.

As a special case, if the field tag is "-", the field is always omitted. Note that a field with name "-" can still be generated using the tag "-,".

Now, if we extend the examples provided above (by @ainar-g and @mvdan) with the ",omitempty" cases, we see, that in this case the anonymous struct is not considered, even if the field in the "parent" struct is empty (zero value). This is the case regardless of the fact, if the final "name" of the field in the encoded json is defined by the struct field name or the struct tag, the behavior is consistent. See: https://play.golang.org/p/VjMa8H7EI9e

Therefore I would argue, that if the "omitting" for ",omitempty" is always decided based on the value in the "parent" struct, the same should be the case for "-", that is, if in the parent struct we have "-", this named field (evaluated either by the struct tag or by the struct field name, in that order) is never shown in the final encoded json.

[mvdan]

Thanks @breml. I agree that this is intuitively what one would understand; it was my first impression too. You also raise a good point about how omitempty does allow shadowing via both struct field name and json field tag name.

For the record, I still think this would be nice to have, for the sake of consistency. Unfortunately, while omitempty is an option, - is not, so you cannot use it while also specifying a field tag name. This is what I tried to explain in #35501 (comment).

For example, taking the original example once again:

type A struct {
	Name string `json:"name"`
}

type B struct {
	A
	Name string `json:"-"`
}

Following the omitempty logic of shadowing via the json field name, we don't get any shadowing, since the names are Name and name. They would only be equal if A.Name didn't have a struct tag, or if we allowed - as an option like `json:"name,-"`, as I showed above.

Aside from turning - into an option, the only other option we have is to always try struct field name shadowing, even if embedded fields have a struct tag field. That is, Name string `json:"-"` would shadow Name string `json:"name"` as well as X string `json:"Name"`.

I assume this is what you meant by your sentence below:

if in the parent struct we have "-", this named field (evaluated either by the struct tag or by the struct field name, in that order) is never shown in the final encoded json.

It's certainly worth a try, and it could work, but I still wonder if it would break any existing valid uses of encoding/json. I have a CL ready, which we can experiment with. If people find it reasonable, we can attempt a merge early in the 1.15 cycle, and revert if any users bring up regressions.

[gopherbot]

Change https://golang.org/cl/211457 mentions this issue: encoding/json: allow "-" to shadow by struct field name

[mvdan]

@ainar-g @breml please give the CL above a look, or give it a try with your programs. When I wrote it in December, I'm pretty sure the examples from this thread were fixed. I've just rebased it.

I still think this is a bit risky, so I don't want to merge it for 1.15 during the freeze. However, any help testing or reviewing the change is very much welcome, so that we can look at merging once the tree reopens in eleven weeks.

[ainar-g]

@mvdan Sorry, I couldn't check the patch earlier. I can confirm that the patch does make the encoder shadow the embedded field. I've skimmed through the patch and left a small stylistic comment with a +1.

[dsnet]

Stepping back a bit, what's the original use-case that inspired this issue? Is it the desire to be able to embed some other struct type, but exclude certain fields from being forwarded?

[ainar-g]

@dsnet, that was a long time ago, but I think that that was indeed the original intent, yes.

[dsnet]

It sounds like a feature to exclude certain fields from some other struct is what's really desired and we're really getting in the weeds of the name conflict resolution logic to hack up something that could be used for that purpose. After all, declaring a new field with the hopes that it cancels out some other field really feels like a hack.

[ainar-g]

That's kind of what the discussion came to the first time I filed this issue. See this comment by me and this comment by @mvdan. I don't really have the need to exclude fields with hacks currently, since I don't work on that original project any more, but having the behaviour defined, documented, and checked would still be nice.

[dsnet]

I agree we should improve the documentation, but I personally think the current behavior is correct. It seems like the best way to describe the behavior is that 1) we ignore all unexported fields and fields with an explicit json:"-" tag, 2) we determine the JSON object name is by deriving it from the json tag if specified or from the Go struct field name, 3) we apply rules similar to how conflicting Go field names from embedding is handled, but applied on the JSON object names.

[mvdan]

That sound fine to me too. I'm happy to abandon my CL in favor of a documentation CL.

[OneOfOne]

@mvdan honestly I'm in favor of your CL, so many times I had to create an identical struct with just the missing field or use reflect / map.

[dsnet]

@OneOfOne

so many times I had to create an identical struct with just the missing field

What does this mean? Can you elaborate on what you're trying to do?

What I described in #35501 (comment) is high-level description of what currently happens. We should think of the name collision logic in terms of the JSON object name, not the Go struct field name. CL/211457 bakes in more information about the original Go struct field name into the name conflict resolution logic (as implemented by the dominantFields function), which is blends steps together 1, 2, and 3 described in my comment. It makes the logic harder to reason about.

[OneOfOne]

My understanding of the CL was that it'd allow:

type A struct {
	Name string `json:"name"`
}

type B struct {
	A
	Name string `json:"-"`
}

To work (as in return {}).

Right now the only real way to omit a field is to either empty it, marshal, put back the old value, or create a whole new struct without that field and copy all the other data to it.

[dsnet]

Right now the only real way to omit a field is to either empty it, marshal, put back the old value, or create a whole new struct without that field and copy all the other data to it.

This sounds like the use-case I mentioned earlier #35501 (comment). Trying to adjust the name conflict rules to make this work is a hack. The existence of a principled way to exclude fields is 1) more clear as it expresses what the author wants and 2) is simpler since it doesn't requiring modifying the existing naming rules to be more complicated.

[dsnet]

the only real way to omit a field

Also, that's not quite true. Using the rules that already exist today, you can do:

type A struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

// AExclude contains a list of fields to exclude from A.
// It must be embedded alongside A.
type AExclude struct {
	F1 string `json:"name"`
}

type B struct {
	A
	AExclude
}

See https://play.golang.org/p/7wHioEMdfaS

[OneOfOne]

the only real way to omit a field

Also, that's not quite true. Using the rules that already exist today, you can do:

type A struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

// AExclude contains a list of fields to exclude from A.
// It must be embedded alongside A.
type AExclude struct {
	F1 string `json:"name"`
}

type B struct {
	A
	AExclude
}

See https://play.golang.org/p/7wHioEMdfaS

I'm have been programming in Go as my main language for over 7 years and you just blew my mind.

[ainar-g]

@dsnet, amazing trick, thanks! Although go vet seems to not like that one bit:

go/tmp/main.go:18:2: struct field Name repeats json tag "name" also at main.go:9

[mvdan]

That sounds like a vet bug; it should really only complain about duplicate tag names living directly in the same struct.

[timothy-king]

@mvdan My intuition is that users should get a warning in the AExclude example. The name AExclude and the comment makes the user's intention clear in this case. But this can also happen by accident that 2 embedded struct fields have the same json:"name" that are cancelling each other.

My hunch is that the number of users that have this happen to them accidentally >> the number of users that do this intentionally. (This is "mind blowing" after all.) I am also more worried about not reporting the accident case than I am about annoying the exclude case. (But my hunches and priorities could be all wrong here.)

It is definitely possible to have an annotation that vet parses to understand the intent and suppress the warning for the intentional exclude. The annotation could be something like a funny field or struct name so it would not be too intrusive. It could also be an end of line comment or a field tag. Not sure the extra complexity is warranted though.

[jos-]

Another (more intuitive) way to omit attributes from json marshalling is:

type A struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

type B struct {
	A                                      // Or *A, both possible
	Name *struct{} `json:"name,omitempty"` // This attribute will be omitted
}

// And marshal something of type B

See https://play.golang.org/p/0OdxDtXcx8-, see also this old blog post.

[epolar]

It's work for me to omit attributes from json marshalling:
type A struct {
Name string json:"name"
}

type B struct {
A
Name string json:"name,-,omitempty"
}