Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/json: ignore tag "-" not working on embedded sub structure when decoding #30701

Open
LouAdrien opened this issue Mar 9, 2019 · 12 comments
Labels
Milestone

Comments

@LouAdrien
Copy link

@LouAdrien LouAdrien commented Mar 9, 2019

What version of Go are you using (go version)?

go1.10.1 darwin/amd64

What operating system and processor architecture are you using (go env)?

go env Output
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/Lou/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/Lou/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10.1/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10.1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/f3/0ldm0j6n5bx3zyyz87pb61gw0000gn/T/go-build911277693=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I am trying to unmarshall a complex object.

type DC struct {

    //other fields
    ReplenishmentData map[string]ProductReplenishment `bson:"-"`
    //other fields
}

type ProductReplenishment struct {
    //Other fields
    SafetyStockInDay int `json:"SafetyStockInDay" bson:"SafetyStockInDay"`
    AlreadyOrderedQuantityForReplenishment *map[float64]*UnitQuantity `json:"-" bson:"-"`
    //Other fields
}

Lets say I decode the following json:

{
  "ReplenishmentData": {
    "000822-099": {
      "SafetyStockInDay": 7
    },
    "001030-001": {
      "SafetyStockInDay": 7
    }
  }
}

Into a structure instance hierarchy in which the AlreadyOrderedQuantityForReplenishment is not empty, after decoding this field will be set to and empty map, overriding the initial value.

What did you expect to see?

After the decode operation, I expect the AlreadyOrderedQuantityForReplenishment map to not be empty.

What did you see instead?

Before
before decode

After
after decode

Why is the decoder not ignoring the field all together as specified in the docs? Am I missing something?

Thanks a lot for any help,

@mvdan

This comment has been minimized.

Copy link
Member

@mvdan mvdan commented Mar 9, 2019

Could you provide a full self-contained example, please?

Also, note that the current Go version is 1.12; 1.10 is about a full year behind on the latest changes to encoding/json.

@mvdan mvdan added the WaitingForInfo label Mar 9, 2019
@mvdan mvdan changed the title json ignore tag (“-”) not working on embedded sub structure when decoding encoding/json: ignore tag "-" not working on embedded sub structure when decoding Mar 9, 2019
@giuscri

This comment has been minimized.

Copy link

@giuscri giuscri commented Mar 9, 2019

This snippet should hit the same problem,

package main

import (
	"encoding/json"
	"log"
)

func main() {
	type C struct {
		ID   string `json:"-"`
		Name string `json:"name"`
	}

	type A struct {
		B map[string]C `json:"b"`
	}

	byt := []byte(`
	{
		"b": {
			"string0": {
				"name": "name0"
			},
			"string1": {
				"name": "name1"
			}
		}
	}
	`)
	b := map[string]C{
		"string0": C{ID: "42", Name: ""},
		"string1": C{ID: "53", Name: ""},
	}
	a := A{B: b}
	log.Println("current value of a:", a)

	if err := json.Unmarshal(byt, &a); err != nil {
		log.Fatalln(err)
	}
	log.Println("current value of a:", a)
}
$ PATH=/home/g/opensource/go/bin:$PATH go version
go version go1.10.1 linux/amd64
$ PATH=/home/g/opensource/go/bin:$PATH go run main.go 
2019/03/09 18:06:42 current value of a: {map[string1:{53 } string0:{42 }]}
2019/03/09 18:06:42 current value of a: {map[string0:{ name0} string1:{ name1}]}

After unmarshalling, ID's are overwritten and set to the default value for string.

I can reproduce that with current master (05051b5) too.

@LouAdrien

This comment has been minimized.

Copy link
Author

@LouAdrien LouAdrien commented Mar 10, 2019

Thanks a lot for the effort @giuscri i was about to create it. Here it is as a playground: https://play.golang.org/p/gBJ9aa_Tbbf

So it appears you discovered the "-" tag is broken event for simple sub types (my test was refering to a complex map as subfield), but indeed a string will be reset to default value too.

@giuscri

This comment has been minimized.

Copy link

@giuscri giuscri commented Mar 10, 2019

Note that the problem is not reproducible on simpler unmarshalling such as

	type B struct {
		ID   string `json:"-"`
		Name string `json:"name"`
	}

https://play.golang.org/p/Axvw-A5zpTs

@LouAdrien

This comment has been minimized.

Copy link
Author

@LouAdrien LouAdrien commented Mar 10, 2019

Yes, first level fields tagged with "-" are ignored properly, the issue arise only in sub structures (potentially only in sub maps?).

@LouAdrien

This comment has been minimized.

Copy link
Author

@LouAdrien LouAdrien commented Mar 12, 2019

@mvdan Are the infos provided enough? If yes can we hope this to be fixed? This issue is quite big for anyone trying to build real life POST apis with complex structures...

Regards,

@LouAdrien

This comment has been minimized.

Copy link
Author

@LouAdrien LouAdrien commented Mar 14, 2019

@mvdan ping, I think all infos provided should be enough, could we remove the "WaitingForInfo" and treat the bug?

Regards,

@mvdan mvdan removed the WaitingForInfo label Mar 14, 2019
@mvdan

This comment has been minimized.

Copy link
Member

@mvdan mvdan commented Mar 14, 2019

The information should be enough. I don't have the bandwidth to look at this bug right now, though.

@LouAdrien

This comment has been minimized.

Copy link
Author

@LouAdrien LouAdrien commented Mar 15, 2019

I see, thanks @mvdan ! Do you have any idea of when you'd have time to handle it? If not possible in the foreseeable future, I'll have to reconsider the architecture of my project and go for another solution. Thanks a lot,

@mvdan

This comment has been minimized.

Copy link
Member

@mvdan mvdan commented Mar 15, 2019

Go releases happen every six months, so even if a fix was out today, you wouldn't be able to use it for another four months. If you need an urgent fix, I'd suggest forking the package and hacking it to do what you want.

@fraenkel

This comment has been minimized.

Copy link
Contributor

@fraenkel fraenkel commented Mar 16, 2019

The code you have provided works correctly.
From the json doc:
To unmarshal a JSON object into a map, Unmarshal first establishes a map to use. If the map is nil, Unmarshal allocates a new map. Otherwise Unmarshal reuses the existing map, keeping existing entries. Unmarshal then stores key-value pairs from the JSON object into the map.

Notice the last sentence, which says it replaces not merges values.

@LouAdrien

This comment has been minimized.

Copy link
Author

@LouAdrien LouAdrien commented Mar 18, 2019

Hm i see, but then it's a design choice that is IMOH very weird. Real-life data is almost always complex and with multiple depth levels, which either needs to send partial data for perf reason, or just can't represent data full structure over JSON (ex: function handlers, map with non primitive types as key, etc). This prevent using go in any real life complexe situation, should this be considered an appropriate feature request? Or do you think it's too much of a change and might break existing application logic that rely on these sub-values be reinitialised?

@katiehockman katiehockman added this to the Go1.13 milestone Mar 18, 2019
@rsc rsc modified the milestones: Go1.13, Go1.14 May 1, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.