Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: encoding/json: allow defining which columns will be marshal #45812

Open
jtorz opened this issue Apr 28, 2021 · 3 comments
Open

proposal: encoding/json: allow defining which columns will be marshal #45812

jtorz opened this issue Apr 28, 2021 · 3 comments
Labels
Projects
Milestone

Comments

@jtorz
Copy link

@jtorz jtorz commented Apr 28, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16.3 linux/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jtorz/.cache/go-build"
GOENV="/home/jtorz/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/jtorz/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jtorz/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3209792062=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Similar issues: #22480, #11939, #45669

The idea of this proposal is to allow defining which columns will be marshal. It would be useful in many situations where we want to reduce the data size like HTTP responses, or marshalling to cache JSON values.

Currently, with the Standard library when we have something like this, the inner Structures are marshal into empty values.
https://play.golang.org/p/UVjAKNTMqW9

package main

import (
	"encoding/json"
	"fmt"
)

type A struct {
	Something1 string `json:"something_1"`
	Something2 string `json:"something_2"`
	Something3 string `json:"something_3"`
	Something4 string `json:"something_4"`
}

type B struct {
	A     A      `json:"a,omitempty"`
	Name  string `json:"name,omitempty"`
	Hello string `json:"hello,omitempty"`
}

func main() {
	b := B{Name: "Jane"}
	var j, _ = json.Marshal(b)
	fmt.Printf("%s\n", j)
}

In the result "a" holds an object with empty values and there are case where thi.

{"a":{"something_1":"","something_2":"","something_3":"","something_4":""},"name":"Joe"}

Solution Proposal

I copied the code from the standard library and did some modifications to allow this. https://github.com/jtorz/jsont

In my module I added the function func MarshalFields(v interface{}, whitelist F) ([]byte, error).

MarshalFields receives a type F map[string]F which works as a whitelist that holds the JSON keys that will be marshal if the key exists in the map, no matter the value, the field is added.

In the next example only the "name" is added to the JSON, all other fields are ignored.

https://play.golang.org/p/ghG6SDyXNLU

In this Example

package main

import (
	"fmt"
	"github.com/jtorz/jsont/v2"
)

type A struct {
	Something1 string `json:"something_1"`
	Something2 string `json:"something_2"`
	Something3 string `json:"something_3"`
	Something4 string `json:"something_4"`
}

type B struct {
	A     A      `json:"a,omitempty"`
	Name  string `json:"name,omitempty"`
	Hello string `json:"hello,omitempty"`
}

func main() {
	b := B{Name: "Joe", Hello: "This will be ignored"}
	j, _ := jsont.MarshalFields(b, jsont.F{
		"name": nil,
	})
	fmt.Printf("%s\n", j)
}
{"name":"Joe"}

Cases

I considered different scenarios like slices, nested structures, or self-referential structures.

The following cases will use the same structs definitions.

type A struct {
	Something1 string `json:"something_1"`
	Something2 string `json:"something_2"`
	Something3 string `json:"something_3"`
	Something4 string `json:"something_4"`
}

type B struct {
	A     A      `json:"a"`
	Name  string `json:"name"`
	Hello string `json:"hello"`
	Bs    []B    `json:"bs,omitempty"`
}

self-referential Slice (Recursive)

b := B{
    Name: "Joe",
    Bs: []B{
        {Name: "Jane"},
        {Name: "John"},
        {Name: "Janah"},
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "name": nil,
    "bs":   jsont.Recursive,
})
fmt.Printf("%s\n", j)

OUTPUT:

{"name":"Joe","bs":[{"name":"Jane"},{"name":"John"},{"name":"Janah"}]}

self-referential Slice (Non Recursive)

b := B{
    Name: "Joe",
    Bs: []B{
        {Hello: "World!"},
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "name": nil,
    "bs": jsont.F{
        "hello": nil,
    },
})
fmt.Printf("%s\n", j)

OUTPUT:

{"name":"Joe","bs":[{"hello":"World!"}]}

Nested Struct (Whole structure)

b := B{
    Hello: "Janah",
    A: A{
        Something1: "value1",
        Something2: "value2",
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "hello": nil,
    "a":     nil,
})
fmt.Printf("%s\n", j)

OUTPUT:

{"hello":"Janah","a":{"something_1":"value1","something_2":"value2","something_3":"","something_4":""}}

Nested Struct (Specific fields)

b := B{
    Hello: "Janah",
    A: A{
        Something1: "value1",
        Something2: "value2",
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "hello": nil,
    "a": jsont.F{
        "something_1": nil,
    },
})
fmt.Printf("%s\n", j)

OUTPUT:

{"hello":"Janah","a":{"something_1":"value1"}}
@gopherbot gopherbot added this to the Proposal milestone Apr 28, 2021
@ianlancetaylor ianlancetaylor changed the title proposal: encoding/json allow defining which columns will be marshal proposal: encoding/json: allow defining which columns will be marshal Apr 28, 2021
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 28, 2021

Can you write in this issue exactly what json.MarshalFields does? It's easier to understand a spec than an implementation. Thanks.

@jtorz
Copy link
Author

@jtorz jtorz commented Apr 29, 2021

I added more information to the Solution Proposal part

@rsc
Copy link
Contributor

@rsc rsc commented May 5, 2021

/cc @dsnet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Proposals
Incoming
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants