encoding/json: unexpected output with SetEscapeHTML(false)

[wxcsdb88]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8 linux/amd64
and
go version go1.8.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/wangxin/work/golang:/home/wangxin/work/golang/src/unicontract"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build900066718=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

In order to get the same serialize result for json data in java(gson), js(sort keys by alph...) , python(rapaidjson) and golang(encoding/json), I found the golang set the htmlescape flag true and it`s result is not the same as others, so I rewrite the encoder and got the output.

The test code as follows:

func Test_SerializeDisableHtmlEscape(t *testing.T) {
	jsonStr := `{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}`
	default_data := Deserialize(jsonStr)
	default_data_jsonStr := SerializePretty(default_data)
	t.Log("----------default serialize result------------:\n", default_data_jsonStr)

	var buf bytes.Buffer
	enc := json.NewEncoder(&buf)
	enc.SetEscapeHTML(false)
	enc.Encode(jsonStr)

	strJson := buf.String()
	t.Log("disableEscapeHTML and output len is", len(strJson), ",content is:\n", strJson)
	strJson = strings.Replace(strJson, "\\", "", -1)
	t.Log("remove the backslash and output len is", len(strJson), ",content is:\n", strJson)
	strJsonWithOutBacklashLen := len(strJson)
	strJson = strings.Trim(strJson, "\"")
	t.Log("after strings trim, the len is", len(strJson), ",expect is", strJsonWithOutBacklashLen-2, ",content is:\n", strJson)

	defaultJsonStr := "\"{\"host\": \"http://localhost:9090\",\"port\": 9090,\"analytics_file\": \"1>=0 && 3+2 <=5 || 4&2 || 2^1\",\"static_file_version\": 1,\"static_dir\": \"E:/Project/goTest/src/\",\"templates_dir\": \"E:/Project/goTest/src/templates/\",\"serTcpSocketHost\": \":12340\",\"serTcpSocketPort\": 12340,\"fruits\": [\"apple\", \"peach\"]}\""
	t.Log("defaultJsonStr len is ", len(defaultJsonStr))
	defaultJsonStr = strings.Replace(defaultJsonStr, "\\", "", -1)
	strdefaultJsonStrWithOutBacklashLen := len(strJson)
	t.Log("defaultJsonStr remove the backslash and output len is", len(defaultJsonStr), ",content is:\n", defaultJsonStr)
	defaultJsonStr = strings.Trim(defaultJsonStr, "\"")
	t.Log("defaultJsonStr after strings trim, the len is", len(defaultJsonStr), ",expect is", strdefaultJsonStrWithOutBacklashLen-2, ",content is:\n", defaultJsonStr)

}

The fully output as follows:

utils_test.go:136: ----------default serialize result------------:
		 {
			"analytics_file": "1\u003e=0 \u0026\u0026 3+2 \u003c=5 || 4\u00262 || 2^1",
			"fruits": [
				"apple",
				"peach"
			],
			"host": "http://localhost:9090",
			"port": 9090,
			"serTcpSocketHost": ":12340",
			"serTcpSocketPort": 12340,
			"static_dir": "E:/Project/goTest/src/",
			"static_file_version": 1,
			"templates_dir": "E:/Project/goTest/src/templates/"
		}
	utils_test.go:144: disableEscapeHTML and output len is 331 ,content is:
		 "{\"host\": \"http://localhost:9090\",\"port\": 9090,\"analytics_file\": \"1>=0 && 3+2 <=5 || 4&2 || 2^1\",\"static_file_version\": 1,\"static_dir\": \"E:/Project/goTest/src/\",\"templates_dir\": \"E:/Project/goTest/src/templates/\",\"serTcpSocketHost\": \":12340\",\"serTcpSocketPort\": 12340,\"fruits\": [\"apple\", \"peach\"]}"
		
	utils_test.go:146: remove the backslash and output len is 299 ,content is:
		 "{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"
		
	utils_test.go:149: after strings trim, the len is 298 ,expect is 297 ,content is:
		 {"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"
		
	utils_test.go:152: defaultJsonStr len is  298
	utils_test.go:155: defaultJsonStr remove the backslash and output len is 298 ,content is:
		 "{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"
	utils_test.go:157: defaultJsonStr after strings trim, the len is 296 ,expect is 296 ,content is:
		 {"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}

What did you expect to see?

after strings trim, the len is 297 ,expect is 297 ,content is:
		 {"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"

What did you see instead?

after strings trim, the len is 298 ,expect is 297 ,content is:
		 {"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"

[bradfitz]

What is SerializePretty? We can't help you with bugs in your own code.

If there's a bug in the standard library, please report what the standard library is doing (with a code example) and what output you expect.

[wxcsdb88]

Thanks, maybe our usage error, the test code as follows:

func Test_JsonStrTrim(t *testing.T) {
	jsonStr := `{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}`

	var buf bytes.Buffer
	enc := json.NewEncoder(&buf)
	// disabled the HTMLEscape for &, <, and > to \u0026, \u003c, and \u003e in json string
	enc.SetEscapeHTML(false)
	/*--------------- 1. trim the result maybe have the extra double quotation in last--------------------*/
	enc.Encode(jsonStr)

	/*---------------- 2. it`s ok change to interface and encode--------------------------*/
	//var jsonStrData interface{}
	//json.Unmarshal([]byte(jsonStr), &jsonStrData)
	//enc.Encode(jsonStrData)

	strJson := buf.String()
	//t.Log("disableEscapeHTML and output len is", len(strJson), ",content is:\n", strJson)
	strJson = strings.Replace(strJson, "\\", "", -1)
	//t.Log("remove the backslash and output len is", len(strJson), ",content is:\n", strJson)
	strJsonWithOutBacklashLen := len(strJson)
	strJson = strings.Trim(strJson, "\"")
	t.Log("after strings trim, the len is", len(strJson), ",expect is", strJsonWithOutBacklashLen-2, ",content is:\n", strJson)
	// the last double quotation except removed

}

output

the last backslash is not removed, with strings.Trim(...)

after strings trim, the len is 298 ,expect is 297 ,content is:
{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"

expect

after strings trim, the len is 297 ,expect is 297 ,content is:
{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}

[bradfitz]

There's a newline at the end of end of your string after the double quote, so your strings.Trim isn't trimming the double quote.

I don't see a bug in Go here.

[wxcsdb88]

Thanks a lot. I have found that strings.TrimSpace() will ok, before trim the ".