encoding/json: misleading error for end-of-input in an escape sequence

[LukeShu]

What version of Go are you using (go version)?

$ go version
go version go1.20.1 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/lukeshu/.cache/go-build"
GOENV="/home/lukeshu/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/lukeshu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/lukeshu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/run/user/1000/tmpdir/go-build3465712561=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I fed the JSON parser a truncated JSON document that has a backslash as the final rune.

package main

import (
	"encoding/json"
	"fmt"
)

func main() {
	var obj any
	err := json.Unmarshal([]byte(`"\`), &obj)
	fmt.Println(err)
}

https://go.dev/play/p/KxvQ6lpnT7I

What did you expect to see?

I expected the error to be unexpected end of JSON input

What did you see instead?

I saw the error invalid character ' ' in string escape code, which is a nonsense error as there is no character in the input at all!

json.Compact and json.Indent have the same issue, while json.Decoder.Decode correctly returns io.ErrUnexpectedEOF.

[thanm]

@dsnet @bradfitz @mvdan per owners.

[hafus]

I think the root case of the issue is the function below:

func (s *scanner) eof() int {
	if s.err != nil {
		return scanError
	}
	if s.endTop {
		return scanEnd
	}
	s.step(s, ' ')
	if s.endTop {
		return scanEnd
	}
	if s.err == nil {
		s.err = &SyntaxError{"unexpected end of JSON input", s.bytes}
	}
	return scanError
}

I do update s.err == nil to be s.err != nil but causes some test cases to fail

[mvdan]

A similar error in the past: #56332

There are a handful of these "confusing error message" bug reports on encoding/json. My intuition is that we shouldn't try to fix each one of them individually. cc @dsnet

[dsnet]

I have an experimental CL in progress that completely switches the "encoding/json" parser to a new implementation, which will fix most (if not all) of these weird error cases.

[LukeShu]

I guess I should mention that I'm finding these issues by fuzzing my own JSON parser against encoding/json. My implementation is GPL, but I'm 100% willing to re-license it for incorporation in to the stdlib if there's anything that you'd like to steal.

In particular, compat/json/compat_test.go is some tests that may be worth stealing; they are tests that I felt necessary to add even though my implementation already passed encoding/json's existing test suite (which I have copied in the borrowed_*_test.go files).