encoding/json: json.Decoder Token method is two times slow than Decode

[bigpigeon]

question

I need Depth limit decode function with json data

but std json did not support, I want to use json.Decoder.Token to implement it

performance test

I write below benchmark to test it's performance

https://gist.github.com/bigpigeon/93f90fe84cf96f10356541c22db6e7dc

# go test -test.bench ".*" decode_test.go 
goos: linux
goarch: amd64
cpu: Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz
BenchmarkJsonDecode-6   	  331059	      4878 ns/op	    3984 B/op	      44 allocs/op
BenchmarkGetToken-6     	  146536	      7969 ns/op	    3064 B/op	     135 allocs/op
PASS
ok  	command-line-arguments	2.907s

the getAllToken is even slower than Decode

What version of Go are you using (go version)?

$ go version
go version go1.19.1 linux/amd64

[dmitshur]

CC @dsnet, @bradfitz, @mvdan.

[dsnet]

Closing as a duplicate of #40128. A huge part of it's slowness is the fact that it allocates.

[dsnet]

On a different note, if you want to a propose a Decoder.SetDepthLimit method, that can filed separately and can be considered.