feat: add ast.Visitor for fully decoding JSON into custom generic data containers without using ast.Node

[zhongxinghong]

Background

In our business, we want to decode the whole JSON into our custom containers which are neither golang's struct nor map[string]interface{}. They are look like these and can't be replaced by other types:

type UserNode interface {}

// the following types implement the UserNode interface.
type (
    UserNull    struct{}
    UserBool    struct{ Value bool }
    UserInt64   struct{ Value int64 }
    UserFloat64 struct{ Value float64 }
    UserString  struct{ Value string }
    UserObject  struct{ Value map[string]UserNode }
    UserArray   struct{ Value []UserNode }
)

There are two ways to meet our needs if we use sonic:

1. Use sonic.Unmarshal() to decode JSON into map[string]interface{} and then use a custom method to recursively convert interface{} to our UserNode types.
2. Decode JSON into sonic/ast.Node types and then use another custom method to recursively convert ast.Node to our UserNode types.

We've also designed a custom JSON library to directly decode JSON into our UserNode types without any IR (intermediate representation: interface{} / reflect.Value / ...). It's based on encoding/json with some jsoniter-go's optimization but it's even faster than the second way above. (see benchmark in appendix)

Sonic/ast.Node is designed for processing partial JSON and its APIs are similar to gjson/sjson. It's faster than using map[string]interface{} as IR for decoding generic data. But it is also a kind of IR after all and have lazy-load design, which are useless in our case but causes performance loss. So we want to introduce a Node-free decoder to speed up this case.

Changes

1. Introduce the ast.Visitor interface to represents a handler during preorder-traversal of a JSON AST.
2. Introduce the ast.Preorder() method based on ast.Parser to preform the preorder-traversal.
3. Add a demo Visitor implementation in ast/visitor_test.go to preform the unit-test / benchmark.

// Visitor handles the callbacks during preorder traversal of a JSON AST.
//
// According to the JSON RFC8259, a JSON AST can be defined by
// the following rules without seperator / whitespace tokens.
//
//  JSON-AST  = value
//  value     = false / null / true / object / array / number / string
//  object    = begin-object [ member *( member ) ] end-object
//  member    = string value
//  array     = begin-array [ value *( value ) ] end-array
//
type Visitor interface {

    // OnNull handles a JSON null value.
    OnNull() error

    // OnTrue handles a JSON true value.
    OnTrue() error

    // OnFalse handles a JSON false value.
    OnFalse() error

    // OnString handles a JSON string value.
    OnString(v string) error

    // OnNumber handles a JSON number value with its type after conversion.
    //
    // For a valid JSON, the v.Int64() method should parse the JSON number
    // correctly and return no error if isInt64 is true, otherwise
    // the v.Float64() method should work.
    OnNumber(v json.Number, isInt64 bool) error

    // OnObjectBegin handles the beginning of a JSON object value with a
    // suggested capacity that can be used to make your custom object container.
    //
    // After this point the visitor will receive a sequence of callbacks like
    // [string, value, string, value, ......, ObjectEnd].
    //
    // Notice that this is a recursive definition which means the value can
    // also be a JSON object / array described by a sequence of callbacks.
    OnObjectBegin(capacity int) error

    // OnObjectKey handles a JSON object key string in member.
    OnObjectKey(key string) error

    // OnObjectEnd handles the ending of a JSON object value.
    OnObjectEnd() error

    // OnArrayBegin handles the beginning of a JSON array value with a
    // suggested capacity that can be used to make your custom array container.
    //
    // After this point the visitor will receive a sequence of callbacks like
    // [value, value, value, ......, ArrayEnd].
    //
    // Notice that this is a recursive definition which means the value can
    // also be a JSON object / array described by a sequence of callbacks.
    OnArrayBegin(capacity int) error

    // OnArrayEnd handles the ending of a JSON array value.
    OnArrayEnd() error
}

Discussions

1. There's still another style to pass the preorder-traversal to user program like simdjson by use cpp-style iterator, but we finally use visitor callback style. The reasons are:
   • The iterator has to record the decoder's context and switch to user program after every Next() calls and recover decoder context to continue the processing in subsequent Next() call. This causes additional performance overhead.
   • The iterator-style decoder's code is not look like the existing ast.Parser code, which brings additional maintenance costs.
2. The Visitor interface is nearly unchangeable after released and might cause some compatibility problem in the future.

Appendix

Go Version

$ go version
go version go1.18.10 darwin/amd64

Benchmark ast.Node vs ast.Visitor

$ cd ast/
$ go test . -run=none -benchmem -bench BenchmarkVisitor -benchtime=100000x
Begin GC looping...
goos: darwin
goarch: amd64
pkg: github.com/bytedance/sonic/ast
cpu: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
BenchmarkVisitor_UserNode/AST-8                   100000            314353 ns/op          41.43 MB/s       97272 B/op        540 allocs/op
BenchmarkVisitor_UserNode/Visitor-8               100000            268001 ns/op          48.59 MB/s       44273 B/op        405 allocs/op
PASS
ok      github.com/bytedance/sonic/ast  58.519s

Benchmark in our business (use our dataset)

goos: darwin
goarch: amd64
pkg: ******
cpu: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
BenchmarkDecode_Case01_Sonic
BenchmarkDecode_Case01_Sonic/sonic
BenchmarkDecode_Case01_Sonic/sonic-8                 200          46148747 ns/op           0.37 MB/s    34555524 B/op     448516 allocs/op
BenchmarkDecode_Case01_Sonic/sonic_ast_node
BenchmarkDecode_Case01_Sonic/sonic_ast_node-8                200          40603735 ns/op           0.42 MB/s    43147743 B/op     287744 allocs/op
BenchmarkDecode_Case01_Sonic/sonic_ast_visitor
BenchmarkDecode_Case01_Sonic/sonic_ast_visitor-8             200          26099894 ns/op           0.65 MB/s    16027529 B/op     229515 allocs/op
BenchmarkDecode_Case01
BenchmarkDecode_Case01/ejson
BenchmarkDecode_Case01/ejson-8                               200          69715883 ns/op           0.24 MB/s    43855800 B/op     679455 allocs/op
BenchmarkDecode_Case01/ijson
BenchmarkDecode_Case01/ijson-8                               200          63586936 ns/op           0.27 MB/s    38632656 B/op     884192 allocs/op
BenchmarkDecode_Case01/tjson
BenchmarkDecode_Case01/tjson-8                               200          31347045 ns/op           0.54 MB/s    20710638 B/op     471284 allocs/op
PASS

• sonic: string --(sonic.UnmarshalFromString)--> map[string]interface{} --(convert)--> UserNode
• sonic_ast_node: string --(sonic.GetFromString / node.LoadAll)--> ast.Node --(convert)--> UserNode
• sonic_ast_visitor: string --(sonic/ast.Preorder)--> UserNode
• ejson: []byte --(encoding/json.NewDecoder, UseNumber)--> map[string]interface{} --(convert)--> UserNode
• ijson: string --(jsoniter-go.NewDecoder, UseNumber)--> map[string]interface{} --(convert)--> UserNode
• tjson: []byte --(our custom JSON library)--> UserNode

NOTE:

• In sonic_ast_visitor, we ignore the capacity from Visitor.OnObjectBegin / Visitor.OnArrayBegin and use capacity = 0. This is different from BenchmarkVisitor_UserNode/Visitor.

[zhongxinghong]

I will perfect the README after CR :)

[liuq19]

Thanks, we will review it later

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (main@6108485). Click here to learn what that means.
The diff coverage is n/a.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@           Coverage Diff           @@
##             main     #471   +/-   ##
=======================================
  Coverage        ?   77.92%           
=======================================
  Files           ?       63           
  Lines           ?    10565           
  Branches        ?        0           
=======================================
  Hits            ?     8233           
  Misses          ?     1971           
  Partials        ?      361           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[AsterDY]

looks great！