perf(*): replace gob with msgpack

[vm-001]

Summary

The Gob has a limitation regarding the arbitrary types serialization (such as map[string]interface{}), and this restricts us from using map[string]interface{} in entities.

The MessagePack is a binary serialization, and it has better performance than Gob. (see benchmarks)

This PR replaced Gob with MessagePack. All the cache keys are changed to a new format (versioning key) to avoid breaking changes.

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller.

References

• https://msgpack.org/

benchmarks

test

import (
	"fmt"
	"testing"
	"time"
)

type User struct {
	Username string                 `json:"username"`
	Password string                 `json:"password"`
	Age      int                    `json:"age"`
	Enabled  bool                   `json:"enabled"`
	Metadata map[string]interface{} `json:"metadata"`
}

var user = User{
	Username: "root",
	Password: "secret",
	Age:      20,
	Enabled:  true,
	Metadata: map[string]interface{}{
		"hello": "world",
		"int":   10,
	},
}

var n = 1000000

func TestJson(t *testing.T) {
	fmt.Println("=========== testing json ===========")
	var bytes []byte
	var err error
	fmt.Println("===>>> serializer")
	now := time.Now()
	for i := 0; i < n; i++ {
		bytes, err = JSON.Serialize(user)
	}
	fmt.Println("total time :", time.Since(now).Milliseconds(), "ms")
	if err != nil {
		panic(err)
	}

	fmt.Println("size: ", len(bytes))
	//fmt.Println(string(b))

	fmt.Println("===>>> deserialize")
	now = time.Now()
	var output User
	for i := 0; i < n; i++ {
		err = JSON.Deserialize(bytes, &output)
	}
	if err != nil {
		panic(err)
	}
	fmt.Println("total time :", time.Since(now).Milliseconds(), "ms")
	fmt.Printf("%+v\n", output)
	fmt.Println("=========== testing json ===========")
	fmt.Println()
	fmt.Println()
}

func TestGob(t *testing.T) {
	fmt.Println("=========== testing gob ===========")
	var bytes []byte
	var err error
	fmt.Println("===>>> serializer")
	now := time.Now()
	for i := 0; i < n; i++ {
		bytes, err = Gob.Serialize(user)
	}
	fmt.Println("total time :", time.Since(now).Milliseconds(), "ms")
	if err != nil {
		panic(err)
	}

	fmt.Println("size: ", len(bytes))

	fmt.Println("===>>> deserialize")
	now = time.Now()
	var output User
	for i := 0; i < n; i++ {
		err = Gob.Deserialize(bytes, &output)
	}
	if err != nil {
		panic(err)
	}
	fmt.Println("total time :", time.Since(now).Milliseconds(), "ms")
	fmt.Printf("%+v\n", output)
	fmt.Println("=========== testing gob ===========")
	fmt.Println()
	fmt.Println()
}

func TestMsgpack(t *testing.T) {
	fmt.Println("=========== testing msgpack ===========")
	var bytes []byte
	var err error
	fmt.Println("===>>> serializer")
	now := time.Now()
	for i := 0; i < n; i++ {
		bytes, err = MsgPack.Serialize(user)
	}
	fmt.Println("total time :", time.Since(now).Milliseconds(), "ms")
	if err != nil {
		panic(err)
	}

	fmt.Println("size: ", len(bytes))

	fmt.Println("===>>> deserialize")
	now = time.Now()
	var output User
	for i := 0; i < n; i++ {
		err = MsgPack.Deserialize(bytes, &output)
	}
	if err != nil {
		panic(err)
	}
	fmt.Println("total time :", time.Since(now).Milliseconds(), "ms")
	fmt.Printf("%+v\n", output)
	fmt.Println("=========== testing msgpack ===========")
	fmt.Println()
}

output:

=========== testing json ===========
===>>> serializer
total time : 301 ms
size:  101
===>>> deserialize
total time : 807 ms
{Username:root Password:secret Age:20 Enabled:true Metadata:map[hello:world int:10]}
=========== testing json ===========


=========== testing gob ===========
===>>> serializer
total time : 1548 ms
size:  176
===>>> deserialize
total time : 6629 ms
{Username:root Password:secret Age:20 Enabled:true Metadata:map[hello:world int:10]}
=========== testing gob ===========


=========== testing msgpack ===========
===>>> serializer
total time : 239 ms
size:  72
===>>> deserialize
total time : 371 ms
{Username:root Password:secret Age:20 Enabled:true Metadata:map[hello:world int:10]}
=========== testing msgpack ===========

conclusion

msgpack > json > gob

[codecov]

Codecov Report

❌ Patch coverage is 93.18182% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/serializer/msgpack.go	83.33%	1 Missing and 1 partial ⚠️
app/app.go	83.33%	1 Missing ⚠️

Flag	Coverage Δ
integration	74.45% <93.18%> (+0.05%)	⬆️
integration-o11	43.51% <77.27%> (+0.06%)	⬆️
unit	13.14% <0.00%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
constants/cache_key.go	100.00% <100.00%> (ø)
db/dao/attempt_dao.go	100.00% <100.00%> (ø)
db/dao/attempt_detail_dao.go	92.30% <100.00%> (ø)
db/dao/dao.go	91.25% <100.00%> (ø)
db/dao/endpoint_dao.go	100.00% <100.00%> (ø)
db/dao/event_dao.go	85.36% <100.00%> (ø)
db/dao/plugin_dao.go	100.00% <100.00%> (ø)
db/dao/source_dao.go	100.00% <100.00%> (ø)
db/dao/workspace_dao.go	100.00% <100.00%> (ø)
pkg/cache/redis.go	65.21% <100.00%> (ø)
... and 2 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.