Tips and Tricks

Handling Numbers

Different implementations of MessagePack deal with numbers differently, partly as a consequence of how each source language expresses integers. (Some languages, like Ruby, do not have unsigned integers, while others, like JavaScript, have no integers at all. Meanwhile, Go, C, and C++ all support signed and unsigned integers up to machine word size as distinct types.)

msgp has a built-in type called msgp.Number that can represent a MessagePack int, uint, float32, and float64. If you are decoding objects serialized by other libraries into Go objects using msgp, you may have to use msgp.Number instead of a builtin numeric type.

Slice/Maps of structs

The easiest way to do direct serialization of base types is to add a separate type, and generate code for that.

If you have an array of Foo structs, add a type for the array, add the slice type:

type Foos []Foo

Then the generated code can be called to serialize

package main

import (
	"fmt"

	"github.com/tinylib/msgp/msgp"
)

//go:generate msgp

type Foo struct {
	A string `msg:"a"`
}

type Foos []Foo

func main() {
	// Marshal
	foos := []Foo{{A: "Hello"}, {A: "World"}}
	body, _ := Foos(foos).MarshalMsg(nil)
	fmt.Printf("foos is encoded as %x\n", body)
	
	// Unmarshal
	var dst Foos
	_, _ = dst.UnmarshalMsg(body)
	// Foos can easily be converted back to []Foo if needed.
	fmt.Println([]Foo(dst))
}

A similar approach can be used for maps.

This is not needed when the slice is only serialized as part of another object.

Lazy Serialization

Much like encoding/json, msgp has a type called msgp.Raw that represents serialized MessagePack. Among other things, you can use msgp.Raw for containers and polymorphism. Take the following, for example:

type Data struct {
    Header map[string]string `msg:"header"`
    Body   msgp.Raw          `msg:"body"`
}

The Data object could reasonably be used as a container for any MessagePack object. Later, if you want to de-serialize a concrete object from the raw data, you would simply call (*msgp.Unmarshaler).UnmarshalMsg([]byte(msgp.Raw)).

Worked Example

package main

import (
	"fmt"

	"github.com/tinylib/msgp/msgp"
)

//go:generate msgp

type Foo struct {
	A       string `msg:"a"`
	Another string `msg:"b"`
}

type Data struct {
	Header uint16   `msg:"header"`
	Body   msgp.Raw `msg:"body"`
}

func main() {
	foo1 := Foo{A: "Hello", Another: "World"}
	body, _ := foo1.MarshalMsg(nil)
	fmt.Printf("foo1 is encoded as %x\n", body)
	data := Data{
		Header: 1,
		Body:   body,
	}
	dataPacket, _ := data.MarshalMsg(nil)
	var (
		readPacket Data
		readFoo    Foo
	)
	_, _ = readPacket.UnmarshalMsg(dataPacket)

	if readPacket.Header == 1 {
		_, _ = readFoo.UnmarshalMsg([]byte(readPacket.Body))
		fmt.Println(readFoo)
	}
}

Serializing Foreign Types

Generating methods for a type defined outside the package in which you are currently working presents some challenges, as Go does not support defining methods for a type outside its originating package. There are, however, some workarounds:

• You can specify -file in your //go:generate directive to be a foreign file (located somewhere else in your GOPATH), which will generate methods in the foreign package directly. Distributing such code would probably require maintaining a fork and/or git submodule.
• You can define an identical struct type in your own package, and pointer-cast from the old type to the one defined in your package when you need to serialize.

The replace directive that makes it easier to serialize foreign primitive types. Example with github.com/google/uuid:

package main

import "github.com/google/uuid"

//go:generate msgp

//msgp:replace uuid.UUID with:UUID
type UUID [16]byte

// Or like that
//msgp:replace uuid.UUID with:[16]byte

type User struct {
  ID uuid.UUID
}

Errors

All of the errors returned by the msgp package implement the msgp.Error interface. Thus, it is simple to distinguish between errors returned from encoding/decoding versus errors returned from other sources. Additionally, msgp errors will tell you whether or not the error allows the user to continue reading/writing to a stream. For example:

f, err := r.ReadFloat64()
if err != nil {
    if msgerr, ok := err.(msgp.Error); ok && msgerr.Resumable() {
        // the next object probably isn't a float64
    }
    // something else went wrong
    log.Fatalln("something broke:", err)
}

Omitempty Support

See Zero Values; Omitempty and Allownil for information on how to omit empty (zero value) struct members from the emitted message package output.

TinyGo Support

As of July 2021, the latest revision now supports compilation under TinyGo.

On microcontrollers, the generated methods MarshalMsg and UnmarshalMsg are recommended instead of EncodeMsg/DecodeMsg, because they can operate without buffers or additional allocations. Likewise the functions from the msgp package that operate on byte slices (e.g. msgp.AppendInt32, msgp.ReadInt32Bytes, etc.) will generally result in faster operation, smaller code size and fewer allocations.

Most msgp.AppendXxxx calls will perform no allocation if you provide a sufficiently large buffer. ReadXxxxBytes calls will require allocation if the underlying type does, e.g. msgp.ReadInt32Bytes performs no allocation except in case of error, since the value has a fixed size. But msgp.ReadStringBytes will perform an allocation for the resulting string. In this case, msgp.ReadStringZC is available to access the raw bytes without allocation or copy.