A test case for packing bytes to send over the network.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
benchmarks
.gitignore
license
read.go
readme.md
serial.go
serial_test.go
write.go

readme.md

serial

This package is a proof-of-concept modeling a general purpose serialization strategy after the articles by Glenn Fiedler.

It is intended to demonstrate the serialization pattern described, which aims to tightly pack bytes to minimize transport costs in a UDP communication system, while also providing a single function for both read and write to reduce potential for human error when modifications to the structure are made.

The code has seen several revisions, balancing execution performance, convenience, and the smallest byte packing possible while using the encoding/binary package in LittleEndian. The latest iteration has traded for convenience at the cost of reduced built-in functionality. This is mostly due to the lack of sane alternatives when dealing with variable size types, most especially complex combinations such as maps or unsized slices.

While it can achieve significantly smaller byte sizes than tools like encoding/gob or even github.com/tinylib/msgp, it comes at the expense of spending many hours fine-tuning and writing the serialization logic by hand.

This code comes with a full set of unit tests, and the benchmarks have been cleanly separated to create a more well defined example without mixing up code.

conclusions

With the MTU of only 1500 bytes, every byte counts when trying to create a communication protocol over UDP that does not fragment.

I have to agree with Glenn Fiedler regarding the fact that the best performance can only be achieved by explicitly defining serialization per structure.

However, depending on your use case the msgp package offers absolutely stellar performance and very low memory consumption. It produces results just under 2x the size of manual serialization, but at zero cognitive overhead. Using this lets you focus on the other parts of the application, and fits very nicely within the same overall packet-based strategy.

On the other hand, I cannot recommend the encoding/gob package. It consumes significantly more memory, and sometimes upwards of 6x the amount of bytes due to the amount of metadata generated. Additionally it is more geared towards streaming, and has heavy dependency on a per-instance cache. This model conflicts with packet based UDP, and new instances are an order of magnitude slower to parse or write messages.

To summarize, if you want to focus on the rest of your application just use the msgp package. If you get to a point where you need to optimize, it takes very little effort to exchange it for custom serialization. However, writing custom serialization is very expensive. It is best to plan according to your applications needs.

references