SerLib

SerLib is a fast, and easy to use serialisation C++ library. The whole library is designed with performance, and ease of use in mind.

Status

Type	Status
Linux & OSX
Windows
Coverage

Features

• Fast: We run our benchmarks on every build to ensure the maximum performance.
• Reliable: We have an extensive testing suite, making sure we do not break forward and backward compatibility.
• Easy to use: The library is completely header only, and does not require extra compile steps nor does it take over your classes before you can use it.
• Bindable: The whole library is designed to be used from a c-interface, so when needed, it can be called from other languages.
• Cross platform: Designed to only use c++2003, with no external dependencies.
• Format is backward and forward compatible, using variable indexes.
• Message usage can be completely streamed, and does not require the full message to start reading.
• Compact: The binary format is very small, even when no compression is used.
• Warning free, compiled on the highest warning level.

#ZPM Installation

In .package.json

"requires": [
		{
			"name": "Zefiros-Software/SerLib",
			"version": "^1.0.0"
		}
]

In premake5.lua

zpm.uses "Zefiros-Software/SerLib"

Manual Installation

The whole library is header only, so for usage in your own code you should clone the repository. And add serialisation/include/ as include path. After this you can use the include #include "serialisation/serialisation.h" in your code to start using the library.

Alternatively if you thinks your projects compiles too slow using this header only library, you can define SERIALISATION_NO_HEADER_ONLY and add the C++ files in serialisation/src to your compile files, along the above stated include directory.

Usage

The library interface only uses 4 functions, each with a different purpose:

• Message::Store([reference], [index]); Stores primitives and objects.
• Message::StoreParent([reference], [index]); Stores the variables from the given inherited class.
• Message::StoreContainer([reference], [index]); Stores container formats.
• MessageHelper::Store([message], [reference]); Begins a store or load sequence.

To ensure backward and forward compatibility, each Store should be accompanied with a variable index. So when a variable is removed, or added, this induces no problems. Note that each index in the classes should be unique.

Example

Say we start with:

class Foo
{
public:
	void OnStore( Message &message )
	{
		message.Store(mVar1, 0);
		message.Store(mVar2, 1);
		message.Store(mVar3, 2);
	}
	
private:
	uint32_t mVar1, mVar2;
	double mVar3;
};

Revision

However when we decide mVar2 is not needed, and we need an extra double we can add it as such:

class Foo
{
public:

	void OnStore( Message &message )
	{
		message.Store(mVar1, 0);
		//message.Store(mVar2, 1);
		message.Store(mVar3, 2);
		message.Store(mVar4, 3);
	}
	
private:
	uint32_t mVar1;
	double mVar3, mVar4;
};

A good practice would be to un-comment deleted Stores, so the index does not get reused on accident!

Primitives

The following types are considered primitives:

• (u)int8_t
• (u)int16_t
• (u)int32_t
• (u)int64_t
• float
• double
• std::string

Example

class Foo
{
public:

	void OnStore( Message &message )
	{
		message.Store(mVar1, 0);
		message.Store(mVar2, 1);
		message.Store(mVar3, 2);
	}
	
private:
	uint32_t mVar1, mVar2;
	double mVar3;
};

Objects

There are several ways to store objects:

• Using templates
• Using interfaces
• Using template specialisation

The following are all equivalent:

class Vec3Impl1
{
public:
	
	void OnStore( Message &message )
	{
		message.Store(mX, 0);
		message.Store(mY, 1);
		message.Store(mZ, 2);
	}
	
private:
	
	double mX, mY, mZ;
}


class Vec3Impl2
	: public ISerialisable
{
public:
	
	void OnStore( Message &message )
	{
		message.Store(mX, 0);
		message.Store(mY, 1);
		message.Store(mZ, 2);
	}
	
private:
	
	double mX, mY, mZ;
}


struct Vec3Impl3
{
	double x, y, z;
}

template<>
void MessageHelper::OnStore( Vec3Impl3 &serialisable, Message &message )
{
	message.Store( serialisable.x, 0 );
	message.Store( serialisable.y, 1 );
	message.Store( serialisable.z, 2 );
}

Example

Storing an object from another object:

class Vec3
{
public:
	
	void OnStore( Message &message )
	{
		message.Store(mX, 0);
		message.Store(mY, 1);
		message.Store(mZ, 2);
	}
	
private:
	
	double mX, mY, mZ;
}

class Foo
{
public:

	void OnStore( Message &message )
	{
		message.Store(mVar1, 0);
		message.Store(mVar2, 1);
	}
	
private:
	Vec3 mVar1, mVar2;
};

Containers

Containers use the StoreContainer interface, and can store both primitives and objects. Currently the following containers are supported:

• std::vector<>
• std::array<> (c++11 only)
• c-arrays
• Any std-iterator compatible container format

Currently not yet supported:

• Associative containers; (st::set<>, std::map<>)
• Unordered associative containers

class Vec3
{
public:
	void OnStore( Message &message )
	{
		message.Store(mX, 0);
		message.Store(mY, 1);
		message.Store(mZ, 2);
	}
	
private:
	
	double mX, mY, mZ;
}

class Foo
{
public:
	void OnStore( Message &message )
	{
		message.StoreContainer(mVar1, 0);
		message.StoreContainer(mVar2, 1);
	}
	
private:
	std::vector< Vec3 > mVar1, mVar2;
};

Inherritance

As you may have noticed we haven't touched the topic of inheritance yet. To be able to effectively use inherritance, without having to spend your precious variable indices, we can use the StoreParent interface. The parents use their own index system.

Example

class Vec3
{
public:
	
	void OnStore( Message &message )
	{
		message.Store(mX, 0);
		message.Store(mY, 1);
		message.Store(mZ, 2);
	}
	
private:
	
	double mX, mY, mZ;
}

class Vec4
	: public Vec3
{
public:
	
	void OnStore( Message &message )
	{
		message.StoreParent<Vec3>(*this, 0);
		message.Store(mW, 0);
	}
	
private:
	
	double mW;
}

class Foo
{
public:

	void OnStore( Message &message )
	{
		message.StoreContainer(mVar1, 0);
		message.StoreContainer(mVar2, 1);
	}
	
private:
	std::vector< Vec4 > mVar1, mVar2;
};

Messages

Our message both serialises, and deserialises so we need to construct the message object correctly.

class Vec3
{
public:
	
	void OnStore( Message &message )
	{
		message.Store(mX, 0);
		message.Store(mY, 1);
		message.Store(mZ, 2);
	}
	
private:
	
	double mX, mY, mZ;
}

void main()
{
	Vec3 vec;
	std::stringstream ss;
	
	Message serMessage( ss, Format::Binary, Mode::Serialise );
	
	MessageHelper::Store( serMessage, vec );
	
	Message deserMessage( ss, Format::Binary, Mode::Deserialise );
	MessageHelper::Store( deserMessage, vec );
}

Flexibility

Don't like OnStore?

If you dislike the name or if it is not in your style, you can change it to your needs.

#define SERIALISATION_CUSTOM_INTERFACE on_store

After this we changed the usage of the library to:

#include "serialisation/serialisation.h"

class Foo
{
public:

	void on_store( Message &message )
	{
		message.Store(mVar1, 0);
	}
	
private:
	uint32_t mVar1;
};

SerLib Limits

• Each class can use up to 27 indices, so we can store 27 separate variables max.
• Each class can have up to 3 parents, we can only use 3 indices for parents.

Planned Features

• Extensive documentation, on both design and usage.
• Compression, based on array type and with delta encoding.
  • FastPFOR integer compression.
  • FFPFOR for floating point compression.
• Static container serialisation. This feature will disable backward and forward compatibility on static containers, but doing so will improve the size and serialisation speed. All types will be stored and grouped by their own types.
• Message migrations, migrating your old message format to your new message formats with nice abstraction.
• Big endian platform support. (Only the runtime is not yet compatible)
• Pointer Patching: Creating objects based off reflection, and make sure references are kept between sessions.
• Reflection based serialisation: Define the outlay of your classes using a reflection library, which then can be used for serialisation.
• XML and JSON serialisation, based on reflection. No need for adjustments, and also makes conversion between binary and text formats possible.
• Extensive custom error handling.

Alternatives

• Boost Serialisation
• Protocol Buffers
• Cap'n Proto

Why SerLib?

We wanted an easy to use, non invasive serialisation library. Unfortunately no library exists that did implement our requirements.

SerLib vs Boost Serialisation

• Boost is a huge dependency, SerLib is header only.
• Boost uses operator overloading, that makes it non bindable. SerLib is c-interface compatible.
• Boost takes over your classes to work; SerLib can use already existing classes.

SerLib vs Protocol Buffers

• Protocol Buffers uses a pre-compiler. SerLib is dynamic and header only.
• Protocol Buffers has issues on non unix systems. SerLib is simple c++2003, and is cross-platform.

Bugs

When a bug is found please insert it in the issue tracker, so we can resolve it as quickly as we can.

Contributing

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request

Authors

• Mick van Duijn m.vanduijn@zefiros.eu
• Paul Visscher p.e.visscher@zefiros.eu
• Koen Visscher k.m.visscher@zefiros.eu

Used By

We are interested to find out what projects use SerLib. We would love it to include your projects here, just shoot us a mail. :)

• Zefiros Engine

License

This project is licensed under the MIT license by Zefiros Software.

Copyright (c) 2017 Zefiros Software.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.