🔌 tinybuf  

Compressed, static-typed binary buffers in HTML5 / Node.js

• 🚀 Designed for real-time HTML5 games (via geckos.io, peer.js or socket.io)
• 🗜️ Lossless and lossy compression, up to ~50% smaller than FlatBuffers or Protocol Buffers
• ✨ Out-of-the-box boolean packing, 16-bit floats, 8-bit scalars, and more
• 🚦 Compile-time safety & runtime validation

tinybuf is safe for use with property mangling & code minification like terser

Why?

tinybuf is small, fast and extensible. Unlike FlatBuffers and Protocol Buffers - which focus on cross-platform languages, limited encoding choices, and generated code - tinybuf is focused soley on fast, native serialization to compressed formats. See comparison table.

Sample Usage

Easily send and receive custom binary formats.

Define formats:

import { encoder, Type } from 'tinybuf';

const GameWorldState = encoder({
  time: Type.UInt,
  players: [{ /* ... */ }]
});

Sending:

// Encode:
const bytes = GameWorldState.encode(myWorld);

Receiving:

// Decode:
const myWorldData = GameWorldState.decode(bytes);

Receiving (many):

import { decoder } from 'tinybuf';

// Create a decoder:
const myDecoder = decoder()
  .on(GameWorldState, (data) => myWorld.update(data))
  .on(ChatMessage, (data) => myHud.onChatMessage(data));

// Handle incoming:
myDecoder.processBuffer(bytes);

Getting Started

Everything you need to quickly encode and decode strongly-typed message formats.

The only requirement for tinybuf is that encoding formats are known by clients, servers and/or peers. You should define encoding formats in some shared module.

Then all you need is:

1. encoder (+types): Define flexible, static-typed encoding formats
2. decoder: Parse incoming binary in registered formats
3. Compression/serialization: Various tips & techniques for making data small

For more information on additional pre/post-processing rules, check out Validation and Transforms.

Installation

# npm
npm install tinybuf

# yarn
yarn add tinybuf

Usage

Define formats

Create an encoding format like so:

import { encoder, Type, Optional } from 'tinybuf';

// Define your format:
const GameWorldData = encoder({
  time: Type.UInt,
  players: [{
    id: Type.UInt,
    isJumping: Type.Boolean,
    position: Optional({
      x: Type.Float,
      y: Type.Float
    })
  }]
});

Then call encode() to turn it into binary (as ArrayBuffer).

// Encode:
const bytes = GameWorldData.encode({
  time: 123,
  players: [
    {
       id: 44,
       isJumping: true,  
       position: {
         x: 110.57345,
         y: -93.5366
       }
    }
  ]
});

bytes.byteLength
// 14

And you can also decode it directly from the encoding type.

// Decode:
const data = GameWorldData.decode(bytes);

Inferred types

The encoder will automatically infer the types for encode() and decode() from the schema provided (see the Types section below).

For example, the return type of GameWorldData.decode(...) from the above example, is:

// data:
{
  time: number,
  players: Array<{
    id: string,
    health: number,
    isJumping: boolean,
    position?: { x: number, y: number } | undefined
  }>
}

Using

You can also use the Decoded<typeof T> helper to add inferred types to any custom method/handler:

import { Decoded } from 'tinybuf';

function updateGameWorld(data: Decoded<typeof GameWorldData>) {
  // e.g. Access `data.players[0].position?.x`
}

Types

Serialize data as a number of lossless (and lossy!) data types

Type	Inferred JavaScript Type	Bytes	About
Type.Int	number	1-8*	Integer between -Number.MAX_SAFE_INTEGER and Number.MAX_SAFE_INTEGER.
Type.Int8	number	1	Integer between -127 to 128.
Type.Int16	number	2	Integer between -32,767 to 32,767.
Type.Int32	number	4	Integer between -2,147,483,647 to 2,147,483,647.
Type.UInt	number	1-8#	Unsigned integer between 0 and Number.MAX_SAFE_INTEGER.
Type.UInt8	number	1	Unsigned integer between 0 and 255.
Type.UInt16	number	2	Unsigned integer between 0 and 65,535.
Type.UInt32	number	4	Unsigned integer between 0 and 4,294,967,295.
Type.Scalar	number	1	Signed scalar between -1.0 and 1.0.
Type.UScalar	number	1	Unsigned scalar between 0.0 and 1.0.
Type.Float64 / Type.Double	number	8	Default JavaScript number type. A 64-bit "double" precision floating point number.
Type.Float32 / Type.Float	number	4	A 32-bit "single" precision floating point number.
Type.Float16 / Type.Half	number	2	A 16-bit "half" precision floating point number. Important Note: Low decimal precision. Max. large values ±65,500.
Type.String	string	1† + n	A UTF-8 string.
Type.Boolean	boolean	1	A single boolean.
Type.BooleanTuple	boolean[]	1¶	Variable-length array/tuple of boolean values packed into 1¶ byte.
Type.Bitmask8	boolean[]	1	8 booleans.
Type.Bitmask16	boolean[]	2	16 booleans.
Type.Bitmask32	boolean[]	4	32 booleans.
Type.JSON	any	1† + n	Arbitrary JSON data, encoded as a UTF-8 string.
Type.Binary	ArrayBuffer	1† + n	JavaScript ArrayBuffer data.
Type.RegExp	RegExp	1† + n + 1	JavaScript RegExp object.
Type.Date	Date	8	JavaScript Date object.
Optional(T)	T | undefined	1	Any optional field. Use the Optional(...) helper. Array elements cannot be optional.
[T]	Array<T>	1† + n	Use array syntax. Any array.
{}	object	none	Use object syntax. No overhead to using object types. Buffers are ordered, flattened structures.

^*Int is a variable-length integer ("varint") which encodes <±64 = 1 byte, <±8,192 = 2 bytes, <±268,435,456 = 4 bytes, otherwise = 8 bytes.

^#UInt is a variable-length unsigned integer ("varuint") which encodes <128 = 1 byte, <16,384 = 2 bytes, <536,870,912 = 4 bytes, otherwise = 8 bytes.

^†Length of payload bytes as a UInt. Typically 1 byte, but could be 2-8 bytes for very large payloads.

^¶2-bit overhead: 6 booleans per byte (i.e. 9 booleans would require 2 bytes).

🗜️ Compression and Serialization

tinybuf comes with powerful encoding types & transforms to make data tiny

It is strongly advised that you don't start with optimizing compression right away. 80% of the win comes just from binary encoding in the first place. Consider revisiting as needed only.

It is highly recommended to read the materials by Glenn Fiedler on Serialization Strategies: Serializing Floating Point Values and State Synchronization: Quantize Both Sides.

Serializing Floats

In JavaScript, all numbers are stored as 64-bit (8-byte) floating-point numbers (or "floats"). These take up a whopping 8 bytes each!

Most of the meaningful gains will come out of compressing floats, including those in 2D or 3D vectors and quaternions. You can compress all visual-only quantities without issue - i.e. if you are using Snapshot Compression Netcode, or updating elements of a HUD.

Quantizing Physics

If you are running a deterministic physics simulation (i.e. State Synchronization / Rollback Netcode), you may need to apply the same quantization to your physics simulation to avoid desynchronization issues or rollback "pops".

Or as Glenn Fiedler suggests, apply the deserialized state on every phyiscs update() as if it had come over the network:

update() {
  // Do physics updates...

  // Quantize:
  const serialized = GameWorldFormat.encode(this.getState());
  const deserialized = GameWorldFormat.decode(serialized);
  this.setState(deserialized);
}

Or for simple cases, you can apply the rounding function to the physics simulation:

update() {
  // Do physics updates...

  // Quantize:
  quantize();
}

quantize() {
  for (const entity of this.worldEntities) {
    // Round everything to the nearest 32-bit representation:
    entity.position.set( Math.fround(player.position.x), Math.fround(player.position.y) );
    entity.velocity.set( Math.fround(player.velocity.x), Math.fround(player.velocity.y) );
  }
}

For reference here are the is a list of the various quantization (rounding) functions for each number type:

Type	Bytes	Quantization function	Use Cases
Type.Float64	8	n/a	Physics values.
Type.Float32	4	Math.fround(x)	Visual values, physics values.
Type.Float16	2	fround16(x)	Limited visual values, limited physics values - i.e. safe for numbers in the range ±65,504, with the smallest precision ±0.00011839976.
Type.Scalar	1	scalarRound(x)	Player inputs - e.g. analog player input (joystick). Values from -1.00 to 1.00.
Type.UScalar	1	uScalarRound(x)	Visual values - e.g. a health bar. Values from 0.00 to 1.00.
Type.Int	1-2*	Math.round(x)	Visual values. *Up to 4-8 bytes for larger values (see Types).

Custom Transforms

You can combine the above built-ins with transforms (see Transforms) to acheive really meaningful compression.

In the following example, we have a myRotation value which is given in absolute radians between 0 and 2π (~6.28319). If we tried to send this as a plain 16-bit float, we would lose a *LOT* of precision, and the rotation would come out visually jerky on the other end.

What we could do instead is set custom transforms that utilize much more of the safe range for 16-bit floats (±65,504):

// Example transform functions that boosts precision by x20,000 by putting
// values into the range ±~62,832, prior to serializing as a 16-bit float.
const toSpecialRange = x => (x * 20_000) - 62_832;
const fromSpecialRange = x => (x + 62_832) / 20_000;

const MyState = encoder({
  myRotation: Type.Float16
})
  .setTransforms({ myRotation: [ toSpecialRange, fromSpecialRange ]});

✨ Parsing formats

By default, each encoder encodes a 2-byte identifier based on the shape of the data.

You can explicitly set Id in the encoder(Id, definition) to any 2-byte string or unsigned integer (or disable entirely by passing null).

Use Decoder

Handle multiple binary formats at once using a decoder:

import { decoder } from 'tinybuf';

const myDecoder = decoder()
  .on(MyFormatA, data => onMessageA(data))
  .on(MyFormatB, data => onMessageB(data));

// Trigger handler (or throw UnhandledBinaryDecodeError)
myDecoder.processBuffer(binary);

Note: Cannot be used with formats where Id was disabled.

Manual handling

You can manually read message identifers from incoming buffers with the static function BinaryCoder.peekIntId(...) (or BinaryCoder.peekStrId(...)):

import { BinaryCoder } from 'tinybuf';

if (BinaryCoder.peekStrId(incomingBinary) === MyMessageFormat.Id) {
  // Do something special.
}

💥 Id Collisions

By default Id is based on a hash code of the encoding format. So the following two messages would have identical Ids:

const Person = encoder({
  firstName: Type.String,
  lastName: Type.String
});

const FavoriteColor = encoder({
  fullName: Type.String,
  color: Type.String
});

NameCoder.Id === ColorCoder.Id
  // true

If two identical formats with different handlers is a requirement, you can explicitly set unique identifiers.

const Person = encoder(1, {
  firstName: Type.String,
  lastName: Type.String
});

const FavoriteColor = encoder(2, {
  fullName: Type.String,
  color: Type.String
});

Identifiers can either be a 2-byte string (e.g. 'AB'), an unsigned integer (0 -> 65,535).

✨ Validation / Transforms

Validation

The great thing about binary encoders is that data is implicitly type-validated, however, you can also add custom validation rules using setValidation():

const UserMessage = encoder({
  uuid: Type.String,
  name: Optional(Type.String),
  // ...
})
.setValidation({
  uuid: (x) => {
    if (!isValidUUIDv4(x)) {
      throw new Error('Invalid UUIDv4: ' + x);
    }
  }
});

Transforms

You can also apply additional encode/decode transforms.

Here is an example where we're stripping out all whitespace:

const PositionMessage = encoder({ name: Type.String })
  .setTransforms({ name: a => a.replace(/\s+/g, '') });

let binary = PositionMessage.encode({ name: 'Hello  There' })
let data = PositionMessage.decode(binary);

data.name
  // "HelloThere"

Unlike validation, transforms are applied asymmetrically.

The transform function is only applied on encode(), but you can provide two transform functions.

Here is an example which cuts the number of bytes required from 10 to 5:

const PercentMessage = encoder(null, { value: Type.String })
  .setTransforms({
    value: [
      (before) => before.replace(/\$|USD/g, '').trim(),
      (after) => '$' + after + ' USD'
    ]
  });

let binary = PercentMessage.encode({ value: ' $45.53 USD' })
let data = PercentMessage.decode(binary);

binary.byteLength
  // 5

data.value
  // "$45.53 USD"

🏓 Comparison Table

Choosing for real-time HTML5 / Node.js applications and games.

Here are some use cases stacked uup.

	tinybuf	FlatBuffers	Protocol Buffers	Raw JSON
Serialization format	Binary	Binary	Binary	String
Schema definition	Native	.fbs files	.proto files	Native
TypeScript Types	Native	Code generation	Code generation	Native
External tooling dependencies	None	cmake and flatc	None*	N/A
Reference data size†	34 bytes	68 bytes	72 bytes	175 bytes (minified)
Fast & efficient	🟢	🟢	🟢	🔴
16-bit floats	🟢	🔴	🔴	🔴
Boolean-packing	🟢	🔴	🔴	🔴
Arbitrary JSON	🟢	🔴	🔴	🟢
Property mangling	🟢	🔴	🔴	🔴
Suitable for real-time data	🟢	🟢	🔴	🔴
Suitable for web APIs	🔴	🔴	🟢	🟢
Supports HTML5 / Node.js	🟢	🟢	🟢	🟢
Cross-language (Java, C++, Python, etc.)	🔴	🟢	🟢	🟢

^†Based on the Reference data formats and schemas

^*When using protobufjs

See Reference data

Sample data (Minified JSON):

{
  "players": [
    {
      "id": 123,
      "position": {
        "x": 1.0,
        "y": 2.0,
        "z": 3.0
      },
      "velocity": {
        "x": 1.0,
        "y": 2.0,
        "z": 3.0
      },
      "health": 1.00
    },
    {
      "id": 456,
      "position": {
        "x": 1.0,
        "y": 2.0,
        "z": 3.0
      },
      "velocity": {
        "x": 1.0,
        "y": 2.0,
        "y": 3.0
      },
      "health": 0.50
    }
  ]
}

tinybuf

const ExampleMessage = encoder({
  players: [
    {
      id: Type.UInt,
      position: {
        x: Type.Float16,
        y: Type.Float16,
        z: Type.Float16
      },
      velocity: {
        x: Type.Float16,
        y: Type.Float16,
        y: Type.Float16
      },
      health: Type.UScalar
    },
  ],
});

FlatBuffers

// ExampleMessage.fbs

namespace ExampleNamespace;

table Vec3 {
  x: float;
  y: float;
  z: float;
}

table Player {
  id: uint;
  position: Vec3;
  velocity: Vec3;
  health: float;
}

table ExampleMessage {
  players: [Player];
}

root_type ExampleMessage;

Protocol Buffers (Proto3)

syntax = "proto3";

package example;

message Vec3 {
  float x = 1;
  float y = 2;
  float z = 3;
}

message Player {
  uint32 id = 1;
  Vec3 position = 2;
  Vec3 velocity = 3;
  float health = 4;
}

message ExampleMessage {
  repeated Player players = 1;
}

Encoding guide

See docs/ENCODING.md for an overview on how most formats are encoded (including the dynamically sized integer types).

Credits

Developed from a hard-fork of Guilherme Souza's js-binary.