Skip to content
Binary serialization for moonscript + LuaJIT
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
doc remove deprecated BlobWriter:rewind and allow negative values in seek() Mar 19, 2019
examples fixed typos Nov 28, 2018
lua
test remove deprecated BlobWriter:rewind and allow negative values in seek() Mar 19, 2019
.gitignore add support for arrays Nov 4, 2018
BlobReader.moon remove deprecated BlobWriter:rewind and allow negative values in seek() Mar 19, 2019
BlobWriter.moon
LICENSE initial commit Oct 29, 2018
README.md
_config.yml
compile.sh refactoring Nov 5, 2018
config.ld
ldoc.css

README.md

moonblob - binary serialization library

moonblob is a compact LuaJIT library written in moonscript that performs serialization into an efficient binary format. It can be used to parse arbitrary binary data, or to serialize data for efficient storage or transmission.

How to use

Lua example

Reading data

BlobReader = require('BlobReader')

-- Load data from file
file = io.open('filename.ext', 'rb')
blob = BlobReader(file:read('*all'))
file:close()

-- Parse raw binary data
u8 = blob:u8()
s16 = blob:s16()
str = blob:cstring()
float = blob:f32()
...

-- Read Lua types
tbl = blob:table()
bool = blob:bool() -- 8 bits, 0 == false
num = blob:number()
str = blob:string()

Writing data

BlobWriter = require('BlobWriter')

blob = BlobWriter()

-- Store raw binary data
blob
	:u8(23)
	:number(123.45)
	:f32(23.0)
	:string('string')
	...

-- Store Lua types
blob
	:write({ key: 'value', tbl: { 1, 2, 3 } }) -- no cycles allowed!
	:write(23)
	:write(true)
	:write('string')

-- Write data to file
file = io.open('filename.ext', 'wb')
file:write(blob:tostring())
file:close()

moonscript example

Reading data

BlobReader = require('BlobReader')

-- Load data from file
file = io.open('filename.ext', 'rb')
blob = BlobReader(file\read('*all'))
file\close!

-- Parse binary data
u8 = blob\u8!
s16 = blob\s16!
str = blob\cstring!
float = blob\f32!
...

-- Read Lua types
tbl = blob\table!
bool = blob\bool! -- 8 bits, 0 == false
num = blob\number!
str = blob\string!

Writing data

BlobWriter = require('BlobWriter')

-- Create a new Blob for writing
blob = BlobWriter!

-- Store binary data
with blob
	\u8(23)
	\number(123.45)
	\f32(23.0)
	\string('string')
	...

-- Store Lua types
with blob
	\write({ key: 'value', tbl: { 1, 2, 3 } }) -- no cycles allowed!
	\write(23)
	\write(true)
	\write('string')

-- Write data to file
file = io.open('filename.ext', 'wb')
file\write(blob\tostring!)
file\close!

Documentation

API reference

Reading and writing Lua types

BlobReader\read and BlobWriter\write can be used to store Lua values along with their type. BlobReader\read can only read data that was previously written by BlobWriter\write.

These data types are supported by BlobReader\read and BlobWriter\write:

  • number (64 bit)
  • string (up to 2^32-1 bytes)
  • boolean
  • table

Type and length information will be added as metadata by the write function. Metadata overhead is 1 byte per value written for type information, and between 1 and 5 bytes per string written for length information. Tables can contain number, string, boolean, and table as key and value types. An error is being thrown if other types or cyclic nested tables are encountered.

Low level I/O

A low level interface is provided for handling arbitrary binary data.

number       -- Lua number (64 bit floating point)
bool         -- boolean value (8 bits; 0 == false)
string       -- Lua string
table        -- Lua table
s8  / u8     -- signed/unsigned 8 bit integer value
s16 / u16    -- signed/unsigned 16 bit integer value
s32 / u32    -- signed/unsigned 32 bit integer value
s64 / u64    -- signed/unsigned 64 bit integer value
vs32 / vu32  -- length-encoded signed/unsigned 32 bit value
f32 / f64    -- 32/64 bit floating point value
raw          -- raw binary data (length must be specified)
cstring      -- zero-terminated string
array        -- sequential table of typed values

To describe the raw data format in a more concise manner, use BlobWriter\pack and BlobReader\unpack. These functions work similar to string.unpack and string.pack in Lua 5.3, although some details are different (fixed instead of native data sizes; more supported data types; some features are not implemented). See API reference for details.

Benchmark results

Source code for benchmarks can be found in the test directory.

Serialization speed (ops/sec, larger is better)

--- deepTable largeNumArray largeU32Array simpleTable smallNumArray smallU8Array
BlobWriter 1,007 1,695 5,908 490,782 460,424 880,942
bitser 422 179 191 313,985 43,573 52,559
binser 478 32 32 231,066 13,192 16,734

Deserialization speed (ops/sec, larger is better)

--- deepTable largeNumArray largeU32Array simpleTable smallNumArray smallU8Array
BlobReader 2,052 555 1,103 576,510 130,497 324,381
bitser 1,880 300 40 543,913 95,137 100,232
binser 1,371 83 77 375,935 29,017 50,566

Serialization size (bytes, smaller is better)

--- deepTable largeNumArray largeU32Array simpleTable smallNumArray smallU8Array
BlobWriter 45,808 524,291 262,147 69 2,042 258
bitser 25,732 393,025 404,089 62 1,338 571
binser 25,837 532,398 589,195 69 1,310 415

Compatibility

The library was developed on and for x86 and x64 machines - other architectures are currently untested.

Since moonblob uses the ffi library and C data types, it is not compatible with vanilla Lua and can only be used with LuaJIT.

License

Copyright 2017, 2018 megagrump

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.