ViewBuffer - Schemaless, Codeless, Self-Describing Zero-Copy Flat Buffer 🚧

Many flat buffer systems require writing schemas and generating code, which slows and can break development. ViewBuffer is different. No schema required, no code gen required, still a flat buffer:

• Self-describing flat buffer; requires no schema; writing encodes the schema automatically.
• No code gen required; data structure is automatically outputted from the ViewBuffer file.
• ViewBuffer is slice based (e.g. native to D, supported in other languages, or emulated)
• Automatic zlib compression / decompression support as required
• Minimal code base so porting to different languages is relatively simple

Current read support for	Current write support for
D, C++	Java

The binary format is relatively simple:

# Bytes	Mandatory	Description	Notes
4	Y	Magic "VBUF"	Value of 0x46554256
1	Y	ViewBuffer version	e.g. 1 at the moment
1	Y	Flags	See below
2	Y	User-defined blob version	e.g. 1
4	Y	Header size
4	Y	Compressed blob size
4	Y	Decompressed blob size
16	N	Struct encoding hash	Depends on flag bit 2
2	N	Struct encoding length	Depends on flag bit 1
1+	N	Struct encoding	Depends on flag bit 1
...
0-3	N	Padding	To 4-byte alignment
4	Y	Number of offsets N	Offsets to slice pointers
4N	N	List of offsets	At least 0 offsets, 4 bytes each
...
1+	Y	Data blob	Compressed on flag bit 0
...

Offsets are bytes from the start of the blob which need adjusting to update pointers depending on where the blob is loaded into memory on the target machine. This is all automatic.

Currently available flags are:

Bit	Effect When Set
0	Blob compression on
1	Includes the struct encoding in the ViewBuffer
2	Includes a truncated SHA-256 hash of the struct encoding

TODO

• Ensure signed / unsigned compatibility
• Add further language support

Reading in D

See D test files.

The project contains version controlled IntelliJ run configurations if you want to use the IntelliJ IDE.

Or to run with dub command line:

dub test --config default     # regular
dub test --config betterc     # BetterC compatibility

Reading in C++

See C++ test files.

g++ -std=c++20 -DVIEWBUF_LOG=0 viewbuffer.cpp test_viewbuffer.cpp -lz -O2 -o test_viewbuffer
 ./test_viewbuffer ../data/fruits.bin ../data/pipeline.bin  

Writing in Java

Use the Java library to create and write data. See Java test files.

Troubleshooting

On Mac if running from the command line, if you get this error

gcc-14: error: unrecognized command-line option '-target'
Error: /opt/homebrew/bin/gcc-14 failed with status: 1
Error ldc2 failed with exit code 1.

you'll probably need to run this in the command line:

export D_COMPILER=ldc2
export CC=clang
export CXX=clang++