go-polo

go-polo is a Go implementation of the POLO Encoding and Object Serialization Scheme. POLO stands for Prefix Ordered Lookup Offsets.

It is intended for use in projects that prioritize deterministic serialization, minimal wire sizes and serialization safety. POLO follows a very strict specification that is optimized for partial decoding and differential messaging. This implementation is compliant with the POLO Specification that describes both the encoding (wire) format as well as implementation guidelines for several languages.

Installation

Install the latest release using the following command

go get -u github.com/sarvalabs/go-polo

Features

Deterministic Serialization

POLO's strict specification is intended to create the same serialized wire for an object regardless of implementation. This is critical for cryptographic security with operations such as hashing which is used to guarantee data consistency and tamper proofing.

High Wire Efficiency

POLO has a highly optimized wire format allows messages to be relatively small, even surpassing Protocol Buffers occassionaly. This is mainly because it supports a larger type based wire tagging that allows some information (especially metadata) to be passed around inferentially and thus reducing the total amount of information actually present in the wire.

This is augmented by the fact that POLO supports Atomic Encoding, because of which simple objects such as integers & strings can be encoded without needing to be wrapped within a larger (structural) message

Partial Encoding/Decoding Constructs

The POLO wire format prefixes all metadata for the wire in the front, this metadata includes the wire type information as well as the offset position of the data. This kind of lookup based offset tags allows us to directly access the data for a particular field order. This capability is currently supported using the Document and Raw constructs.

The Document construct is used to create a string indexed collection of Raw object and it collapses into a special form of the POLO wire called the document-encoded wire with the WireDoc wire type. It's unique in that it preserves the field name (string index) for the data unlike regular POLO encoding and is similar to encoding schemes like JSON or YAML and consequently consumes more wire space.

The Raw construct is useful for capturing the wire data for a specific field in the struct. It can also be used to define a structure that 'skips' the unrequired fields by capturing their raw wire instead of decoding them.

Partial encodign/decoding capabilities for this implementation are unfinished can be extended to support field order based access/write for regular POLO encoded wires in the future.

Custom Encoding/Decoding Buffers

POLO describes two buffers, Polorizer and Depolorizer which are write-only and read-only respectively, allowing sequential encoding/decoding of objects and wire elements into them. This capability can be leveraged to implement the Polorizable and Depolorizable interfaces which describe the custom serialization form for an object.

Note: This capability can be dangerous if not implemented correctly, it generally recommended that both interfaces be implemented and are evenly capable of encoding/decoding the same contents to avoid inconsistency. It is intended to be used for object such as Go Interfaces which are not supported by default when using the reflection based Polorize and Depolorize functions.

Differential Messaging (Coming Soon)

POLO's partially encoding and field order based indexing (and string based indexing for document encoded wires) allows the possibility for messaging that only allows the transmission of the difference between two states, this is useful for any version managment system where the same data is incrementally updated and transmitted, the ability to index the difference and only transmit the difference can result in massive reduction in the wire sizes for these use cases that often re-transmit already available information.

Examples

Simple Polorization & Depolorization (Encoding/Decoding)

go-polo/polo_test.go

Lines 16 to 64 in e53ea07

	// Fruit is an example for a Go struct
	type Fruit struct {
	Name string
	Cost int `polo:"cost"`
	Alias []string `polo:"alias"`
	}
	// ExamplePolorize is an example for using the Polorize function to
	// encode a Fruit object into its POLO wire form using Go reflection
	//
	//nolint:lll
	func ExamplePolorize() {
	// Create a Fruit object
	orange := &Fruit{"orange", 300, []string{"tangerine", "mandarin"}}
	// Serialize the Fruit object
	wire, err := Polorize(orange)
	if err != nil {
	log.Fatalln(err)
	}
	// Print the serialized bytes
	fmt.Println(wire)
	// Output:
	// [14 79 6 99 142 1 111 114 97 110 103 101 1 44 63 6 150 1 116 97 110 103 101 114 105 110 101 109 97 110 100 97 114 105 110]
	}
	// ExampleDepolorize is an example for using the Depolorize function to
	// decode a Fruit object from its POLO wire form using Go reflection
	func ExampleDepolorize() {
	wire := []byte{
	14, 79, 6, 99, 142, 1, 111, 114, 97, 110, 103, 101, 1, 44, 63, 6, 150, 1, 116,
	97, 110, 103, 101, 114, 105, 110, 101, 109, 97, 110, 100, 97, 114, 105, 110,
	}
	// Create a new instance of Fruit
	object := new(Fruit)
	// Deserialize the wire into the Fruit object (must be a pointer)
	if err := Depolorize(object, wire); err != nil {
	log.Fatalln(err)
	}
	// Print the deserialized object
	fmt.Println(object)
	// Output:
	// &{orange 300 [tangerine mandarin]}
	}

Custom Polorization & Depolorization (Encoding/Decoding)

go-polo/polo_test.go

Lines 66 to 175 in e53ea07

	// CustomFruit is an example for a Go struct that
	// implements the Polorizable and Depolorizable interfaces
	type CustomFruit struct {
	Name string
	Cost int
	Alias []string
	}
	// Polorize implements the Polorizable interface for
	// CustomFruit and allows custom serialization of Fruit objects
	func (fruit CustomFruit) Polorize() (*Polorizer, error) {
	fmt.Println("Custom Serialize for Fruit Invoked")
	// Create a new Polorizer
	polorizer := NewPolorizer()
	// Encode the Name field as a string
	polorizer.PolorizeString(fruit.Name)
	// Encode the Cost field as an integer
	polorizer.PolorizeInt(int64(fruit.Cost))
	// Create a new Polorizer to serialize the Alias field (slice)
	aliases := NewPolorizer()
	// Encode each element in the Alias slice as a string
	for _, alias := range fruit.Alias {
	aliases.PolorizeString(alias)
	}
	// Encode the Polorizer containing the alias field contents as packed data
	polorizer.PolorizePacked(aliases)
	return polorizer, nil
	}
	// Depolorize implements the Depolorizable interface for
	// CustomFruit and allows custom deserialization of Fruit objects
	func (fruit *CustomFruit) Depolorize(depolorizer *Depolorizer) (err error) {
	fmt.Println("Custom Deserialize for Fruit Invoked")
	// Convert the Depolorizer into a pack Depolorizer
	depolorizer, err = depolorizer.DepolorizePacked()
	if err != nil {
	return fmt.Errorf("invalid wire: not a pack: %w", err)
	}
	// Decode the Name field as a string
	fruit.Name, err = depolorizer.DepolorizeString()
	if err != nil {
	log.Fatalln("invalid field 'Name':", err)
	}
	// Decode the Cost field as a string
	Cost, err := depolorizer.DepolorizeInt()
	if err != nil {
	log.Fatalln("invalid field 'Cost':", err)
	}
	fruit.Cost = int(Cost)
	// Decode a new Depolorizer to deserialize the Alias field (slice)
	aliases, err := depolorizer.DepolorizePacked()
	if err != nil {
	log.Fatalln("invalid field 'Alias':", err)
	}
	// Decode each element from the Alias decoder as a string
	for !aliases.Done() {
	alias, err := aliases.DepolorizeString()
	if err != nil {
	log.Fatalln("invalid field element 'Alias':", err)
	}
	fruit.Alias = append(fruit.Alias, alias)
	}
	return nil
	}
	// ExampleCustomEncoding is an example for using custom serialization and deserialization on the
	// CustomFruit type by implementing the Polorizable and Depolorizable interfaces for it.
	//
	//nolint:govet, lll
	func ExampleCustomEncoding() {
	// Create a CustomFruit object
	orange := &CustomFruit{"orange", 300, []string{"tangerine", "mandarin"}}
	// Serialize the Fruit object
	wire, err := Polorize(orange)
	if err != nil {
	log.Fatalln(err)
	}
	// Print the serialized bytes
	fmt.Println(wire)
	// Create a new instance of CustomFruit
	object := new(CustomFruit)
	// Deserialize the wire into the CustomFruit object (must be a pointer)
	if err := Depolorize(object, wire); err != nil {
	log.Fatalln(err)
	}
	// Print the deserialized object
	fmt.Println(object)
	// Output:
	// Custom Serialize for Fruit Invoked
	// [14 79 6 99 142 1 111 114 97 110 103 101 1 44 63 6 150 1 116 97 110 103 101 114 105 110 101 109 97 110 100 97 114 105 110]
	// Custom Deserialize for Fruit Invoked
	// &{orange 300 [tangerine mandarin]}
	}

Wire Decoding with Any

go-polo/polo_test.go

Lines 177 to 214 in e53ea07

	// ExampleWireDecoding is an example for using the Any type to capture
	// the raw POLO encoded bytes for a specific field of the Fruit object.
	//
	//nolint:govet, lll
	func ExampleWireDecoding() {
	// RawFruit is a struct that can capture the raw POLO bytes of each field
	type RawFruit struct {
	Name Any
	Cost int
	Alias []string
	}
	// Create a Fruit object
	orange := &Fruit{"orange", 300, []string{"tangerine", "mandarin"}}
	// Serialize the Fruit object
	wire, err := Polorize(orange)
	if err != nil {
	log.Fatalln(err)
	}
	// Print the serialized bytes
	fmt.Println(wire)
	// Create a new instance of RawFruit
	object := new(RawFruit)
	// Deserialize the wire into the RawFruit object (must be a pointer)
	if err := Depolorize(object, wire); err != nil {
	log.Fatalln(err)
	}
	// Print the deserialized object
	fmt.Println(object)
	// Output:
	// [14 79 6 99 142 1 111 114 97 110 103 101 1 44 63 6 150 1 116 97 110 103 101 114 105 110 101 109 97 110 100 97 114 105 110]
	// &{[6 111 114 97 110 103 101] 300 [tangerine mandarin]}
	}

Polorizer (Encoding Buffer)

go-polo/polorizer_test.go

Lines 12 to 42 in e53ea07

	// ExamplePolorizer is an example for using the Polorizer to encode the fields of a Fruit object
	// using a Polorizer which allows sequential encoding of data into a write-only buffer
	//
	//nolint:lll
	func ExamplePolorizer() {
	// Create a Fruit object
	orange := &Fruit{"orange", 300, []string{"tangerine", "mandarin"}}
	// Create a new Polorizer
	polorizer := NewPolorizer()
	// Encode the Name field as a string
	polorizer.PolorizeString(orange.Name)
	// Encode the Cost field as an integer
	polorizer.PolorizeInt(int64(orange.Cost))
	// Create a new Polorizer to serialize the Alias field (slice)
	aliases := NewPolorizer()
	// Encode each element in the Alias slice as a string
	for _, alias := range orange.Alias {
	aliases.PolorizeString(alias)
	}
	// Encode the Polorizer containing the alias field contents as packed data
	polorizer.PolorizePacked(aliases)
	// Print the serialized bytes in the Polorizer buffer
	fmt.Println(polorizer.Bytes())
	// Output:
	// [14 79 6 99 142 1 111 114 97 110 103 101 1 44 63 6 150 1 116 97 110 103 101 114 105 110 101 109 97 110 100 97 114 105 110]
	}

Depolorizer (Decoding Buffer)

go-polo/depolorizer_test.go

Lines 13 to 70 in e53ea07

	// ExampleDepolorizer is an example for using the Depolorizer to decode the fields of a Fruit object
	// using a Depolorizer which allows sequential decoding of data from a read-only buffer
	func ExampleDepolorizer() {
	wire := []byte{
	14, 79, 6, 99, 142, 1, 111, 114, 97, 110, 103, 101, 1, 44, 63, 6, 150, 1, 116,
	97, 110, 103, 101, 114, 105, 110, 101, 109, 97, 110, 100, 97, 114, 105, 110,
	}
	// Create a new instance of Fruit
	object := new(Fruit)
	// Create a new Depolorizer from the data
	depolorizer, err := NewDepolorizer(wire)
	if err != nil {
	log.Fatalln("invalid wire:", err)
	}
	depolorizer, err = depolorizer.DepolorizePacked()
	if err != nil {
	log.Fatalln("invalid wire:", err)
	}
	// Decode the Name field as a string
	object.Name, err = depolorizer.DepolorizeString()
	if err != nil {
	log.Fatalln("invalid field 'Name':", err)
	}
	// Decode the Cost field as a string
	Cost, err := depolorizer.DepolorizeInt()
	if err != nil {
	log.Fatalln("invalid field 'Cost':", err)
	}
	object.Cost = int(Cost)
	// Decode a new Depolorizer to deserialize the Alias field (slice)
	aliases, err := depolorizer.DepolorizePacked()
	if err != nil {
	log.Fatalln("invalid field 'Alias':", err)
	}
	// Decode each element from the Alias decoder as a string
	for !aliases.Done() {
	alias, err := aliases.DepolorizeString()
	if err != nil {
	log.Fatalln("invalid field element 'Alias':", err)
	}
	object.Alias = append(object.Alias, alias)
	}
	// Print the deserialized object
	fmt.Println(object)
	// Output:
	// &{orange 300 [tangerine mandarin]}
	}

Document Encoding

go-polo/document_test.go

Lines 12 to 74 in e53ea07

	// ExampleDocument is an example for using the Document object's method to partially encode
	// fields as properties into it and then serialize it into document encoded POLO bytes
	//
	//nolint:lll
	func ExampleDocument() {
	// Create a new Document
	document := make(Document)
	// Encode the 'Name' field
	if err := document.Set("Name", "orange"); err != nil {
	log.Fatalln(err)
	}
	// Encode the 'cost' field
	if err := document.Set("cost", 300); err != nil {
	log.Fatalln(err)
	}
	// Encode the 'alias' field
	if err := document.Set("alias", []string{"tangerine", "mandarin"}); err != nil {
	log.Fatalln(err)
	}
	// Print the Document object and it serialized bytes
	fmt.Println(document)
	fmt.Println(document.Bytes())
	// Output:
	// map[Name:[6 111 114 97 110 103 101] alias:[14 63 6 150 1 116 97 110 103 101 114 105 110 101 109 97 110 100 97 114 105 110] cost:[3 1 44]]
	// [13 175 1 6 69 182 1 133 2 230 4 165 5 78 97 109 101 6 111 114 97 110 103 101 97 108 105 97 115 14 63 6 150 1 116 97 110 103 101 114 105 110 101 109 97 110 100 97 114 105 110 99 111 115 116 3 1 44]
	}
	// ExamplePolorizeDocument is an example for using PolorizeDocument to encode a
	// struct into a Document and then further serializing it into document encoded POLO bytes
	//
	//nolint:lll
	func ExamplePolorizeDocument() {
	// Create a Fruit object
	orange := &Fruit{"orange", 300, []string{"tangerine", "mandarin"}}
	// Encode the object into a Document
	document, err := PolorizeDocument(orange)
	if err != nil {
	log.Fatalln(err)
	}
	// Print the Document object
	fmt.Println(document)
	// Serialize the Document object
	// This can also be done with document.Bytes()
	wire, err := Polorize(document)
	if err != nil {
	log.Fatalln(err)
	}
	// Print the serialized Document
	fmt.Println(wire)
	// Output:
	// map[Name:[6 111 114 97 110 103 101] alias:[14 63 6 150 1 116 97 110 103 101 114 105 110 101 109 97 110 100 97 114 105 110] cost:[3 1 44]]
	// [13 175 1 6 69 182 1 133 2 230 4 165 5 78 97 109 101 6 111 114 97 110 103 101 97 108 105 97 115 14 63 6 150 1 116 97 110 103 101 114 105 110 101 109 97 110 100 97 114 105 110 99 111 115 116 3 1 44]
	}

Document Decoding

go-polo/document_test.go

Lines 76 to 124 in e53ea07

	// ExampleDepolorizeDocument_ToDocument is an example of using the Depolorize
	// function to decode a document-encoded wire into a Document object
	//
	//nolint:govet, lll
	func ExampleDepolorizeDocument_ToDocument() {
	wire := []byte{
	13, 175, 1, 6, 69, 182, 1, 133, 2, 230, 4, 165, 5, 78, 97, 109, 101, 6, 111, 114, 97,
	110, 103, 101, 97, 108, 105, 97, 115, 14, 63, 6, 150, 1, 116, 97, 110, 103, 101, 114,
	105, 110, 101, 109, 97, 110, 100, 97, 114, 105, 110, 99, 111, 115, 116, 3, 1, 44,
	}
	// Create a new Document
	doc := make(Document)
	// Deserialize the document bytes into a Document
	if err := Depolorize(&doc, wire); err != nil {
	log.Fatalln(err)
	}
	// Print the decoded Document
	fmt.Println(doc)
	// Output:
	// map[Name:[6 111 114 97 110 103 101] alias:[14 63 6 150 1 116 97 110 103 101 114 105 110 101 109 97 110 100 97 114 105 110] cost:[3 1 44]]
	}
	// ExampleDepolorizeDocument_ToStruct is an example of using the Depolorize
	// function to decode a document encoded wire into a Fruit object
	//
	//nolint:govet
	func ExampleDepolorizeDocument_ToStruct() {
	wire := []byte{
	13, 175, 1, 6, 69, 182, 1, 133, 2, 230, 4, 165, 5, 78, 97, 109, 101, 6, 111, 114, 97,
	110, 103, 101, 97, 108, 105, 97, 115, 14, 63, 6, 150, 1, 116, 97, 110, 103, 101, 114,
	105, 110, 101, 109, 97, 110, 100, 97, 114, 105, 110, 99, 111, 115, 116, 3, 1, 44,
	}
	// Create a new instance of Fruit
	object := new(Fruit)
	// Deserialize the document bytes into the Fruit object
	if err := Depolorize(object, wire, DocStructs()); err != nil {
	log.Fatalln(err)
	}
	// Print the deserialized object
	fmt.Println(object)
	// Output:
	// &{orange 300 [tangerine mandarin]}
	}

Contributing

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as below, without any additional terms or conditions.

License

© 2023 Sarva Labs Inc. & MOI Protocol Developers.

This project is licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

The SPDX license identifier for this project is MIT OR Apache-2.0.