Data Handlers

Introduction

The network library defines support for serialisation and descriptions of native Python types. This can be extended for custom types using the handler API.

Supported Types

• Integer (int)
• Floating Point (float)
• String (str)
• Boolean (bool)
• Bytes (bytes)
• Sets (set)
• Lists (list)
• Replicable reference (Replicable)
• Replicable class reference (type(Replicable))

Sequences

Sets and lists are variable sized and iterable constructs which can be replicated at full length across the network. Currently the elements of each object must be of the same (specified type). These are dangerous to use because of their bandwidth footprint with large structures.

Sequences now feature compression. The current handlers (sets, lists) define an attribute unique_members which determines whether members are unique (and therefore cannot be compressed). Sets are the sole iterable handler with this value set to True.

There are different "types" of compression. You can force the handler to use compression, or no compression with network.enums.IterableCompressionType.compress[|no_compress]. The other options is network.enums.IterableCompressionType.auto which will sample both the compressed and uncompressed iterable and use the smallest one, which will take longer to determine (as it's somewhat double serialisation).

If compression (auto/compress) is enabled, boolean value iterables will use BitFields to further reduce the data size. At present most of the time is spend unpacking sequences, rather than packing them into bytes.

Handler API

Requesting Handlers

To access a handler for a data type, first we must import some machinery

from network.handler_interfaces import get_handler
from network.descriptors import TypeFlag

The TypeFlag object represents a value which has certain attributes, hence should not be shared between handler requests, save for identical data variables (such as two 8 bit Integers)

eight_bit_int_flag = TypeFlag(int)
thirty_two_bit_int_flag = TypeFlag(int, max_bits=32)
# Or using another argument integers leverage
thirty_two_bit_int_flag = TypeFlag(int, max_value=(2**32-1))

Now we ask for a handler for this type

eight_bit_int_flag = TypeFlag(int)

int_handler = get_handler(eight_bit_int_flag)

Handlers have a number of methods for interfacing with Bytes, such as pack, unpack_from, unpack_merge and size. Mutable types may have defined unpack_merge in their handler, whilst this is not useful for immutable values.

We can write an integer to bytes using our handler

int_bytes = int_handler.pack(135)
# Or read bytes and return an integer
int_value, bytes_occupied = int_handler.unpack_from(int_bytes)

unpack_from will read only the necessary bytes for the handler type, so it can be used to read a length prefix which is followed by an encoded string. This is how the String handler is implemented. The optional offset argument can be used to offset the starting position from which to read data from the bytes object.

Registering Handlers

To register a handler, we will need a provided function

from network.handler_interfaces import register_handler

Let's look at the Bytes handler. It is defined in the serialiser module, and uses composition to pack individual values. Handlers may be specific to individual requests, which we leverage in this case to determine the maximum length of the byte string

class BytesHandler:

    def __init__(self, type_flag):
        header_max_value = type_flag.data.get("max_length", 255)
        self.packer = handler_from_int(header_max_value)

    def pack(self, bytes_string):
        return self.packer.pack(len(bytes_string)) + bytes_string

    def size(self, bytes_string):
        length, length_size = self.packer.unpack_from(bytes_string)
        return length + length_size

    def unpack_from(self, bytes_string, offset=0):
        length, length_size = self.packer.unpack_from(bytes_string, offset)
        end_index = length + length_size
        value = bytes_string[length_size + offset: end_index + offset]

        return value, end_index

register_handler(bytes, BytesHandler, is_condition=True)

For mutable values, unpack_merge is defined as

def unpack_merge(cls, value, bytes_string, offset=0):
    # Do something with value
    return bytes_occupied

We register the handler using register_handler, which accepts the value type it is requested for, the Handler class that is used for that value type, and an optional is_condition argument which indicates that the handler argument is callable and accepts the TypeFlag (or subclass, e.g Attribute) instance used to request it.

For integers we use a selector, rather than a class, for our condition, and return an individual handler. Calling a class object returns an instance of the handler class, and calling the below selector will return an individual handler instance.

def int_selector(type_flag):
    if "max_value" in type_flag.data:
        return handler_from_int(type_flag.data["max_value"])

    return handler_from_bit_length(type_flag.data.get('max_bits', 8))

Network Description

Typically, we use hash() for descriptions of objects, but this is not always possible. In order to solve this, there are two different ways of implementing a description method

Firstly, we can give the values to be handled a __description__" method. This is an example description defined for the network.enums.Roles` Enumeration

def __description__(self):
    return hash((self.context, self.local, self.remote))

However. some objects do not give us permission to write / define attributes. So we can use a registration function as we did for the handler

First import the register function

from network.handler_interfaces import register_description

Now create the method and register it

def vector_description(vector):
    return hash((vector.x, vector.y, vector.z))


register_description(vector, vector_description)