Data Handlers

Introduction

The network library defines support for serialisation and descriptions of native Python types. This can be extended for custom types using the handler API.

Supported Types

• Integer (int)
• Floating Point (float)
• String (str)
• Boolean (bool)
• Bytes (bytes)
• Sets (set)
• Lists (list)
• Replicable reference (Replicable)
• Replicable class reference (type(Replicable))

Sequences

Sets and lists are variable sized and iterable constructs which can be replicated at full length across the network. Currently the elements of each object must be of the same (specified type). These are dangerous to use because of their bandwidth footprint with large structures.

Sequences now feature compression. The current handlers (sets, lists) define an attribute unique_members which determines whether members are unique (and therefore cannot be compressed). Sets are the sole iterable handler with this value set to True.

There are different "types" of compression. You can force the handler to use compression, or no compression with network.enums.IterableCompressionType.compress[|no_compress]. The other options is network.enums.IterableCompressionType.auto which will sample both the compressed and uncompressed iterable and use the smallest one, which will take longer to determine (as it's somewhat double serialisation), and exists as the default for serialisation.

If compression (auto/compress) is enabled, boolean value iterables will use BitFields to further reduce the data size. At present most of the time is spend unpacking sequences, rather than packing them into bytes.

To (unwisely) nest sequences, define a nesting type flag:

def nested_list(type_flag=TypeFlag(int)):
    return TypeFlag(list, element_type=type_flag)

flag = nested_list(nested_list())
# Used for packing [[1,2,3], [4,5,6]]
handler = get_handler(flag)

Handler API

Requesting Handlers

To access a handler for a data type, first we must import some machinery

from network.handler_interfaces import get_handler
from network.descriptors import TypeFlag

The TypeFlag object represents a value which has certain attributes, hence should not be shared between handler requests, save for identical data variables (such as two 8 bit Integers)

eight_bit_int_flag = TypeFlag(int)
thirty_two_bit_int_flag = TypeFlag(int, max_bits=32)
# Or using another argument integers leverage
thirty_two_bit_int_flag = TypeFlag(int, max_value=(2**32-1))

Now we ask for a handler for this type

eight_bit_int_flag = TypeFlag(int)

int_handler = get_handler(eight_bit_int_flag)

Handlers have a number of methods for interfacing with Bytes, such as pack, unpack_from, unpack_merge and size. Mutable types may have defined unpack_merge in their handler, whilst this is not useful for immutable values.

We can write an integer to bytes using our handler

int_bytes = int_handler.pack(135)
# Or read bytes and return an integer
int_value, bytes_occupied = int_handler.unpack_from(int_bytes)

unpack_from will read only the necessary bytes for the handler type, so it can be used to read a length prefix which is followed by an encoded string. This is how the String handler is implemented. The optional offset argument can be used to offset the starting position from which to read data from the bytes object.

Additionally, handlers can be given specific loggers (if they support logging), as is done in the FlagSerialiser class.

eight_bit_int_flag = TypeFlag(int)
some_logger = getLogger("SomeLogger")

# This won't be useful, as the handler doesn't support logging
int_handler = get_handler(eight_bit_int_flag, logger=some_logger)

Registering Handlers

To register a handler, we will need a provided function

from network.handler_interfaces import register_handler

Let's look at a pseudocode Bytes handler. As most handlers, it uses composition to pack individual values. Handlers may be specific to individual requests, which we leverage in this case to determine the maximum length of the byte string

from network.handlers import IHandler


class BytesHandler(IHandler):

    def __init__(self, type_flag, logger):
        super().__init__(type_flag, logger)

        header_max_value = type_flag.data.get("max_length", 255)
        self.packer = _handler_from_int(header_max_value)

    def pack(self, bytes_string):
        return self.packer.pack(len(bytes_string)) + bytes_string

    def pack_multiple(self, bytes_strings, count):
        lengths = [len(x) for x in bytes_strings]
        return self.packer.pack_multiple(lengths) + b''.join(bytes_strings)

    def size(self, bytes_string):
        length, length_size = self.packer.unpack_from(bytes_string)
        return length + length_size

    def unpack_from(self, bytes_string, offset=0):
        length, length_size = self.packer.unpack_from(bytes_string, offset)
        end_index = length + length_size
        value = bytes_string[length_size + offset: end_index + offset]

        return value, end_index

    def unpack_multiple(self, bytes_string, count, offset=0):
        lengths, lengths_offset = self.packer.unpack_multiple(bytes_string, count, offset)
        _offset = offset
        offset += lengths_offset
        data = []
        for length in lengths:
            data.append(bytes_string[offset: offset + length])
            offset += length

        return data, offset - _offset

register_handler(bytes, BytesHandler)

For mutable values, unpack_merge is defined as

def unpack_merge(cls, value, bytes_string, offset=0):
    # Do something with value
    return bytes_occupied

We register the handler using register_handler, which accepts the value type it should be retrieved for and a Handler callback that will be called by get_handler to return the appropriate handler for a specific request.

The handler may be any callable object, and accept two arguments; the TypeFlag instance used to request the handler (or subclass (e.g Attribute)), and a logger object. This allows for more verbose error messages in the event that deserialisation / serialisation fails internally.

For integers we use a selector, rather than the base class of a handler object. For performance and convenience reasons, we would prefer to return primitive serializers as singletons, so we return the class instead of instantiating it.

def _int_handler(flag, logger):
    """Return the correct int handler using meta information from a given type_flag"""
    if "max_value" in flag.data:
        new_cls = _handler_from_int(flag.data["max_value"])

    else:
        new_cls = _handler_from_bit_length(flag.data.get('max_bits', 8))

    return new_cls

Data Description

Typically, we use hash() for descriptions of objects, but this is not always possible. In order to solve this, there are two different ways of implementing a description method

Firstly, we can give the values to be handled a __description__" method. This is an example description defined for the network.enums.Roles` Enumeration

def __description__(self):
    return hash((self.context, self.local, self.remote))

However. some objects do not give us permission to write / define attributes. So we can use a registration function as we did for the handler

First import the register function

from network.handler_interfaces import register_description

Now create the method and register it

def vector_description(vector):
    return hash((vector.x, vector.y, vector.z))


register_description(vector, vector_description)