Simple Protocol Buffers

Problem:

Defining a suitable communication protocol can be difficult and hard to maintain when new fields are added to the protocol at a later point.

Solution:

Looking at the Google library Protocol Buffers I found a convenient soluton to my problem. But, I wanted to take it a step further and create a DSL on top of the protocol buffers themselves and from this allow for the following properties.

• Protocols are broken into proto classes.
• Protocols are defined in .pb files.
• The .pb files are compiled into their equivalent Python classes.
• Objects sent will be serialized into a binary format that can be represented in Base64.
• Objects read can be deserialized back into their Python object.

This mimics the current Protocol Buffers offered by Google, but to allow for the flexibility for my projects I am building ontop of this and adding the following additional functionality.

• Protocols can be composed from other Protocols
• Protocol compositions allow for a inheritence structure, without the complications (no overloading).

This above allows for the following situation: header.pb

package example

proto Header ->
	id -> Int
	length -> Int

message.pb

package example

import header

proto Message | Header | ->
	message -> String
	from -> String
	to -> List:String
	timestamp -> Date

heartbeat.pb

package example

import message

proto HeartBeat | Message | ->
	heartbeatId -> Int

The above definitions show that a HeartBeat is a composition of a HeartBeat and a Message, but a Message is also a composition of a Message and a Header.

This composition can be expanded into a single definition as shown below:

package example

proto HeartBeat ->
	id -> Int
	length -> Int
	message -> String
	from -> String
	to -> List:String
	timestamp -> Date
	heartbeatId -> Int

But, this much more verbose version is a HeartBeat message that contains much more data that ultimately should not be handled in the HeartBeat protocol and should be pushed down into parent protocols. This also allows for polymorphism when dealing with the messages for example that along with the heart beat there is another message as such:

package example

import message

proto Hosts | Message | ->
	hosts -> List:String

from example.hosts import Hosts
from example.heartbeat import HeartBeat

hs = Hosts()
hs.hosts.concat( [ "10.0.0.1", "10.0.0.2", "10.0.0.3" ] )

hb = HeartBeat()
hb.heartBeatId = 100

messages = [ hs, hb ]

for m in messages:
    send( messages, to="10.0.0.5", from="10.0.0.6" )

In the case above the send message simply works with Message protocols. And, thus does not care that there is a list of hosts or a heartBeatId tacked onto the messages. When the message is finally sent the method serialize is called and everything is handled automatically and the receiving client will properly read the messages.