So where we left off in :doc:`context`, we were left with the following packets:
import enum
import pak
class FelinePacket(pak.Packet):
class Context(pak.Packet.Context):
...
class Header(pak.Packet.Header):
size: pak.UInt8
String = pak.PrefixedString({
0: pak.UInt8,
1: pak.Uint16
})
class FurType(enum.Enum):
LongHaired = 0
ShortHaired = 1
MediumHaired = 2
class CatPicturesRequest(FelinePacket):
fur_type: pak.Enum(pak.UInt8, FurType)
class CatPicturesResponse(FelinePacket):
cat_pictures: String[None]
And we read these packets by passing them to read_packet, like so:
packet = read_packet(CatPicturesRequest)
But with many packet protocols, you simply don't know what packet you're going to be sent next, so something like read_packet(CatPicturesRequest) isn't applicable. So what do these protocols look like?
Well, packets in this sort of protocol almost always have some sort of "ID" that identifies what kind of packet it is. This ID could be sent in any sort of way, but it tends to live in the packet header, like the size of a packet might. So how would our protocol look if it were like this?
Instead of the client asking the server for URLs to pictures of cats with a certain fur type, the client will send a packet asking the server for a list of numbers that correspond to certain cats with a certain fur type. We will call these numbers "cat IDs" and they will be represented by a :class:`.UInt16`. The client can then send a different packet, bundling a single cat ID, asking for further information about the corresponding cat, namely its birth date and a URL to a picture of the cat.
Let's mock these packets up:
class CatIDsRequest(FelinePacket):
fur_type: pak.Enum(pak.UInt8, FurType)
class CatIDsResponse(FelinePacket):
fur_type: pak.Enum(pak.UInt8, FurType)
cat_ids: pak.UInt16[None]
class CatInfoRequest(FelinePacket):
cat_id: pak.UInt16
class CatInfoResponse(FelinePacket):
cat_id: pak.UInt16
picture_url: String
# The cat's birth date is represented by
# a 'UInt16' for the year, a 'UInt8' for
# the month, and a 'UInt8' for the day.
birth_year: pak.UInt16
birth_month: pak.UInt8
birth_day: pak.UInt8
It's unknown which packet will be received ahead of time, as the client can send a CatIDsRequest or CatInfoRequest packet at any time, and the server can similarly send a CatIDsResponse or CatInfoResponse packet at any time. Therefore, each packet will have a :class:`.UInt8` in its header, situated before the packet size, which will correspond to which packet is being received. This number is called the "packet ID". The packet IDs for CatIDsRequest and CatInfoRequest will be 0 and 1 respectively, and CatIDsResponse and CatInfoResponse will similarly have IDs of 0 and 1 respectively.
So how would we use Pak to implement this protocol? First, we should look at the :meth:`.Packet.id` classmethod. This method can optionally take in a :class:`.Packet.Context` parameter, and returns the ID of the packet. If there is no ID, then the method returns None; by default, :class:`.Packet`s have no ID. We can override this method to set the packet IDs, like so:
.. testcode::
:hide:
import enum
import pak
class FelinePacket(pak.Packet):
class Header(pak.Packet.Header):
id: pak.UInt8
size: pak.UInt8
class Context(pak.Packet.Context):
def __init__(self, *, version=1):
self.version = version
super().__init__()
def __hash__(self):
return hash(self.version)
def __eq__(self, other):
if not isinstance(other, FelinePacket.Context):
return NotImplemented
return self.version == other.version
# This isn't really what our 'String' type looked like,
# but it's how it looked in protocol version 1.
String = pak.PrefixedString(pak.UInt16)
class FurType(enum.Enum):
LongHaired = 0
ShortHaired = 1
MediumHaired = 2
.. testcode::
class CatIDsRequest(FelinePacket):
fur_type: pak.Enum(pak.UInt8, FurType)
@classmethod
def id(cls, *, ctx):
return 0
We can then use it like so:
.. testcode::
assert CatIDsRequest.id() == 0
Note that we did not need to specify the ctx parameter despite not defaulting it in our overriding of the id method. This is because Pak will handle the ctx parameter being unspecified for you, always passing you a proper :class:`.Packet.Context`.
Cool, so now our CatIDsRequest packet has an ID. How do we get that into the header? Basically the same way we got the packet size into the header in :doc:`stringy`:
class FelinePacket(pak.Packet):
class Header(pak.Packet.Header):
id: pak.UInt8
size: pak.UInt8
class Context(pak.Packet.Context):
...
We added the id field to our packet header, before the size field as described earlier. The :class:`.Packet.Header` machinery will call the :meth:`.Packet.id` method (with an appropriate :class:`.Packet.Context`) and put it in the header:
.. testcode::
packet = CatIDsRequest(fur_type=FurType.MediumHaired)
assert packet.header() == FelinePacket.Header(id=0, size=1)
assert packet.pack() == (
# Packet ID of '0'.
b"\x00" +
# Packet data size of '1'.
b"\x01" +
# Fur type of 'FurType.MediumHaired'.
b"\x02"
)
Cool, so now we know how to add packet IDs. But it is a bit much that we have to define a whole classmethod to simply have an ID of 0. It's not too much for one or a few packets, but it would add up for a full fledged protocol. We're not even touching the ctx parameter; we're not doing any real work at all. Luckily for us though, Pak alleviates this concern. We can simply set the ID like so:
.. testcode::
class CatIDsRequest(FelinePacket):
id = 0
fur_type: pak.Enum(pak.UInt8, FurType)
Pak will transform our simply setting the id attribute so that you still call the :meth:`.Packet.id` method like normal; the interface stays the same:
.. testcode::
assert CatIDsRequest.id() == 0
We still call the :meth:`.Packet.id` classmethod, working the same as before.
Let's fill out the IDs on all of our packets now:
class CatIDsRequest(FelinePacket):
id = 0
fur_type: pak.Enum(pak.UInt8, FurType)
class CatIDsResponse(FelinePacket):
id = 0
fur_type: pak.Enum(pak.UInt8, FurType)
cat_ids: pak.UInt16[None]
class CatInfoRequest(FelinePacket):
id = 1
cat_id: pak.UInt16
class CatInfoResponse(FelinePacket):
id = 1
cat_id: pak.UInt16
picture_url: String
# The cat's birth date is represented by
# a 'UInt16' for the year, a 'UInt8' for
# the month, and a 'UInt8' for the day.
birth_year: pak.UInt16
birth_month: pak.UInt8
birth_day: pak.UInt8
That packets have the same ID as another may seem like an issue at first, after all the ID is meant to uniquely identify which packet is being received, but it's actually okay since in each pair of packets with the same ID, one is received by the client, and one is received by the server; therefore each received type of packet has a unique ID. Packets bound to the client have unique IDs among clientbound packets, and packets bound to the server have unique IDs among serverbound packets.
Finally now we can worry about how we actually send and receive these packets. Sending them is easy, so we'll start with that. Here's our write_packet function from previous sections:
def write_packet(packet):
# Pack the packet into raw data.
# This will pack the header as well.
packet_data = packet.pack()
# Write the packet data to the client.
write_data(packet_data)
And... that's it. We don't have to change anything. The header machinery takes care of prefixing the data with the packet ID for us. Nice. So how about receiving packets? Well here's our previous read_packet function:
def read_packet(packet_cls):
# Read the data for the header. Our header
# has a static size, so we know how much to
# read beforehand.
header_data = read_data(FelinePacket.Header.size())
# Unpack the header from the header data.
header = FelinePacket.Header.unpack(header_data)
# Get the packet data from the client.
packet_data = read_data(header.size)
# Unpack the packet from the data and return it.
return packet_cls.unpack(packet_data)
We'll have to change this in a couple ways. First of all, our previous function takes in a packet_cls parameter to know which :class:`.Packet` to unpack. This doesn't work for us anymore as we don't know which :class:`.Packet` we're receiving beforehand. Therefore we'll need to get rid of the parameter entirely, and figure out which class to use based on the packet ID contained in the header. Let's see what that looks like:
def read_packet():
# Read the data for the header. Our header
# has a static size, so we know how much to
# read beforehand.
header_data = read_data(FelinePacket.Header.size())
header = FelinePacket.Header.unpack(header_data)
# Set 'packet_cls' based on the ID in the header.
#
# We only have to worry about serverbound packets
# since we are playing the part of the server in
# our protocol.
if header.id == 0:
packet_cls = CatIDsRequest
elif header.id == 1:
packet_cls = CatInfoRequest
else:
# There are other ways to handle unknown
# packets, but here we will raise an error.
raise ValueError("Invalid packet ID")
# Get the packet data from the client.
packet_data = read_data(header.size)
# Unpack the packet from the data and return it.
return packet_cls.unpack(packet_data)
First we unpack the header like before. Then we use the id attribute of the header to see which packet we're receiving, stored in packet_cls. If we don't recognize the ID, we raise a :exc:`ValueError`. Then we read the packet data and unpack it like before.
Note
Best practice would involve passing a :class:`.Packet.Context` to the :class:`.Packet` operations we use. We neglect to do so here for the sake of tutorial code.
There is a part that's kind of subpar though, our whole if/elif chain in there. It's not very scalable, and it requires two completely separate sources of truth; there's the ID specified in the packet definition and then the ID specified in our read_packet function. We could maybe alleviate that by doing == CatIDsRequest.id() instead of == 0 which would be an improvement, but then we're still defining what packets exist and what packets can be received in two separate places. If we add a new packet, we need to add it in two places: its class definition, and here in our read_packet function.
Thankfully, Pak addresses this issue for us, with the :meth:`.Packet.subclass_with_id` classmethod. With it we can ask for a subclass of our main :class:`.Packet` class, FelinePacket, which has the appropriate ID, allowing us to have a single sourch of truth: our packet definitions. But here's the sticking point: the way we have our classes set up right now, we have multiple subclasses with the same ID. This wasn't a problem before because the packet IDs are unique within their serverbound/clientbound set of packets, and we were just manually checking the IDs. We can alleviate this issue by fiddling with our inheritance tree a bit though:
.. testcode::
class ServerboundFelinePacket(FelinePacket):
pass
class ClientboundFelinePacket(FelinePacket):
pass
Here we define two new classes, ServerboundFelinePacket and ClientboundFelinePacket. Their class definitions are empty, as they only exist to separate serverbound and clientbound packets in our inheritance tree. Then we can make our actual packets inherit from the correct class:
.. testcode::
class CatIDsRequest(ServerboundFelinePacket):
id = 0
fur_type: pak.Enum(pak.UInt8, FurType)
class CatIDsResponse(ClientboundFelinePacket):
id = 0
fur_type: pak.Enum(pak.UInt8, FurType)
cat_ids: pak.UInt16[None]
class CatInfoRequest(ServerboundFelinePacket):
id = 1
cat_id: pak.UInt16
class CatInfoResponse(ClientboundFelinePacket):
id = 1
cat_id: pak.UInt16
picture_url: String
# The cat's birth date is represented by
# a 'UInt16' for the year, a 'UInt8' for
# the month, and a 'UInt8' for the day.
birth_year: pak.UInt16
birth_month: pak.UInt8
birth_day: pak.UInt8
Now every ServerboundFelinePacket and every ClientboundFelinePacket has a unique ID. So let's test out :meth:`.Packet.subclass_with_id`:
.. testcode::
assert ServerboundFelinePacket.subclass_with_id(0) is CatIDsRequest
Since CatIDsRequest is the serverbound packet with ID 0, it is returned from our call to :meth:`.Packet.subclass_with_id`.
Cool. But what if we were to pass an ID that doesn't correspond to any packet? In that case, :meth:`.Packet.subclass_with_id` will return None. So, armed with this new tool, let's rewrite our read_packet function:
def read_packet():
# Read the data for the header. Our header
# has a static size, so we know how much to
# read beforehand.
header_data = read_data(FelinePacket.Header.size())
header = FelinePacket.Header.unpack(header_data)
# Set 'packet_cls' based on the ID in the header.
#
# We only have to worry about serverbound packets
# since we are playing the part of the server in
# our protocol.
packet_cls = ServerboundFelinePacket.subclass_with_id(header.id)
if packet_cls is None:
# There are other ways to handle unknown
# packets, but here we will raise an error.
raise ValueError("Invalid packet ID")
# Get the packet data from the client.
packet_data = read_data(header.size)
# Unpack the packet from the data and return it.
return packet_cls.unpack(packet_data)
We were able to get rid of our if/elif chain and replace it with a much simpler call to :meth:`.Packet.subclass_with_id`, resulting in what I would say is much nicer code.
In the previous :doc:`context` section, we explored how our String type could change how it marshals to and from raw data based on the version of our protocol. Similarly, packet IDs can change can change their value based on our protocol version. So how would we go about handling that?
Well, if we recall, when we overrode the :meth:`.Packet.id` classmethod, we had a ctx parameter available to us. This ctx parameter will name an appropriate :class:`.Packet.Context`, in this case our FelinePacket.Context, which will contain a version attribute telling us our protocol version. So let's say that in protocol version 0, our packet IDs are as they are now, but in version 1, they're swapped, so that CatIDsRequest and CatIDsResponse have ID 1 and CatInfoRequest and CatInfoResponse have ID 0. Let's see how we could model this:
class CatIDsRequest(ServerboundFelinePacket):
@classmethod
def id(cls, *, ctx):
if ctx.version < 1:
return 0
return 1
fur_type: pak.Enum(pak.UInt8, FurType)
class CatIDsResponse(ClientboundFelinePacket):
@classmethod
def id(cls, *, ctx):
if ctx.version < 1:
return 0
return 1
fur_type: pak.Enum(pak.UInt8, FurType)
cat_ids: pak.UInt16[None]
class CatInfoRequest(ServerboundFelinePacket):
@classmethod
def id(cls, *, ctx):
if ctx.version < 1:
return 1
return 0
cat_id: pak.UInt16
class CatInfoResponse(ClientboundFelinePacket):
@classmethod
def id(cls, *, ctx):
if ctx.version < 1:
return 1
return 0
cat_id: pak.UInt16
picture_url: String
# The cat's birth date is represented by
# a 'UInt16' for the year, a 'UInt8' for
# the month, and a 'UInt8' for the day.
birth_year: pak.UInt16
birth_month: pak.UInt8
birth_day: pak.UInt8
Here we replaced all our previous lines which simply set the id attribute to a number with full-fledged classmethods, returning a different number depending on ctx.version. And well, this works, and it's mostly clear what's going on, but it's also not as clear as what we had before, with simply setting the id attribute to a number. It's less declarative, more imperative.
If you'll recall, we had a similar situation in :ref:`versioned-string`. The code we had for changing how strings were marshaled depending on the protocol version was also more imperative than declarative, and wasn't super readable. We solved this issue by introducing the concept of typelikes, and registered :class:`dict`s as typelike, resulting in an API that allowed us to define our String type like so:
String = pak.PrefixedString({
0: pak.UInt8,
1: pak.UInt16,
})
This was decently more declarative and readable than what we had before. Maybe we can have a similar API for packet IDs?
In fact, we can have a similar API for packet IDs! In the end, we'll be able to define IDs like this:
class CatInfoRequest(ServerboundFelinePacket):
id = {
0: 0,
1: 1,
}
cat_id: pak.UInt16
So how do we get there? Pak provides a utility for this issue: :class:`.DynamicValue`. This is used to transform one value into a classmethod-ish thing, which can provide different "return" values based on a ctx parameter and the initial value for the :class:`.DynamicValue`. :class:`.Packet` will automatically try to make the id attribute we set into a :class:`.DynamicValue`. So for us, we want to make it so :class:`dict`s will get enrolled in this machinery, and return the appropriate ID values based on our protocol version. Let's walk through it:
class VersionedDynamicValue(pak.DynamicValue):
...
First we create a class which inherits from :class:`.DynamicValue` named VersionedDynamicValue. This is what will be instantiated when things like :class:`.Packet` interact with the :class:`.DynamicValue` machinery.
class VersionedDynamicValue(pak.DynamicValue):
_type = dict
...
Next we set the _type attribute to the :class:`dict` type so that instances of :class:`dict` will be changed into a VersionedDynamicValue.
class VersionedDynamicValue(pak.DynamicValue):
_type = dict
def __init__(self, initial_value):
self.version_info = initial_value
...
Now we add an __init__ method which accepts an initial_value parameter which names the initial :class:`dict` value for our VersionedDynamicValue, which we then store in the version_info attribute.
.. testcode::
class VersionedDynamicValue(pak.DynamicValue):
_type = dict
def __init__(self, initial_value):
self.version_info = initial_value
def get(self, *, ctx=None):
return self.version_info[ctx.version]
Finally we add the get method, which accepts a ctx parameter that will name either an appropriate :class:`.Packet.Context` or an appropriate :class:`.Type.Context`, or None. In this method we return the appropriate value based on the protocol version stored within the ctx parameter.
If we did everything right, we should be able to do the following:
.. testcode::
class CatIDsRequest(ServerboundFelinePacket):
id = {
0: 0,
1: 1,
}
fur_type: pak.Enum(pak.UInt8, FurType)
ctx_version_0 = FelinePacket.Context(version=0)
ctx_version_1 = FelinePacket.Context(version=1)
assert CatIDsRequest.id(ctx=ctx_version_0) == 0
assert CatIDsRequest.id(ctx=ctx_version_1) == 1
Pretty swanky I'd say. We got to keep a declarative API, and have it similar to the typelike API we made in :doc:`context`. But how would we go about handling all these packets, especially at scale? Let's head onto :doc:`handle` to find out.