RFC packet processing.

[TimonPost]

This is an proposal about how to process packets based on different reliabilities and priories.

There are two library I stole some ideas from.

1. LiteNet (C# networking library)
2. RakNet (C++ networking library)

There are three components essential for what or library should have.

1. Channels like LiteNet.
2. Streams like RakNet.
3. Packet priorities like Raknet.

Channels

Bot of two libraries are working with the concept of 'Channels', let me clarify what 'Channel' means.
An 'Channel' will process packets based on there reliabilities property.

Reliabilities property's

Examples of RakNet and LiteNet there reliability property's:

• LiteNet's reliability types.I like
• RakNet's reliability types.

I want suppose we want to support the following ones:

/// Unreliable. Packets can be dropped, duplicated or arrive without order
///
/// *Details*
///
///  1. Unreliable
///  2. No guarantee for delivery.
///  3. No guarantee for order.
///  4. No way of getting dropped packet
///  5. Duplication possible
///
/// Basically just bare UDP
Unreliable,
/// Reliable. All packets will be sent and received, but without order
///
/// *Details*
///
///  1. Reliable.
///  2. Guarantee of delivery.
///  3. No guarantee for order.
///  4. Packets will not be dropped.
///  5. Duplication not possible
///
/// Basically this is almost TCP like without ordering of packets.
ReliableUnordered,
/// Unreliable. Packets can be dropped, but never duplicated and arrive in order
///
/// This will create an reliable ordered packet.
///
/// *Details*
///
///  1. Unreliable.
///  2. No guarantee of delivery.
///  3. Guarantee for order.
///  4. Packets can be dropped but you will be able to retrieve dropped packets.
///  5. Duplication not possible
///
/// Basically this is UDP with the ability to retrieve dropped packets by acknowledgements.
SequencedOrdered,
/// Unreliable. Packets can be dropped, and arrive out of order but you will be able to retrieve dropped packet.
///
/// *Details*
///
///  1. Unreliable.
///  2. No guarantee of delivery.
///  3. No Guarantee for order.
///  4. Packets can be dropped but you will be able to retrieve dropped packets.
///  5. Duplication not possible
///
/// Basically this is UDP with the ability to retrieve dropped packets by acknowledgements.
SequencedUnordered,
/// *Details*
///
///  1. Reliable.
///  2. Guarantee of delivery.
///  3. Guarantee for order.
///  4. Packets will not be dropped.
///  5. Duplication not possible
///
/// Basically this is almost TCP like with ordering of packets.
ReliableOrdered,

We should somehow process packets with different reliability property's.
There fore we use 'Channels' to separate the process concerns.

Implementation

So we define an trait called Channel (LiteNet):

trait Channel {
    /// Add an packet to queue awaiting to be processed before send.
    fn add_to_queue(packet: Packet);
    /// Process all packets in queue and send them out. 
    fn send_next_packets();
    /// progress the received packets.
    fn process_packet(packet: Packet);
}

Next, we define channels which implements the Channel trait.

1. ReliableChannel (see)

   This channel will be reliable and manage the reliability of packets it could also order packets as RakNet does. And queue them for the socket to send.

2. SequencedChannel (see)

   This channel will be unreliable and can order packets as RakNet does. It will only take in the newest data. And queue them for the socket to send.

3. UnreliableChanel (see)

   This is bare UDP as discussed before. Packets are directly processed. And queue them for the socket to send.

When a packet arrives and it is processed by a channel decided by the packet reliability property.
Next, we can notify the user by using, for example, the mpsc channels.

Note for @LucioFranco no we don't use one socket for each channel. The channels will only process data and queue data for the client to send.

Streams

Next topic I want to discuss are the streams from RakNet, RakNet has a nice concept of how to order packets (check out ordering streams for more info)

So what are those ordering streams?

You can think of ordering streams as something to separate the ordering of packets that have totally no relations to one and another.

So when a game-developer sends data to all the clients it might want to send some data ordered; some data unordered while other data needs to be send sequenced etc.

Let's take a look at the following data the dev might be sending:

1. Player movement, we want to order player movement because we don't care about old positions.
2. Bullet movement, we want to order bullet movement because we don't care about old positions of bullets.
3. Chat messages, we want to order chat messages because it is nice to see text in the correct order.

Player movement and chat messages are totally not related to one and another. You don't want the movement packets to be disturbed if a chat messages are dropped.
It would be nice if we can order player movement, chat messages separated from each other.

This is exactly where ordering streams are for.
We should let the user specify on which stream their packets should be ordered.
The user can, for example, say: "Let me put all chat messages on ordering stream 1 and all movement packets on ordering stream 2".
This way you can have different types of packets ordered separately from each other.

Why let the user control streams?

1. Let's take for example a player who is shooting bullets at something.
   It makes absolute sense to order both player movement and bullet position in the same ordering stream.
   You wouldn't want the shot to originate from the wrong position.
   So you'd put player firing packets on the same stream as movement packets (stream 1),
   and that if a movement packet arrived later than a firing packet but was actually sent earlier the firing packet would not be given to you until the movement packet arrived.
2. We can't interpreter what the contents of packets are whereon we decide where to order packets.

Packet Priority

A packet should also have some priority. Based on the priority we will decide which goes out first.
I did not fully research this yet but it is also not that important yet.

We basically have the following priority.

/// The highest possible priority.
///
/// 1. These message trigger sends immediately, and are generally not buffered or aggregated into a single datagram.
/// Messages at HighPriority priority and lower are buffered to be sent in groups at 10-millisecond intervals
immediate priority,
/// For every 2 ImmediatePriority messages, 1 HighPriority will be sent.
HighPriority,
/// The second lowest priority an datagram could be.
///
/// For every 2 HighPriority messages, 1 MediumPriority will be sent.
MediumPriority,
/// The lowest priority an datagram could be.
///
/// For every 2 MediumPriority messages, 1 LowPriority will be sent.
LowPriority

High priority packets go out before medium priority packets and medium priority packets go out before low priority packets do.

Check RakNet out for more information

General ideas

I think now the current architecture is quite closed for modification and closed for extension.
With this channel idea, we could be already more flexible.

Also, fragmentation is kind of handled throughout the code.
I like to see fragmentation into its own type. So we could make it optional.
When in development we need to move allot of the current code.

Idea to split this project up.

To prevent big PR's I want to spit this project up. The changes above have an impact on PacketProcessor, SocketState.

1. Create Channel trait, and implement different channels without logic.
2. Move processing logic out of SocketState and PacketProcessing client (ideas to spit this more up?)
3. Implement packet priority.

I'll be starting to implement some basic stuff if you guys agree with the above proposal. I think it is a nice way to handle or data. To note is that I did look at how other libraries were doing this and that I am not just making this all up. RakNet has been developt over 13 years. So I think there idea's are pretty solid.

Related issues to this RFC

• Thinking of packet processing logic. #39
• Congestion avoidance behavior #27
• Review quinn crate for possible uses #37

[OvermindDL1]

The Channels description you use above are more like RakNet Packet Types (UnorderedReliable, OrderedReliable, SequencedReliable, etc... etc...). Channels in RakNet is a byte from 0-255 that is only used on the reliable ordered and reliable sequenced types, the channel gives multiple 'paths' that the packets are reliable in relation to so that if you are sending, say, a 5 meg file of assets on channel 42 then entity position update information on channel 64-72 or whatever are not affected and forced to hold until the file is complete, which could take potentially multiple minutes.

[TimonPost]

No, I am not wrong, I see where our ideas are not matching. As you look from the client API perspective from RakNet we are talking about orderingChannels. Send takes in an argument of orderingChannel indeed.
However, ordering streams (bottom of page) is the term used internally in the code base to describe exactly what you are saying, as you can see here

You have to take note that I use concepts from different libraries. Like when I talk about channels I talk about the channels from LiteNet. When I write about the ordering streams I talk about your channels description.

[OvermindDL1]

I've never heard of LiteNet and I used to program on (well, on and with, I was a code contributor for some parts of it) RakNet (back in its more closed source'y days) so I only know the RakNet terminology (and its terminology does match what I see in the general networking world as well).

[OvermindDL1]

Also, as for 'ordering streams', that sounds like that only ordered packets can have different 'channels', but what about unordered? Reliable Unordered packets are super common in a game but they still need their own channels, not for ordering but to handle the sliding window and retransmission integration. They are more a 'reliability stream' rather than an 'ordering stream'.

[TimonPost]

I think now you are mixing two concepts. The ordering of packets separately from each other wich RakNet does and processing data according to the reliability type like the Channels as above described.

Like they state on their website: "Packets that are not ordered or sequenced at all, i.e. UNRELIABLE AND RELIABLE, have no bearing on sequences. That parameter is ignored by the send function for those types of packets".

The streams I talk about and RakNet uses is about ordering different packets separately from each other. This does not have to do anything with "sliding window and retransmission integration". Those things will be managed in the "Channels" I discussed above. It doesn't make sense to have "ordering" streams for unordered packets.

I agree that Reliable Unordered packets are very common. And that's where the channels are for. But as far as the ordering, like RakNet concerns, that will happen within those channels.

I don't want to confuse other readers about mixing those concepts please read my RFC carefully I have links to the code of what I mean and how I want to use different concepts from different libraries, as far as I know, I have explained it clear enough there unless you state otherwise.

[OvermindDL1]

Yeah if they have no ordering or sequencing at all (I don't count sequencing as ordering, since it's not) then there is no need for a channel.

Reliable Unordered are actually very common in some things, like chat messages (which should include their own timestamp for history usage), initial state updates for things that don't change after, etc... etc...

[TimonPost]

I think I get what you're saying. I did not mention the sequence streams. As said on the website "RakNet provides 32 ordering streams available for ordered packets and 32 ordering streams available for sequenced packets."

Is that you mean with supporting also ordering streams for Reliable Unordered packets? If yes I also take in account "ordering streams available for sequenced packets"

[fhaynes]

@TimonPost

Player movement, we want to order player movement because we don't care about old positions.
Bullet movement, we want to order bullet movement because we don't care about old positions of bullets.
Chat messages, we want to order chat messages because it is nice to see text in the correct order.

Typo? Or do we want to order for all three cases, or just chat messages?

[fhaynes]

@TimonPost do you see this going into 0.1, or for 0.2?

[TimonPost]

In this case, the game-developer wants to order all of them, but later up I describe you might want to order player movement and bullet movement together while you want to order chat messages separately.

0.1 having different reliability in the library will is very handy unless you state otherwise but issue #39 is also placed under 0.1. There are some things that need to be changed. But I don't have problems moving this to 0.2.

Maybe better, since we need to also create resending of dropped packets. We currently only support SequencedUnordered.

[fhaynes]

Well, my concern is adding a bunch of stuff to 0.1. I think it might be better to put some of the proposed features into 0.2 ,but we can figure that out after working out the details of the RFC.

[fhaynes]

@TimonPost

Why let the user control streams?

What about a user that doesn't want to control streams at that level? Would we provide reasonable defaults or guidance?

[TimonPost]

If the user doesn't want to specify streams we could have some default streams where we order on. This is also used in RackNet. Like it has some reserved stream id's it uses for ordering.

As for 0.2, I might think we could drop the ordering stream part which is a bit difficult to implement right. And focus on the channel proposal which will be only some moving of code from one place to another. What do you think?

[fhaynes]

Note I added #37 as a related issue. So far, we have this RFC, a QUIC extension proposal, and a suggestion to use the Quinn crate. How do we want to reconcile all these? =)

[fhaynes]

And focus on the channel proposal which will be only some moving of code from one place to another

I agree with this. Channels seem like a common primitive in everything we've looked at so far.

[TimonPost]

That's what I was thinking of having different reliability types will be essential for our library. Having support for QUIQ is an alternative which could be moved to later. I need to notice that maybe we are even able to add QUIQ support with the channel idea. I am not sure however, need to look into that as I am developing.

[fhaynes]

I think we should include crypto as part of this RFC as well. If we're going to do it, we need to do it right and bake it in from the beginning.

[fhaynes]

And I like the idea of first providing an abstraction layer to allow for pluggable backend types via a Trait.

[TimonPost]

I want everthing to be pluggable like encryption, the fragmentation that you could plug them onto some kind of stream of bytes. Each doing whats nesisairly. Also want to make packet processing optional, it needs some thought, we might discuss that inside discord. As far as encryption goes I will take a look and see if I could also included it into this design.

[LucioFranco]

This is great @TimonPost, I have a few comments, but that will have to come later this weekend. I have some good ideas how to improve the Channel trait.

One, idea that I had, is that I think we should create a terminology doc that defines what a channel, stream, etc are.

[TimonPost]

@LucioFranco Oke, cool I'll be glad to hear from you later. And yea the terminology doc will be our book which I will update later once this is some solid concept and implemented. Actually, I think I did not go into to much detail about Channels in the RFC so hopefully, there are no misunderstandings.

[LucioFranco]

I am curious with the channels are you suggesting that, we expose multiple mpsc channels to the user?

[TimonPost]

For now, I think closing this RFC is the best since it the idea is merged. When having other ideas please create a new proposal.