Enhance protocol support

[iceiix]

Support multiple protocols:

https://wiki.vg/Protocol_History
https://wiki.vg/Protocol_version_numbers#Versions_after_the_Netty_rewrite

Game version	Protocol version	Code	Supported?
1.16.4, .3, .2	754, 753, 751	#390	✓
1.16.1	736	#345	✓
1.15.2	578	#338	✓
1.15.1	575	#252	✓
1.14.4	498	#207	✓
1.14.3	490	#194	✓
1.14.2	485	#170	✓
1.14.1	480	#151	✓
1.14	477	#132	✓
19w02a	452	iceiix/steven#82	✓
18w50a	451	iceiix/steven#79	✓
1.13.2	404	iceiix/steven#67	✓
1.12.2	340	iceiix/steven#40, iceiix/steven#54	✓
1.11.2	316	iceiix/steven#38, iceiix/steven#48, iceiix/steven#54	✓
1.11	315	iceiix/steven#54 *	✓
1.10.2	210	iceiix/steven#14	✓
1.9.2	109	iceiix/steven#15	✓
1.9	107	iceiix/steven#16	✓
15w39c	74	iceiix/steven#17, iceiix/steven#56	✓
1.8.9	47	iceiix/steven#37, iceiix/steven#57	✓
1.7.10	5	iceiix/steven#64	✓

[iceiix]

Backported earlier protocol support from commit history, on branches:

• 1.11 (315): current
• 1.10.2 (210): 1.10.2 protocol support (210) steven#14
• 1.9.2 (109): 1.9.2 protocol support (109) steven#15
• 1.9 (107): 1.9 protocol support (107) steven#16
• 15w39c (74): 15w39c protocol support (74) steven#17

Tested these against older servers and they work, but, only one protocol version is supported at a time, i.e., these revert newer protocol support.

There is earlier protocol support in history, but while the project was in its infancy:

• 15w38b (73): iceiix/steven@8a2c247 Minor changes
• 15w37a (71): iceiix/steven@99b157f Correctly mark 15w39c as the supported version
• 15w36c (69): iceiix/steven@ab2336f Complete protocol implementation

The initial commit used 15w36c (69), but steven-rust's predecessor, steven-go, goes back further (changing SupportedProtocolVersion instead of SUPPORTED_PROTOCOL), lots of snapshots:

• 15w36c (69): Thinkofname/steven-go@0e8828f
• 15w35d (65): Thinkofname/steven-go@951d5ed
• 15w35b (63): Thinkofname/steven-go@83d73b6
• 15w34d (61): Thinkofname/steven-go@05ad2cb
• 15w33c (57): Thinkofname/steven-go@bfe8da2
• 15w32c (54): Thinkofname/steven-go@6f9ee9c
• 1.8.3 (47): Thinkofname/steven-go@b906a7e

1.8.3 (47) was added in Thinkofname/steven-go@d3ff0a2 all: more work, the second commit after initial commit.

https://wiki.vg/Protocol_History lists protocol versions (but oddly, seems to be missing 47 = 1.8.3?), and https://wiki.vg/Protocol_version_numbers#Versions_after_the_Netty_rewrite to last known documentation. https://minecraft.gamepedia.com/Java_Edition_version_history is also a useful reference. The latest version at the time of this writing is 1.13.2 (404, released October 22nd, 2018) so Steven is a few years behind, only supporting 1.11 (315, released November 14th, 2016).

Supporting newer versions is of course desirable, but should older version support just be dropped? It could be nice to have it around to support servers that haven't upgraded. Not every version, but some more common or useful ones. 1.7.6 up to 1.7.10 use protocol version 5, before then there was a "large protocol rework". Cuberite, the C++ server, does this: https://github.com/cuberite/cuberite/tree/master/src/Protocol - classes for major versions 1.9, 1.8, 1.10, 1.11, 1.12 and they are working on 1.13. https://github.com/cuberite/cuberite/blob/master/src/Protocol/ProtocolRecognizer.h#L20

#define MCS_CLIENT_VERSIONS "1.8.x, 1.9.x, 1.10.x, 1.11.x, 1.12.x"
#define MCS_PROTOCOL_VERSIONS "47, 107, 108, 109, 110, 210, 315, 316, 335, 338, 340"
#define MCS_LATEST_PROTOCOL_VERSION 340

[iceiix]

src/protocol/packet.rs defines packets using state_packets! macro, defined in src/protocol/mod.rs with macro_rules!, then numbered incrementally in src/macros.rs with create_ids!:

macro_rules! create_ids {
    ($t:ty, ) => ();
    ($t:ty, prev($prev:ident), $name:ident) => (
        #[allow(non_upper_case_globals)]
        pub const $name: $t = $prev + 1;
    );

so the packet identifiers are assigned from the order of definition. But the IDs changed over protocol versions, especially for the next major release after the currently supported version (1.11 to 1.12), and again in 1.13, so it could be worthwhile to factor out the IDs from the order in the code. Add a layer of indirection? Map (protocol version, packet id) to each packet? src/protocol/mod.rs:

                pub mod internal_ids {
                    create_ids!(i32, $($name),*);
                }

                $(
                    #[derive(Default, Debug)]
                    $(#[$attr])* pub struct $name {
                        $($(#[$fattr])* pub $field: $field_type),+,
                    }

                    impl PacketType for $name {

                        fn packet_id(&self) -> i32 { internal_ids::$name }

adds symbols in internal_ids:: which map to the id

[iceiix]

Expand the create_ids! macro to start, then change it per version? Using trace_macros: https://gist.github.com/iceiix/fc3941d7bc0c639f7eae9d90eacc1212

[iceiix]

Packet IDs are also available at, for example: https://github.com/PrismarineJS/minecraft-data/blob/master/data/pc/1.11/protocol.json

[iceiix]

Merged version_from_ping and flexible packet_ids branches. Note, dropped this commit which was the beginning of supporting 1.10.2, maybe use macros instead to avoid more redundant duplication? iceiix/steven@8f488d9

[iceiix]

A few observations from developing protocol support on branches, there are a several kinds of protocol changes: Additive, purely adding new content in a backwards-compatible way. This is of course the easiest to support multi-protocol, since the code for e.g. new packets or enum fields can remain in the application but not be used. Shifting IDs, this occurs fairly frequently during major protocol updates, when new packets are added others are commonly renumbered. Explicitly assigned packet IDs make this easier to support: need to expand self::$state::$dir::internal_ids::$name to also include protocol version, different lists for each. Implementation changes are the most difficult to support, examples include changing a protocol field value or behavior. However, maybe this could be in many cases be recast as adding new packets and changing IDs, i.e., if incompatibly-changed packets are added as new packets, only used for that protocol. For example, 1.12.2+'s keep-alives with i64 ids would be a different packet than pre-1.12.2 keep-alives with varints, mapped accordingly in the respective protocols. Some changes may require special-casing different paths in the code, especially if the incompatible packet is sent from us (serverbound, client-to-server). But not always, this particular change, keep-alives, could be handled something like this:

// fn entity_tick(&mut self, renderer: &mut render::Renderer, delta: f64) {
                       self pck {

                            KeepAliveClientbound_i64 => on_keep_alive_i64,
                            KeepAliveClientbound_varint => on_keep_alive_varint,
// impl Server {
    fn on_keep_alive_i64(&mut self, keep_alive: packet::play::clientbound::KeepAliveClientbound_i64) {
        self.write_packet(packet::play::serverbound::KeepAliveServerbound_i64 {
            id: keep_alive.id,
        });
    }

    fn on_keep_alive_varint(&mut self, keep_alive: packet::play::clientbound::KeepAliveClientbound_varint) {
        self.write_packet(packet::play::serverbound::KeepAliveServerbound_varint {
            id: keep_alive.id,
        });
    }

[iceiix]

Merged in the 1.12.2 update (#40) to the main branch, so now the other version branches (#14 #15 #16 #17 #37) have conflicts, would need to rebase on top of 1.11.2 (#48) to resolve, but maybe reopen each against the preceding version? Anyways these branches are only for reference until implement multi-protocol support. Next up 1.13.2 (404), latest stable at the time of this writing: https://wiki.vg/Protocol, lots of changes: https://wiki.vg/Protocol_History#1.13.2

[iceiix]

Implemented all the working branches into multi-protocol, so 7 versions are now supported: 1.12.2, 1.11.2, 1.11, 1.10.2, 1.9.2, 1.9, 15w39c. The snapshot protocol had the biggest changes, so although it is kind of unusual to support snapshots, it was good to get out of the way, since older release versions will encompass many of its changes, and it helped inform how to refactor the code, the basic design described above. Now to expand the protocol support slightly, in both directions, only a few more to add.

[iceiix]

Although Steven is already far down the path of its own protocol serialization implementations, this project is worth mentioning: http://protodef.io https://github.com/ProtoDef-io ProtoDef. There are a bunch of protocol definitions in ProtoDef at https://github.com/PrismarineJS/minecraft-data, and there are wrappers for JavaScript, Python, Elixir, Java, and Haskell, with the possibility to create more, PrismarineJS/minecraft-data#35 mentions Rust. They have a few generators, node-protodef written in JS, elixir-protodef in Elixir (leverages the Erlang VM) and the new one (at least there is talk about prismarine-chunk switching to it: PrismarineJS/prismarine-chunk#45) seems to be protodefc, "Compiler for protodef written in Rust", it supports Javascript and Python but they also have Java and Rust listed in the readme. On one hand it would be nice having Prismarine centralize the work of defining new protocol schemas, but on the other hand Steven would still need to adopt them and it adds more dependencies for what is already 70% done. Nonetheless, certainly useful for reference.

[iceiix]

1.8.9 support is in (iceiix/steven#57), next is 1.7.10 (iceiix/steven#64), then the big one, 1.13.2, fortunately prismarine-chunk just published 1.10.1 with 1.13 chunk format support to use for reference: https://github.com/PrismarineJS/prismarine-chunk/commits/master and https://github.com/PrismarineJS/prismarine-chunk/blob/master/src/pc/1.13/chunk.js

[iceiix]

With 1.13.2 now supported as of iceiix/steven#67, all currently planned protocol versions are now supported (1.7.10 - 1.13.2, major versions), at least their initial support, not that there isn't any bugs but it is a start. Only future multiprotocol work would be for new versions as they are released (1.14 when?), and bugfixes, so I'm closing this feature enhancement issue.