1.8.9 (47) multiprotocol support

[iceiix]

http://wiki.vg/index.php?title=Protocol&oldid=7368
https://wiki.vg/Protocol_version_numbers#Versions_after_the_Netty_rewrite
https://wiki.vg/Protocol_History#1.8
https://github.com/PrismarineJS/minecraft-data/blob/master/data/pc/1.8/protocol.json

Borrows from #37 1.8.x protocol support (47)
Part of https://github.com/iceiix/steven/issues/18 Enhance protocol support

• Packet changes
• Metadata changes
• Support both chunk formats (1.8/1.9+)
• Working chunk format parsing
• Support both metadata formats (1.8/1.9+)

[iceiix]

thread '' panicked at 'bad packet id 0x26 in Clientbound Play', src/protocol/versions/v1_8_9.rs:1:1

This is MapChunkBulk, a very important packet: #37 (comment)

Fixed some shifted IDs but the biggest change is in the chunk system, https://wiki.vg/Protocol_History#15w35a "Removed Map Chunk Bulk (0x26)" this is present in 1.8.x: https://wiki.vg/index.php?title=Protocol&oldid=7368#Map_Chunk_Bulk https://wiki.vg/index.php?title=Protocol&oldid=7368#Chunk_Data "To reduce the number of bytes, this packet is used to send chunks together for better compression results". Without handling this, connecting to a server gets further but panics: thread 'main' panicked at 'Err: Err("missing packet")', src/server/mod.rs:396:33

ChunkDataBulk in steven-go for comparison: Thinkofname/steven-go@05ad2cb#diff-8b80bab7c379f59fe83cb4ee2d015ea7L169 Thinkofname/steven-go@bfe8da2#diff-8b80bab7c379f59fe83cb4ee2d015ea7R169

https://github.com/PrismarineJS/minecraft-data/blob/master/data/pc/1.8/protocol.json#L1423

[iceiix]

Parsing the chunk meta should be straightforward, a length-prefixed (VarInt) array of data similar to the meta in ChunkData:

but as for the chunk data itself (in ChunkDataBulk), an "Array of Chunk"... with the Chunk Column Count referring to "Number of elements in each of the following arrays", both meta and data arrays. How does Prismarine represent this? https://github.com/PrismarineJS/minecraft-data/blob/master/data/pc/1.8/protocol.json#L1423 packet_map_chunk_bulk:

                  "countType": "varint",
                  "type": [ "container", [
                      { "name": "x", "type": "i32" },
                      { "name": "z", "type": "i32" },
                      { "name": "bitMap", "type": "u16" }
                    ] ] } ] },
            { "name": "data", "type": "restBuffer" }

a "restBuffer"... so it reads until the end of the packet. This is possible since packet payloads are length-prefixed. restBuffer is also used in packet_custom_payload (0x3f), PluginMessageClientbound in Steven, which represents it as:

            packet PluginMessageClientbound { 
                field channel: String =,
                field data: Vec<u8> =,
            } 

so can use Vec<u8> for resBuffers. steven-go does something similar:

type ChunkDataBulk struct {
 	SkyLight     bool
 	ChunkX       []int32 `length:"VarInt"`
 	ChunkZ       []int32 `length:"VarInt"`
 	ChunkBitmask []int32 `length:"VarInt"`
 	Data         []byte  `length:"remaining"`
 }

but with a large implementation: Thinkofname/steven-go@05ad2cb#diff-442a0bf4a093c9dcdef66107c0880cbeL1804

[iceiix]

Chunk bulk data not yet parsed, but crashes elsewhere:

0x0f => SpawnMob_u8_i32
about to parse f buf Cursor { inner: [0, 15, 145, 1, 54, 0, 0, 12, 117, 0, 0, 2, 64, 0, 0, 31, 178, 52, 0, 226, 0, 0, 253, 141, 0, 0, 0, 0, 33, 1, 44, 130, 0, 3, 0, 4, 0, 102, 65, 160, 0, 0, 71, 0, 0, 0, 0, 8, 0, 9, 0, 12, 0, 13, 0, 14, 0, 15, 0, 127], pos: 2 }
ty=0
ty=0
ty=9
ty=0
ty=14
thread 'main' panicked at 'Err: Err("unknown metadata type")', src/server/mod.rs:409:33

The entity metadata format was significantly changed: https://wiki.vg/Protocol_History#15w33c

1.9+: https://wiki.vg/index.php?title=Entity_metadata&oldid=7416#Entity_Metadata_Format
1.8.9-: https://wiki.vg/index.php?title=Entity_metadata&oldid=7415#Entity_Metadata_Format

Packets using types::Metadata:

SpawnMob, SpawnMob_u8, SpawnMob_u8_i32
SpawnPlayer_f64, SpawnPlayer_i32
EntityMetadata

[iceiix]

Parsing with node-minecraft-protocol (as used in mineflayer) for comparison, 0x0f SpawnMob_u8_i32 packet:

>>> file("/tmp/p","wb").write("".join(map(chr,[15, 145, 1, 54, 0, 0, 12, 117, 0, 0, 2, 64, 0, 0, 31, 178, 52, 0, 232, 0, 0, 253, 141, 0, 0, 0, 0, 33, 1, 44, 130, 0, 3, 0, 4, 0, 102, 65, 160, 0, 0, 71, 0, 0, 0, 0, 8, 0, 9, 0, 12, 0, 13, 0, 14, 0, 15, 0, 127])))

> mc=require('./');d=mc.createDeserializer({version:"1.8",state:mc.states.PLAY});d.on('data',(parsed)=>{console.log('parsed',parsed)});d.write(fs.readFileSync("/tmp/p"))
parsed { data: 
   { name: 'spawn_entity_living',
     params: 
      { entityId: 145,
        type: 54,
        x: 3189,
        y: 576,
        z: 8114,
        yaw: 52,
        pitch: 0,
        headPitch: -24,
        velocityX: 0,
        velocityY: -627,
        velocityZ: 0,
        metadata: [Array] } },
  metadata: { size: 59 },

0f // packet 0x0f SpawnMob_u8_i32 (spawn_entity_living)
91 01 // entityId: 145, Zombie :: LivingEntity :: LivingEntityBase :: Entity
36 // type: 54
00 00 0c 75 // x: 3189
00 00 02 40 // y: 576
00 00 1f b2 // z: 8114
34 // yaw
00 // pitch
e8 // headPitch
00 00 // velocityX
fd 8d // velocityY
00 00 // velocityZ

metadata:

00 00           // 0, byte, flags
21 01 2c        // 1, short, air
82 00           // 2, string, name tag
03 00           // 3, byte, always show name tag
04 00           // 4, byte, silent
66 41 a0 00 00  // 6, float, health
47 00 00 00 00  // 7, int, potion effect color
08 00           // 8, byte, is potion ambient
09 00           // 9, byte, number of arrows
0c 00           // 12, byte, is child
0d 00           // 13, byte, is villager
0e 00           // 14, byte, is converting
0f 00           // 15, byte, AI disabled
7f              // end 

[iceiix]

Got metadata parsing working:

parsed metadata = Metadata[ 7=Int(0), 14=Byte(0), 15=Byte(0), 13=Byte(0), 9=Byte(0), 8=Byte(0), 12=Byte(0), ]
parsed metadata = Metadata[ 8=Byte(0), 2=String(""), 13=Byte(0), 12=Byte(0), 7=Int(0), 15=Byte(0), 3=Byte(0), 9=Byte(0), 0=Byte(0), 1=Short(300), 4=Byte(0), 14=Byte(0), 6=Float(20.0), ]
parsed metadata = Metadata[ 7=Int(0), 9=Byte(0), 14=Byte(0), 13=Byte(0), 12=Byte(0), 15=Byte(0), 8=Byte(0), ]
parsed metadata = Metadata[ 8=Byte(0), 2=String(""), 14=Byte(0), 6=Float(20.0), 7=Int(0), 12=Byte(0), 13=Byte(0), 4=Byte(0), 1=Short(300), 15=Byte(0), 0=Byte(0), 3=Byte(0), 9=Byte(0), ]
parsed metadata = Metadata[ 14=Byte(0), 12=Byte(0), 7=Int(0), 8=Byte(0), 15=Byte(0), 13=Byte(0), 9=Byte(0), ]
parsed metadata = Metadata[ 8=Byte(0), 6=Float(2.0), 2=String(""), 0=Byte(0), 9=Byte(0), 3=Byte(0), 7=Int(0), 1=Short(300), 15=Byte(0), 4=Byte(0), 12=Byte(0), 14=Byte(0), 13=Byte(0), ]
parsed metadata = Metadata[ 15=Byte(0), 7=Int(0), 13=Byte(0), 9=Byte(0), 8=Byte(0), ]
parsed metadata = Metadata[ 3=Byte(0), 4=Byte(0), 6=Float(20.0), 1=Short(300), 7=Int(0), 0=Byte(0), 2=String(""), 15=Byte(0), 8=Byte(0), 13=Byte(0), 9=Byte(0), ]

however, this currently replaces the 1.9+ metadata format in src/types/metadata.rs, will need to think about how to support both (parameterized? separate type?), next crash is at:

thread 'main' panicked at 'Err: Err("Failed to read all of packet 0x4, had 1 bytes left")', src/server/mod.rs:409:33

        0x04 => EntityEquipment

            packet EntityEquipment {
                field entity_id: VarInt =,
                field slot: VarInt =,
                field item: Option<item::Stack> =,
            }

https://wiki.vg/index.php?title=Protocol&oldid=7368#Entity_Equipment changed slot to Short

[iceiix]

No longer instantly crashes, but after logging in for some time, a different crash:

thread 'main' panicked at 'attempt to shift left with overflow', src/types/bit/map.rs:95:20
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at libstd/sys_common/backtrace.rs:59
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get
             at libstd/panicking.rs:476
   5: std::panicking::continue_panic_fmt
             at libstd/panicking.rs:390
   6: std::panicking::try::do_call
             at libstd/panicking.rs:325
   7: core::char::methods::<impl char>::escape_debug
             at libcore/panicking.rs:77
   8: core::char::methods::<impl char>::escape_debug
             at libcore/panicking.rs:52
   9: steven::types::bit::map::Map::get
             at src/types/bit/map.rs:95
  10: steven::world::World::load_chunk
             at src/world/mod.rs:600
  11: steven::server::Server::on_chunk_data_no_entities_u16
             at src/server/mod.rs:974
  12: steven::server::Server::entity_tick
             at src/server/mod.rs:376
  13: steven::server::Server::tick
             at src/server/mod.rs:339
  14: steven::main
             at src/main.rs:268
  15: std::rt::lang_start::{{closure}}
             at /rustc/e6c5cf923489f4aed6c9e36a170fab9b6179d8da/src/libstd/rt.rs:74
  16: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:310
  17: panic_unwind::dwarf::eh::read_encoded_pointer
             at libpanic_unwind/lib.rs:102
  18: std::panicking::update_count_then_panic
             at libstd/panicking.rs:289
             at libstd/panic.rs:392
             at libstd/rt.rs:58
  19: std::rt::lang_start
             at /rustc/e6c5cf923489f4aed6c9e36a170fab9b6179d8da/src/libstd/rt.rs:74

related to chunk loading, so likely caused by missing ChunkDataBulk handling

[iceiix]

1.8 chunk format: https://wiki.vg/index.php?title=Chunk_Format&oldid=6124
1.9 chunk format: https://wiki.vg/index.php?title=Chunk_Format&oldid=7411
1.8 vs 1.9: https://wiki.vg/index.php?title=Chunk_Format&diff=7411&oldid=6124

1.9 (from 1.8) in prismarine: https://github.com/PrismarineJS/prismarine-chunk/pull/23/files

https://github.com/PrismarineJS/prismarine-chunk/blob/master/src/pc/1.8/section.js
https://github.com/PrismarineJS/prismarine-chunk/blob/master/src/pc/1.8/chunk.js

[iceiix]

Created a new MetadataBase trait, with default implementations, this avoids the code duplication for get/put/put_raw. But how to implement other traits on this trait? The fmt::Debug trait implementations are identical for Metadata18 and Metadata19:

// TODO: reduce duplication
impl fmt::Debug for Metadata19 {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "Metadata[ ")?;
        for (k, v) in &self.map {
            write!(f, "{:?}={:?}, ", k, v)?;
        }
        write!(f, "]")
    }
}
impl fmt::Debug for Metadata18 {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "Metadata[ ")?;
        for (k, v) in &self.map {
            write!(f, "{:?}={:?}, ", k, v)?;
        }
        write!(f, "]")
    }
}

as is the Default trait:

// TODO: reduce duplication
impl Default for Metadata19 {
    fn default() -> Self {
        Self::new()
    }
}
impl Default for Metadata18 {
    fn default() -> Self {
        Self::new()
    }
}

https://doc.rust-lang.org/book/ch10-02-traits.html#using-trait-bounds-to-conditionally-implement-methods says it is possible to implement blanket methods like impl<T: Display> ToString for T {, full code at https://habr.com/post/274455/:

impl<T: Display> ToString for T {
    fn to_string(&self) -> String {
        format!("{}", self)
    }
}

but fails because the Display trait isn't implemented in this crate. Attempt:

 impl<T: MetadataBase> fmt::Debug for T {
      fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
          write!(f, "Metadata[ ")?;
         for (k, v) in &self.map {
              write!(f, "{:?}={:?}, ", k, v)?;
          }
          write!(f, "]")
      }
  }

fails:

rror[E0119]: conflicting implementations of trait `std::fmt::Debug` for type `&_`:                                                                                                                                                                         
   --> src/types/metadata.rs:314:1                                                                                                                                                                                                                          
    |                                                                                                                                                                                                                                                       
314 | impl<T: MetadataBase> fmt::Debug for T {                                                                                                                                                                                                              
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                
    |                                                                                                                                                                                                                                                       
    = note: conflicting implementation in crate `core`:                                                                                                                                                                                                     
            - impl<'_, T> std::fmt::Debug for &T                                                                                                                                                                                                            
              where T: std::fmt::Debug, T: ?Sized;                                                                                                                                                                                                          
    = note: downstream crates may implement trait `types::metadata::MetadataBase` for type `&_`                                                                                                                                                             
                                                                                                                                                                                                                                                            
error[E0210]: type parameter `T` must be used as the type parameter for some local type (e.g. `MyStruct<T>`)                                                                                                                                                
   --> src/types/metadata.rs:314:1                                                                                                                                                                                                                          
    |                                                                                                                                                                                                                                                       
314 | impl<T: MetadataBase> fmt::Debug for T {                                                                                                                                                                                                              
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ type parameter `T` must be used as the type parameter for some local type                                                                                                                                      
    |                                                                                                                                                                                                                                                       
    = note: only traits defined in the current crate can be implemented for a type parameter                                                                                                                                                                
                                                                                                                                                                                                                                                            
error: aborting due to 2 previous errors                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                            
Some errors occurred: E0119, E0210.                                                                                                                                                                                                                         
For more information about an error, try `rustc --explain E0119`.       

https://stackoverflow.com/questions/50040596/how-do-i-derive-a-trait-for-another-trait gives this example:

trait Contract: std::fmt::Debug {}

and points to http://smallcultfollowing.com/babysteps/blog/2015/01/14/little-orphan-impls/. trait MetadataBase: fmt::Debug { compiles, but where to go from there? If I add the fmt() method in that impl, it doesn't implement fmt::Debug.

[iceiix]

Maybe could be reduced in size with better knowledge of Rust, but this works well enough for now