1.12.2 protocol support (340)

[iceiix]

https://github.com/iceiix/steven/issues/18
https://wiki.vg/Protocol_History#1.12.2
https://wiki.vg/index.php?title=Protocol&oldid=14204

[iceiix]

parsing packet id 3c direction Clientbound in state Play
thread 'main' panicked at 'Err: Err("unknown metadata type")', src/server/mod.rs:397:33

pub const EntityMetadata: i32 = 0x3c;

src/types/metadata.rs

                _ => return Err(protocol::Error::Err("unknown metadata type".to_owned())),

https://wiki.vg/index.php?title=Entity_metadata&oldid=14048#Entity_Metadata_Format

New metadata type is 13, which is "NBT Tag" https://wiki.vg/NBT

[iceiix]

Gets to the terrain:

but crashes shortly after, some encoding error:

parsing packet id 3c direction Clientbound in state Play
write packet TeleportConfirm { teleport_id: 1 }
thread 'main' panicked at 'Err: IOError(Custom { kind: InvalidData, error: StringError("stream did not contain valid UTF-8") })', src/server/mod.rs:397:33

7: steven::server::Server::entity_tick
at src/server/mod.rs:397
8: steven::server::Server::tick
at src/server/mod.rs:336
9: steven::main
at src/main.rs:268

                    Err(err) => panic!("Err: {:?}", err),

Packet handling in entity_tick(). An error returned from rx.try_recv()?

entity_tick pck = Ok(EntityLookAndMove(EntityLookAndMove { entity_id: 1573, delta_x: -1078, delta_y: -636, delta_z: 317, yaw: 52, pitch: 0, on_ground: false }))
entity_tick pck = Ok(EntityHeadLook(EntityHeadLook { entity_id: 1573, head_yaw: 46 }))
entity_tick pck = Ok(EntityLookAndMove(EntityLookAndMove { entity_id: 574, delta_x: -573, delta_y: 228, delta_z: 128, yaw: 54, pitch: 0, on_ground: false }))
entity_tick pck = Ok(EntityHeadLook(EntityHeadLook { entity_id: 574, head_yaw: 108 }))
entity_tick pck = Err(IOError(Custom { kind: InvalidData, error: StringError("stream did not contain valid UTF-8") }))
thread 'main' panicked at 'Err: IOError(Custom { kind: InvalidData, error: StringError("stream did not contain valid UTF-8") })', src/server/mod.rs:398:33

The error is definitely from nbt::Tag::read_from(buf):

                     match nbt::Tag::read_from(buf) {
                          Ok(nbt) => m.put_raw(index, nbt),
                          Err(err) => println!("Metadata NBT tag parsing error: {:?}", err),
                      }   

Metadata NBT tag parsing error: IOError(Custom { kind: InvalidData, error: StringError("stream did not contain valid UTF-8") })

src/nbt/mod.rs:

pub fn read_string<R: io::Read>(buf: &mut R) -> Result<String, protocol::Error> { 
    let len: i16 = buf.read_i16::<BigEndian>()?;
    let mut ret = String::new();
    buf.take(len as u64).read_to_string(&mut ret)?;
    Result::Ok(ret)
}

https://doc.rust-lang.org/nightly/std/io/trait.Read.html#method.read_to_string

If the data in this stream is not valid UTF-8 then an error is returned and buf is unchanged.

[iceiix]

i16 is 3338 (0x0d0a), reading into a Vec with read_to_end instead of read_to_string:

pub fn read_string<R: io::Read>(buf: &mut R) -> Result<String, protocol::Error> { 
    let len: i16 = buf.read_i16::<BigEndian>()?;
    let mut ret = String::new();
    println!("read_string len = {:?}", len);
    if len > 3000 { 
        let mut vec = vec![];
        let result = buf.take(len as u64).read_to_end(&mut vec);
        println!("vec = {:?}", vec);
        println!("result = {:?}", result);
    } 

    buf.take(len as u64).read_to_string(&mut ret)?;
    Result::Ok(ret)
}

bytes are [0, 0, 0, 255]. Bytes 0x0d0a for length are suspicious, CRLF? 0d 0a 00 00 00 ff, or a coincidence. 13, 10 (the last terminating sequence is 255 = 0xff):

read Ok(69) = [0, 60, 188, 34, 0,
0, 0,
1, 1, 172, 2,
2, 3, 0,
3, 6, 0,
4, 6, 0,
5, 6, 0,
6, 0, 0,
7, 2, 65, 160, 0, 0,
8, 1, 0,
9, 6, 0,
10, 1, 0,
11, 2, 0, 0, 0, 0,
12, 1, 0,
13, 0, 0,
14, 0, 1,
15, 13, 10, 0, 0, 0, 16, 13, 10, 0, 0, 0,
255]

parsing packet id 3c direction Clientbound in state Play
metadata read_from index=0, ty=0
metadata read_from ok Metadata[ 0=Byte(0), ]
metadata read_from index=1, ty=1
metadata read_from ok Metadata[ 0=Byte(0), 1=Int(300), ]
metadata read_from index=2, ty=3
metadata read_from ok Metadata[ 0=Byte(0), 1=Int(300), 2=String(""), ]
metadata read_from index=3, ty=6
metadata read_from ok Metadata[ 3=Bool(false), 0=Byte(0), 1=Int(300), 2=String(""), ]
metadata read_from index=4, ty=6
metadata read_from ok Metadata[ 4=Bool(false), 3=Bool(false), 0=Byte(0), 1=Int(300), 2=String(""), ]
metadata read_from index=5, ty=6
metadata read_from ok Metadata[ 4=Bool(false), 3=Bool(false), 0=Byte(0), 1=Int(300), 2=String(""), 5=Bool(false), ]
metadata read_from index=6, ty=0
metadata read_from ok Metadata[ 4=Bool(false), 3=Bool(false), 0=Byte(0), 1=Int(300), 2=String(""), 6=Byte(0), 5=Bool(false), ]
metadata read_from index=7, ty=2
metadata read_from ok Metadata[ 4=Bool(false), 3=Bool(false), 0=Byte(0), 1=Int(300), 7=Float(20.0), 2=String(""), 6=Byte(0), 5=Bool(false), ]
metadata read_from index=8, ty=1
metadata read_from ok Metadata[ 4=Bool(false), 3=Bool(false), 0=Byte(0), 1=Int(300), 7=Float(20.0), 2=String(""), 6=Byte(0), 8=Int(0), 5=Bool(false), ]
metadata read_from index=9, ty=6
metadata read_from ok Metadata[ 4=Bool(false), 3=Bool(false), 0=Byte(0), 1=Int(300), 7=Float(20.0), 2=String(""), 6=Byte(0), 8=Int(0), 5=Bool(false), 9=Bool(false), ]
metadata read_from index=10, ty=1
metadata read_from ok Metadata[ 4=Bool(false), 3=Bool(false), 0=Byte(0), 1=Int(300), 7=Float(20.0), 2=String(""), 6=Byte(0), 10=Int(0), 8=Int(0), 5=Bool(false), 9=Bool(false), ]
metadata read_from index=11, ty=2
metadata read_from ok Metadata[ 4=Bool(false), 3=Bool(false), 0=Byte(0), 1=Int(300), 11=Float(0.0), 7=Float(20.0), 2=String(""), 6=Byte(0), 10=Int(0), 8=Int(0), 5=Bool(false), 9=Bool(false), ]
metadata read_from index=12, ty=1
metadata read_from ok Metadata[ 4=Bool(false), 12=Int(0), 3=Bool(false), 0=Byte(0), 1=Int(300), 11=Float(0.0), 7=Float(20.0), 2=String(""), 6=Byte(0), 10=Int(0), 8=Int(0), 5=Bool(false), 9=Bool(false), ]
metadata read_from index=13, ty=0
metadata read_from ok Metadata[ 4=Bool(false), 12=Int(0), 3=Bool(false), 0=Byte(0), 1=Int(300), 11=Float(0.0), 7=Float(20.0), 2=String(""), 6=Byte(0), 13=Byte(0), 10=Int(0), 8=Int(0), 5=Bool(false), 9=Bool(false), ]
metadata read_from index=14, ty=0
metadata read_from ok Metadata[ 4=Bool(false), 14=Byte(1), 12=Int(0), 3=Bool(false), 0=Byte(0), 1=Int(300), 11=Float(0.0), 7=Float(20.0), 2=String(""), 6=Byte(0), 13=Byte(0), 10=Int(0), 8=Int(0), 5=Bool(false), 9=Bool(false), ]
metadata read_from index=15, ty=13
read_string len = 0
read_string len = 3338
vec = [0, 0, 0, 255]

[iceiix]

"15, 13, 10, 0, 0, 0, 16, 13, 10, 0, 0, 0, 255" looks more like it should be maybe parsed as:

15, 13, 10, 0, 0, 0,
16, 13, 10, 0, 0, 0,
255

(Note that metadata type 13 is NBT in 1.12.2, but changed in 1.13.2 to OptBlockID (VarInt) and NBT is 14.)

https://wiki.vg/NBT says TAG_Compound is 10 and that every file will begin with this.

I.e. if it's inside a Compound, then each tag will begin with the TAG_id, and then a string (the tag's name), and finally the payload

This is why it is trying to parse a String. What actually is this?

[iceiix]

It is technically possible to skip unparsed packets, since the length field is included, though not cleanest to do here, trying it for now to make progress. 0x3c EntityMetadata and 0x4d Advancements. Something about the entity metadata's NBT fields, and advancements structure is yet to be written. But now it fails on:

thread 'main' panicked at 'Err: Err("Failed to read all of packet 0x1F, had 7 bytes left")', src/server/mod.rs:398:33

[iceiix]

10, 0, 0, 0 is supposed to parse as an empty compound NBT tag. But src/nbt/mod.rs read_from calls Tag::read_type(10, buf), already passing the type byte (10). python3 nbtlib on Linux doesn't seem to parse correctly, or I am misusing it, but this library does: https://github.com/PrismarineJS/prismarine-nbt

> require('prismarine-nbt').parse(new Buffer('\x0a\x00\x00\x00'), console.log)

and DEBUG=minecraft-protocol node examples/echo.js localhost 25565 in https://github.com/PrismarineJS/mineflayer (a useful multi-protocol client, albeit command-line only, and written in JavaScript) shows how the packet is supposed to be parsed:

  minecraft-protocol read packet play.entity_metadata +1ms
  minecraft-protocol {
  minecraft-protocol   "entityId": 375,
  minecraft-protocol   "metadata": [
  minecraft-protocol     {
  minecraft-protocol       "key": 0,
  minecraft-protocol       "type": 0,
  minecraft-protocol       "value": 0
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 1,
  minecraft-protocol       "type": 1,
  minecraft-protocol       "value": 300
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 2,
  minecraft-protocol       "type": 3,
  minecraft-protocol       "value": ""
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 3,
  minecraft-protocol       "type": 6,
  minecraft-protocol       "value": false
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 4,
  minecraft-protocol       "type": 6,
  minecraft-protocol       "value": false
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 5,
  minecraft-protocol       "type": 6,
  minecraft-protocol       "value": false
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 6,
  minecraft-protocol       "type": 0,
  minecraft-protocol       "value": 0
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 7,
  minecraft-protocol       "type": 2,
  minecraft-protocol       "value": 20
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 8,
  minecraft-protocol       "type": 1,
  minecraft-protocol       "value": 0
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 9,
  minecraft-protocol       "type": 6,
  minecraft-protocol       "value": false
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 10,
  minecraft-protocol       "type": 1,
  minecraft-protocol       "value": 0
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 11,
  minecraft-protocol       "type": 2,
  minecraft-protocol       "value": 0
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 12,
  minecraft-protocol       "type": 1,
  minecraft-protocol       "value": 0
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 13,
  minecraft-protocol       "type": 0,
  minecraft-protocol       "value": -1
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 14,
  minecraft-protocol       "type": 0,
  minecraft-protocol       "value": 0
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 15,
  minecraft-protocol       "type": 13,
  minecraft-protocol       "value": {
  minecraft-protocol         "type": "compound",
  minecraft-protocol         "name": "",
  minecraft-protocol         "value": {}
  minecraft-protocol       }
  minecraft-protocol     },
  minecraft-protocol     {
  minecraft-protocol       "key": 16,
  minecraft-protocol       "type": 13,
  minecraft-protocol       "value": {
  minecraft-protocol         "type": "compound",
  minecraft-protocol         "name": "",
  minecraft-protocol         "value": {}
  minecraft-protocol       }
  minecraft-protocol     }
  minecraft-protocol   ]
  minecraft-protocol } +0ms

so it is indeed two empty compound NBT tags (15, 13, 10, 0, 0, 0 and 16, 13, 10, 0, 0, 0). The parsing in Steven is somehow getting desynchronized. What is a similar packet using NBT which can be used as an example? Slot takes NBT, example packet: ClickWindow, but Steven represents Slot as Option<item::Stack>. UpdateBlockEntity (0x09) also has a nbt field, represented as Option<nbt::NamedTag>. Its trait Serializable implementation is in src/protocol/mod.rs:

impl Serializable for Option<nbt::NamedTag>{
    fn read_from<R: io::Read>(buf: &mut R) -> Result<Option<nbt::NamedTag>, Error> {
        let ty = buf.read_u8()?;
        if ty == 0 {
            Result::Ok(None)
        } else {
            let name = nbt::read_string(buf)?;
            let tag = nbt::Tag::read_from(buf)?;
            Result::Ok(Some(nbt::NamedTag(name, tag)))
        }
    }
    fn write_to<W: io::Write>(&self, buf: &mut W) -> Result<(), Error> {
        match *self {
            Some(ref val) => {
                buf.write_u8(10)?;
                nbt::write_string(buf, &val.0)?;
                val.1.write_to(buf)?;
            }
            None => buf.write_u8(0)?,
        }
        Result::Ok(())
    }
}

but Option<nbt::NamedTag> is an optional NBT. "May be a TAG_END (0)". TAG_END is "It is only ever used inside a TAG_Compound, and is not named despite being in a TAG_Compound"

TAG_STRING is "A length-prefixed UTF-8 string. The prefix is an unsigned short (thus 2 bytes) signifying the length of the string in bytes". Is this why there are three zeros, the first 00 00 immediately after 0a is the tag name, zero-length string? Then 00 is TAG_END? prismarine-js requires the full '\x0a\x00\x00\x00' buffer, it errors on parsing truncations '\x0a\x00\x00' (PartialReadError: Read error for value.compound : undefined), '\x0a\x00' (PartialReadError: Read error for name.$count : undefined), '\x0a' (same). But why does src/nbt/mod.rs read_type on compound tags bail early?

            10 => {
                let mut c = Tag::new_compound();
                loop {
                    let ty = buf.read_u8()?;
                    if ty == 0 {
                        break;
                    }
                    let name: String = read_string(buf)?;
                    c.put(&name[..], Tag::read_type(ty, buf)?);
                }
                Ok(c)
            }

but if I change to break later, after reading the top-level string name:

            10 => {
                let mut c = Tag::new_compound();
                loop {
                    let ty = buf.read_u8()?;
                    let name: String = read_string(buf)?;
                    if ty == 0 {
                        break;
                    }
                    c.put(&name[..], Tag::read_type(ty, buf)?);
                }
                Ok(c)
            }

then thread 'main' panicked at 'Err: IOError(Custom { kind: InvalidData, error: StringError("stream did not contain valid UTF-8") })', src/server/mod.rs:398:33. The last read string is:

'\rMaxSpawnDelay\x03 \x02\x00\x05Delay\x00\x14\x03\x00\x01x\x00\x00\x00}\x03\x00\x01y\x00\x00\x00\x1a\x03\x00\x01z\x00\x00\x00\xdc\x08\x00\x02id\x00\x15minecraft:mob_spawner\x02\x00\nSpawnRange\x00\x04\x02\x00\rMinSpawnDelay\x00\xc8\x00\n\x00\x00\x03\x00\x01x\x00\x00\x00{\x03\x00\x01y\x00\x00\x00\x1a\x03\x00\x01z\x00\x00\x00\xdb\x08\x00\x02id\x00\x0fminecraft:chest\x00'.decode('utf8')
Traceback (most recent call last):
File "", line 1, in
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xdc in position 49: invalid continuation byte

[iceiix]

Parsing similarly to Optionnbt::NamedTag seems to work:

metadata read_from index=15, ty=13
metadata read_from ok Metadata[ 14=Byte(1), 5=Bool(false), 4=Bool(false), 13=Byte(0), 15=NBTTag(NamedTag("", Compound({}))), 11=Float(0.0), 1=Int(300), 8=Int(0), 7=Float(20.0), 3=Bool(false), 0=Byte(0), 10=Int(0), 2=String(""), 6=Byte(0), 9=Bool(false), 12=Int(0), ]
metadata read_from index=16, ty=13
metadata read_from ok Metadata[ 14=Byte(1), 5=Bool(false), 4=Bool(false), 13=Byte(0), 15=NBTTag(NamedTag("", Compound({}))), 11=Float(0.0), 1=Int(300), 8=Int(0), 7=Float(20.0), 3=Bool(false), 16=NBTTag(NamedTag("", Compound({}))), 10=Int(0), 0=Byte(0), 2=String(""), 6=Byte(0), 9=Bool(false), 12=Int(0), ]
parsed packet = Some(EntityMetadata(EntityMetadata { entity_id: 2290, metadata: Metadata[ 14=Byte(1), 5=Bool(false), 4=Bool(false), 13=Byte(0), 15=NBTTag(NamedTag("", Compound({}))), 11=Float(0.0), 1=Int(300), 8=Int(0), 7=Float(20.0), 3=Bool(false), 16=NBTTag(NamedTag("", Compound({}))), 10=Int(0), 0=Byte(0), 2=String(""), 6=Byte(0), 9=Bool(false), 12=Int(0), ] }))

[iceiix]

thread 'main' panicked at 'Err: Err("Failed to read all of packet 0x2B, had 1 bytes left")', src/server/mod.rs:398:33
https://wiki.vg/index.php?title=Protocol&oldid=14204#Craft_Recipe_Response

[iceiix]

read Ok(37) = [0, 26, 34, 123, 34, 116, 114, 97, 110, 115, 108, 97, 116, 101, 34, 58, 34, 100, 105, 115, 99, 111, 110, 110, 101, 99, 116, 46, 116, 105, 109, 101, 111, 117, 116, 34, 125]
parsing packet id 1a direction Clientbound in state Play
parsed packet = Some(Disconnect(Disconnect { reason: Text(TextComponent { text: "UNHANDLED", modifier: Modifier { extra: None, bold: None, italic: None, underlined: None, strikethrough: None, obfuscated: None, color: Some(RGB(255, 0, 0)) } }) }))
read Ok(0) = [0]

"UNHANDLED" is a translation problem in src/format.rs, this is actually: '\x00\x1a"{"translate":"disconnect.timeout"}'

[iceiix]

The first advancement:

{
  "data": {
    "name": "advancements",
    "params": {
      "reset": true,
      "advancementMapping": [
        {
          "key": "minecraft:adventure\/sleep_in_bed",
          "value": {
            "parentId": "minecraft:adventure\/root",
            "displayData": {
              "title": "{\"translate\":\"advancements.adventure.sleep_in_bed.title\"}",
              "description": "{\"translate\":\"advancements.adventure.sleep_in_bed.description\"}",
              "icon": {
                "blockId": 355,
                "itemCount": 1,
                "itemDamage": 0
              },
              "frameType": 0,
              "flags": { "_unused": 0, "hidden": 0,  "show_toast": 1,  "has_background_texture": 0  },
              "xCord": 1,
              "yCord": 4
            },
            "criteria": [ { "key": "slept_in_bed"  }  ],
          }

bytes from icon:

01 63 // 355 blockId
01 // 1 count
00 00 // 0 damage
00 00 // NBT = empty?
00 00 00 02 // 0x2 show_toast, flags
3f 80 00 00 // 1.0 xCord (convert with https://babbage.cs.qc.cuny.edu/IEEE-754.old/32bit.html)
40 80 00 00 // 4.0 yCord
01 0c // varint number of criteria
"slept_in_bed" // criteria key

something is seriously desynchronized, seems to be reading the blockId from the end of flags and beginning of xCord (02 3f = 575), then count from the middle of xCord (0x80 = 128), then damage from teh end of xCord (00 00 = 0), then the type of NBT from the beginning of yCord (0x40 = 64), reading too late in the stream, what gobbled it up earlier?

Advancement read_from
AdvancementDisplay read_from
about to read_from title
about to read_from description
about to read_from icon
Option<Stack> Serializable read_from
Option<Stack> Serializable read_from id=575
Option<Stack> Serializable read_from id=575
count=128
damage=0
Option<nbt::NamedTag read_from
ty = 64
write packet TeleportConfirm { teleport_id: 1 }
thread 'main' panicked at 'Err: IOError(Custom { kind: InvalidData, error: StringError("stream did not contain valid UTF-8") })', src/server/mod.rs:398:33
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Advancement read_from
AdvancementDisplay read_from
about to read_from title
title = "inecraft:adventure/root\u{1}9{"translate":"advancements.adventure.sleep_in_bed.title"}?{"translate":"advancements"
about to read_from description
description = "adventure.sleep_in_bed.description"}\u{1}c\u{1}\u{0}\u{0}\u{0}\u{0}\u{0}\u{0}\u{0}"
about to read_from icon

earlier in Advancement, parent_id is read wrong:

Advancement read_from
id = "minecraft:adventure/sleep_in_bed"
parent_id = Some("\u{18}")

"minecraft:adventure/sleep_in_bed" // id, parses ok
01 // has_parent, Some
18 ??? parsed as "\u"{18}"
"minecraft:adventure/root"

[iceiix]

Implemented the Option parsing (curiously, would have thought deserializing Option<> would do it for me, maybe only in packet definitions?), now it gets much further, but crashes at the end:

Advancement read_from
id = "minecraft:recipes/root"
parent_id = None
display_data = None
criteria = ["impossible"]
requirements = [["impossible"]]
write packet TeleportConfirm { teleport_id: 1 }
thread 'main' panicked at 'Err: IOError(Custom { kind: UnexpectedEof, error: StringError("failed to fill whole buffer") })', src/server/mod.rs:398:33

This means the Advancement mapping is now parsing entirely, but the next fields are not. Expect "identifiers": [] (array of strings, varint length), then progress mapping array of advancement progress. I have both of these defined inline:

                field identifiers: LenPrefixed<VarInt, String> =,
                field progress: LenPrefixed<VarInt, LenPrefixed<VarInt, Option<i64>>> =,

but progress is clearly wrong, missing two identifier strings: "The identifier of the advancement" and "The identifier of the criterion.", can't get away with simple definitions here (though maybe for criterion progress).

[iceiix]

This is about done, seems to work well. Going to use 1.12.2 as the mainline version for now, can backport to 1.11.2 later for https://github.com/iceiix/steven/issues/18 by undoing this merge.