create packet structure

[adambujak]

No description provided.

[stephen99scott]

@adambujak @harneetMac Here's my first stab at the packet structure. 40 represents the number of samples per gesture. Specifying the max/fixed count makes it easier to populate the arrays with data.

syntax = "proto2";

message DataPacket {
    repeated sint32 quaternion_w = 1 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated sint32 quaternion_x = 2 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated sint32 quaternion_y = 3 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated sint32 quaternion_z = 4 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 flex_thumb = 5 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 flex_index = 6 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 flex_middle = 7 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 flex_ring = 8 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 flex_little = 9 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 touch1 = 10 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 touch2 = 11 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 touch3 = 12 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 touch4 = 13 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 touch5 = 14 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 touch6 = 15 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 touch7 = 16 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 touch8 = 17 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 touch9 = 18 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 touch10 = 19 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
    repeated uint32 touch11 = 20 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
}

[stephen99scott]

We could pad the message with a header/footer at transmission with something like (where 0x16 and 0x27 are arbitrary values):

{
   uint8_t SYNC_BEGIN 0x16
   uint16_t SIZE
   DataPacket
   uint8_t SYNC_END 0x27
}

[adambujak]

if we're framing it ourselves i think we should add a crc

[stephen99scott]

Yeah a crc would be good to have. Also, it might make more sense to structure the data like this with a nested repeated message:

message DataPacket {
	repeated Sample data = 1 [(nanopb).max_count = 40, (nanopb).fixed_count = true];
}

message Sample {
    required sint32 quaternion_w = 1;
    required sint32 quaternion_x = 2;
    required sint32 quaternion_y = 3;
    required sint32 quaternion_z = 4;
    required uint32 flex_thumb = 5;
    required uint32 flex_index = 6;
    required uint32 flex_middle = 7;
    required uint32 flex_ring = 8;
    required uint32 flex_little = 9;
    required uint32 touch1 = 10;
    required uint32 touch2 = 11;
    required uint32 touch3 = 12;
    required uint32 touch4 = 13;
    required uint32 touch5 = 14;
    required uint32 touch6 = 15;
    required uint32 touch7 = 16;
    required uint32 touch8 = 17;
    required uint32 touch9 = 18;
    required uint32 touch10 = 19;
    required uint32 touch11 = 20;
}

[harneetMac]

FYI, quaternion data is sint16 (signed 2 bytes).

[stephen99scott]

Protobuf doesn't support int16 as far as I'm aware

[harneetMac]

Also, maybe you read this already. Google's website on protobuff warns users from using "required" field.

[harneetMac]

Protobuf doesn't support int16 as far as I'm aware

yeah you are right

[stephen99scott]

I think they don't recommend required fields due to potential backwards compatibility issues if you change your packet structure. That's mostly a concern if you have an application with many users and you want to make sure users with old versions are still supported by the server in the future. For our case it makes sense for all fields to be required

[stephen99scott]

@adambujak @harneetMac What do you think of this structure? GestureData is only used on the Android side to construct the packet with all the GestureData. I was thinking including the letter in the data could make it easier for us for training. I.e. we can save a bunch of delimited protobuf packets to a file which we can copy to a desktop to train with. Also the sample packets include the sample id to ensure that the samples are placed in the correct order in the gesture data.

message GestureData {
    required string letter = 1;
    repeated Sample samples = 2;
}

message Sample {
    required uint32 sample_id = 1;

    message IMUData {
        required sint32 quaternion_w = 1;
        required sint32 quaternion_x = 2;
        required sint32 quaternion_y = 3;
        required sint32 quaternion_z = 4;
        required sint32 lin_acc_x = 5;
        required sint32 lin_acc_y = 6;
        required sint32 lin_acc_z = 7;
    }

    required IMUData imuData = 2;

    message FlexData {
        required uint32 flex_thumb = 1;
        required uint32 flex_index = 2;
        required uint32 flex_middle = 3;
        required uint32 flex_ring = 4;
        required uint32 flex_little = 5;
    }

    required FlexData flexData = 3;

    message TouchData {
        required uint32 touch1 = 1;
        required uint32 touch2 = 2;
        required uint32 touch3 = 3;
        required uint32 touch4 = 4;
        required uint32 touch5 = 5;
        required uint32 touch6 = 6;
        required uint32 touch7 = 7;
        required uint32 touch8 = 8;
        required uint32 touch9 = 9;
        required uint32 touch10 = 10;
        required uint32 touch11 = 11;
    }

    required TouchData touchData = 4;
}

We would transmit Sample packets to the Android app padded like this:

{
   uint8_t size = sizeof(sample) + sizeof(CRC)
   Sample sample
   int32_t crc = crc(sample)
}

[stephen99scott]

Switching the touch sensors to bool brings the maximum encoded size down to 100 bytes:

message GestureData {
    required string letter = 1;
    repeated Sample samples = 2;
}

message Sample {
    required uint32 sample_id = 1;

    message IMUData {
        required sint32 eul_p = 1;
        required sint32 eul_r = 2;
        required sint32 eul_y = 3;
        required sint32 lin_acc_x = 4;
        required sint32 lin_acc_y = 5;
        required sint32 lin_acc_z = 6;
    }

    required IMUData imuData = 2;

    message FlexData {
        required uint32 flex_thumb = 1;
        required uint32 flex_index = 2;
        required uint32 flex_middle = 3;
        required uint32 flex_ring = 4;
        required uint32 flex_little = 5;
    }

    required FlexData flexData = 3;

    message TouchData {
        required bool touch1 = 1;
        required bool touch2 = 2;
        required bool touch3 = 3;
        required bool touch4 = 4;
        required bool touch5 = 5;
        required bool touch6 = 6;
        required bool touch7 = 7;
        required bool touch8 = 8;
        required bool touch9 = 9;
        required bool touch10 = 10;
        required bool touch11 = 11;
    }

    required TouchData touchData = 4;
}