Notes to self

• Our proposed grammar is flexible enough to describe any binary format. However, the parser generator would have to be very complex to fully support the grammer (probably similar to 'real' parser generators like ANTLR in complexity. It may be worthwhile to introduce some restrictions (in lexical analysis) to reduce the complexity.

  • Fixed fields must be the same for all (root level) objects in type and size and only one fixed value may be different for different objects. e.g.

    Object1
    {
        header : uint32 = 0x01020304;
        message_type : uint8 = 1;
        field1 : int16;
        field2 : float32;
        (...)
        footer : uint32 = 0x04030201;    
    }
    
    Object2
    {
        field1 : int16;
        message_type : uint8 = 1;        
        field2 : float32;
        (...)  
    }
    

    is not allowed.

    This would make generating a parser a lot easier since we don't need complex iteration of tokens and backtracking. Instead we can simply find the start of a packet and determine the type based on the differing fixed value.

• The current definition of the grammar can support versioning in protocols by using fixed fields:

some_object_v1
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 = 1;
    uint32[6];
}

some_object_v2
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 = 2;
    field_exclusive_to_v2 : uint16;
    uint16;
    uint32[5];
}

However, this method will not work on versions of the protocol that are not defined. Even if these future versions are backwards compatible (as v2 in this example is backwards compatible with v1). There are a number of ways to solve this. We could for instance introduce introduce relational operators:

some_object_v1
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 = 1;
    uint32[6];
}

some_object_v2
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 >= 2;
    field_exclusive_to_v2 : uint16;
    uint16;
    uint32[5];
}

Another option would be to add some type of qualifier to the field

some_object_v1
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 = 1;
    uint32[6];
}

some_object_v2
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 = 2 <version>;
    field_exclusive_to_v2 : uint16;
    uint16;
    uint32[5];
}

The latter would only be worthwhile if there are other reasons to add qualifier fields.

The third option would be to not add version support to the parser, but instead rely on the client of the parser to handle the differences in the protocol. If the protocol versions are backwards compatible, this should be easy to do for the client (simply not use those fields that are not part of the version of the protocol). If the protocol is not backwards compatible, we do not need any special provisions for it in the grammar any way, since we would need specific descriptions for each field of the protocol.