Skip to content

Notes to self

Jump to bottom
Bouke Grevelt edited this page Jul 6, 2018 · 6 revisions

  • Our proposed grammar is flexible enough to describe any binary format. However, the parser generator would have to be very complex to fully support the grammer (probably similar to 'real' parser generators like ANTLR in complexity. It may be worthwhile to introduce some restrictions (in lexical analysis) to reduce the complexity.

    • Fixed fields must be the same for all (root level) objects in type and size and only one fixed value may be different for different objects. e.g.

      Object1
{
    header : uint32 = 0x01020304;
    message_type : uint8 = 1;
    field1 : int16;
    field2 : float32;
    (...)
    footer : uint32 = 0x04030201;    
}

Object2
{
    field1 : int16;
    message_type : uint8 = 1;        
    field2 : float32;
    (...)  
}

      is not allowed.

      This would make generating a parser a lot easier since we don't need complex iteration of tokens and backtracking. Instead we can simply find the start of a packet and determine the type based on the differing fixed value.

  • The current definition of the grammar can support versioning in protocols by using fixed fields:

some_object_v1
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 = 1;
    uint32[6];
}

some_object_v2
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 = 2;
    field_exclusive_to_v2 : uint16;
    uint16;
    uint32[5];
}

However, this method will not work on versions of the protocol that are not defined. Even if these future versions are backwards compatible (as v2 in this example is backwards compatible with v1). There are a number of ways to solve this. We could for instance introduce introduce relational operators:

some_object_v1
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 = 1;
    uint32[6];
}

some_object_v2
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 >= 2;
    field_exclusive_to_v2 : uint16;
    uint16;
    uint32[5];
}

Another option would be to add some type of qualifier to the field

some_object_v1
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 = 1;
    uint32[6];
}

some_object_v2
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    version_field : uint8 = 2 <version>;
    field_exclusive_to_v2 : uint16;
    uint16;
    uint32[5];
}

The latter would only be worthwhile if there are other reasons to add qualifier fields.

The third option would be to not add version support to the parser, but instead rely on the client of the parser to handle the differences in the protocol. If the protocol versions are backwards compatible, this should be easy to do for the client (simply not use those fields that are not part of the version of the protocol). If the protocol is not backwards compatible, we do not need any special provisions for it in the grammar any way, since we would need specific descriptions for each field of the protocol.

  • The grammar does not include a method to indicate the endianness of the data that is being parsed

  • We may want some method of including information about fields as is often included in protocol specifications. This information could then be used in the generated parser code as comments for the field getters (or something similar).

There are a number of ways we can do this. Simply adding a string to a field:

car
{
    wheels : uint8 "The number of weeks on the car. Normally four.";
    top_speed : float32 "The top speed of the vehicle in kilometers per hour";
    fuel_consumption : float32 "The fuel consumption of the vehicle in kilometers per liter";
}

Or adding the string as a field property

car
{
    wheels : uint8 <description "The number of weeks on the car. Normally four.">;
    top_speed : float32 <description "The top speed of the vehicle in kilometers per hour">;
    fuel_consumption : float32 <description "The fuel consumption of the vehicle in kilometers per liter">;
}

But there is probably a better way to do this...

  • Do we want the parser to be able to handle 'mid stream' data? I don't think parser generators like ANTLR are able to do this, but for these binary protocols this is probably a feature worth having. This would require the data format to include some fixed value fields to determine the start (or at least some arbitrary position) of the message. If we want this, we also have to think about how this affects protocols that do not have fields like this (as is sometimes that case for protocols that rely on the lower level protocols (such as TCP) for 'packetization'.

  • We may want to include some validation rules to the grammar. For some fields, we may define a value (e.g. field : = ;) that is not required, but expected. In other words, if it does not match, that should not stop parsing, rather issue a warning in the log or something similar. Another thing that we could include in the parsing is the validation of checksums. This would require the user to be able to write the validation code as part of the protocol specification. This is something to really think about. One could argue that the method used for checksum validation is part of the protocol and should therefore be part of the protocol description. On the other hand, adding a definition for checksum validation (which could basically be anything the write of the protocol can image) to the grammar can greatly increase the complexity of the grammar. A way to include this feature without increasing complexity of the grammer would be to allow the user to write code that is directly inserted into the generated parser code:

some_object_v2
{
    field_one : uint32 = 0x01020304;
    field_two : uint32;
    size_field : uint16;
    (...)
    checksum : uint32 = { std::uint32_t checksum = 0; for ( int i = 0 ; i < size_field ; ++ i) { checksum += data[i]; } return checksum; }
}

The code written by the user would have to be the body of a method returning the expected value. The method would need to have access to the raw message data. In this example, we used 'data' as a keyword for accessing that data. The downside of this approach is that the protocol description is now tied to the language chosen for the code generation, which is something we typically do not want to do.

  • How will we handle fields in an encapsulating object that size a collection? For instance:
SomeObject
{
    size_field : uint16;
    (...)
    embedded : 
    {
        (...)
        collection : uint16[size_field];
    }
}

This is pretty clear. However, if we don't want to inline we end up with something less obvious:

Embedded
{
    (...)
    collection : uint16[size_field];
}

SomeObject
{
    size_field : uint16;
    (...)
    embedded : Embedded;
}

Now 'Embedded' reference a field name that doesn't exist in it's scope. It's not a real problem, because at SA time we will know whether or not there is a field in the complete data type with this name, but it does make the code less readable (and therefore potentially more error-prone)

  • How do we handle equal named field in nested structures?
SomeObject
{
    collection_size : uint16;
    collection : {
        a : uint32;
        b : float32;
        collection_size : uint16;
        collection : float64[collection_size];
    }
}

Both collection and collection_size are defined in both the encapsulating and the embedded data type. I don't see any potential problems in the reuse of 'collection'. 'collection_size' however, is refered to in the code (as the size field of the inner collection) which introduces ambiguity. We could choose not to allow this, or to use the C-type language approach of shadowing (and warning about the shadowing).

  • Naming of inline objects When we define an object inline
A : {
   ...;
   ...;
   b : {...; ...; ...;}
}

the inline object type is not named. The field in A is named 'b', but there is no name for the object itself. When we generate code for the object, it needs a name. My original thought was to deduce the type name from the field name. In this case, the name of the object would be 'B'. However, there are a number of problems with this:

Tree : {
   stem : { circomference : f64; color : u8 { light_brown, brown, dark_brown, weird_brown, almost_black}; };
   number_of_branches : u16;
   branches : {
      number_of_leaves : u32;
      leaves : {
        color : { green = 0; brown = 1 ; red = 3; yellow = 4; }
      }[number_of_leaves]
   }[number_of_branches]
}

In this example we would get an object names 'Leaves' while 'Leave' would have been a far more logical choice. Also we would get two enumerations with the same name ('Color') in a different scope, which gives us another shadowing issue.

* Should we enforce naming conventions (for instance Upper case for object and enum names and lower case for field names?

Clone this wiki locally