BiPaGe2

BiPaGe is a Binary Parser Generator. The program will generated code to parse binary structures based on definitions of those structure. It has the following features

• Data in big and little endian.
• Non-standard width integers

The following back-ends (code generators) are available

• C++

Using BiPaGe

BiPaGe takes in input file that contains one or more structure definitions and produces parser and generator code for those datatypes in the chosen language. In BiPaGe a data structure is defined in the follwing way

<struture name>
{
   <field name 1> : <type>;
   <field name 2> : <type>;
   etc.
}

For example:

DataFrame
{
   Preamble : uint64;
   DataField1: int32;
   DataField2: float64;
   CRC: uint32;
}

To generate the parser and builder code, run bipage, feeding it the input file python bipage.py -i <input file> -o <output director>

BiPaGe supports the following optional command line arguments

• --cpp-no-validate-builder-input When supplied, no input validation code will be generated by the c++ backend. See the 'Non-standard sized integers' section for more details.
• --cpp-no-to-string When supplied, no to_string method will be generated by the c++ backend in the parser class.

Comments

BiPaGe supports C/C++ style comments

MyDataType
{
    field1 : float64; // Single line comment
    /* Multi
    Line
    Comment
    */
    field2 : int16;
}

Basic types

The following basic types are supported

• Signed integer: int8, int16, int32, int64 as well as the short hand aliases s8, s16, s32, and s64.
• Unsigned integer: uint8, uint16, uint32, uint64 as well as the short hand aliases u8, u16, u32, and u64.
• Floating point: float32, float64, f32, and f64

Non-standard sized integers

BiPaGe also support non-standard sized signed and unsigned integers. These type of fields are often employed to save space, for example in streaming data protocols. Non standard fields have to be declared inside what we will refer to as a 'capture scope'. The capture scope is the underlying type that encompasses the non-standard sized integer types.

DataFrame
{
  Preamble : uint64;
  {
    SmallSigned : int6;
    SmallUnsigned: uint6;
    Nibble: uint4;
  }
  CRC: uint32;
}

The capture scope is necessary to support different endiannesses; we have to know how the encoding side treated this data to be able to transform it to the platform endian. All fields in a capture scope should add up to a standard size. The fields in the capture scope in the example above add up to 16 bits (6+6+4).

Validation

Most languages only support standard width integers. That means that the generated setter code for non-standard field in the builder class will have a wider type argument than the actual data value. This is certainly true for C++ (which is currently our only the only supported backend) where the setter for SmallSigned in the example above will look something like void SmallSigned(std::int8_t val). That means that the caller of that setter can supply a value that does not fit in the underlying data type. Unfortunately there is no way to have this validated at compile time. Because of that, run time validation code is generated in these setters by default. This runtime validation code generation can be disabled by supplying the --cpp-no-validate-builder-input command line argument.

Bit flags

Single bit fields are modeled as flag fields. Like non-standard width integers they need to be inside a capture scope and the size of all fields in the capture scope need to add up to a standard integer size.

OptionalDataContainer
{
    {
        a_valid: flag;
        b_valid: flag;
        c_valid: flag;
        d_valid: flag;
        e_valid: flag;
        f_valid: flag;
        g_valid: flag;
        h_valid: flag;
    }
    a : u32;
    b : s16;
    c : f64;
    d : u8;
    e : u8;
    f : s64;
    g : f32;
    h : u16;
}

Enumerations

Enumeration field types can be used by defining an enumeration and referencing it as the field type. An enumeration consists of

• A name
• The (integer) base type of the enumeration
• A set or enumerators that consist of a name and a value

<enumeration name> : <base type>
{
   <enumerator name> = <enumerator value>,
   <enumerator name> = <enumerator value>,
   ...
   <enumerator name> = <enumerator value>
}

For example

Fruit : uint16
{
    apple = 0,
    pear = 1,
    banana = 8,
    strawberry = 36 
}

FruitBasket
{
    typeOfFruit1 : Fruit;
    fruitCount1 : u32;
    typeOfFruit2 : Fruit;
    fruitCount2 : u32;
    typeOfFruit3 : Fruit;
    fruitCount3 : u32;
}

Any integer type can be used for the base type. That includes non-standard width integers, so we can do things like this

Fruit : int2
{
    apple = -2,
    pear = -1,
    banana = 0,
    strawberry = 1 
}
FruitBasket
{
    {
        typeOfFruit1 : Fruit;
        fruitCount1 : u6;
    }
    {
        typeOfFruit2 : Fruit;
        fruitCount2 : u6;
    }
    {
        typeOfFruit3 : Fruit;
        fruitCount3 : u6;
    }
}

Inline enumerations

As a shorthand, enumerations can be defined inline:

BagOfFruit
{
    typeOfFruit : u8 
    {
        apple = 0,
        pear = 1,
        banana = 2,
        strawberry = 3 
    };
    quantity : u16;
}

Paddding fields

Sometimes data structures contain padding fields; fields that take up size in the structure but are not used to store any data. Some formats refer to these as 'reserved' fields. These fields can be modeled in BiPaGe by omitting the field name.

MyDataStructure
{
    startOfFrame : u16;
    u16; // <-- 2 bytes of padding
    size : u32;
}

Collections

Collections are supported for floating point types, standard size integers and (non-inline) enumerations with a standard size integer as a base type. The size of the collection can be

fixed:

MyDataStructureWithACollection
{
    foo : u8;
    bar : f64;
    collection : s32[15];
}

MyEnum : u16
{
    lorem = 0,
    ipsum = 10
}

MyDataStructureWithACollection
{
    foo : u8;
    bar : f64;
    collection : MyEnum[15];
}

Based on another field

MyDataStructureWithACollection
{
    foo : u8;
    bar : f64;
    collection_size: u16;
    collection : s32[collection_size];
}

Based on an expression

MyDataStructureWithACollection
{
    packet_size : u32;
    foo: s16;
    bar: f64;
    padding : u8[packet_size - (16/8) - (64/8)];
}

The following expressions are supported:

• Brackets field: u8[(field2 + field3) * 2]
• Addition field: u8[some_other_field + 4]
• Subtraction field: u8[some_other_field - 3]
• Multiplication padding: u8[number_of_ints * 4]
• Division samples: f64[(total_size - header) / 8]
• Ternary optional_collection: s32[some_flag? 100 : 0]
• (In)Equality data1: s32[data_enum_field == payload.data1? 100 : 0] (and similarly !=)
• Relational field: f32[some_other_field > 20 ? 100 : 10] (and similarly >=, <, <=)

Endianness

⚠️ Code generated by BiPaGe should only be run on little endian platforms!: This section is about handling data that has been encoded on a big endian platform and encoding data to be sent to a big endian platform.

You can define the endianness of the data structures in a file. If the endianness of the data is not defined in the input file, little endian is assumed. The endianness can be defined once at the top of the file. This endianness applies to all data structures defined in the file

@bigendian;

DataStructure
{
...
}

You can also explicitly define little endian, but it, as this is also the default, it has no real effect.

@litleendian;

DataStructure
{
...
}

If big endian is defined, byte swapping will occur both in the parser and in the builer. E.g the parser will assume that incoming data is in big endian format and the builder will ensure that the serialized data is in big endian format.

Namespace

You can add a namespace to the input file by adding namespace <namespace> at the start of the file (but after the endianness specifier if you include that as well). Nested namespaces are supported by using a dot in between the different namespaces.

@litleendian;
namespace Some.Awesome.Namespace;
DataStructure
{
...
}

Adding a namespace to the input file will instruct the compiler to put all generated code in that namespace (or whatever comes closest to a namespace in the target language).

Imports

Types (currently only enumerations but data structures will be supported in the future) can be imported from other bipage imput files. This can be useful if you want to use the same data type in multiple bipage files.

file1.bp

MyEnum : u8
{
    lorem = 0,
    ipsum = 10
}

file2.bp

import "file1.bp";
Foo
{
    field1 : MyEnum;
}

If the imported type is in a namespace, the fully qualified name should be used

file1.bp

namespace file2.name.space;
MyEnum
{
    lorem = 0,
    ipsum = 10
}

file2.bp

import "file1.bp";
Foo
{
    field1 : file2.name.space.MyEnum;
}