Simple Binary Encoding (SBE)

Step 1: SbeTool documentation & helpful information

• SBE version: v1-0-STANDARD.
• real-logic simple-binary-encoding github:.
• real-logic SBE tool: The SBE tool is a command line utility that can be used to generate codecs and validate message declaration schemas. The tool is written in Java and can run as an executable JAR file.

Step 2: Running the SbeTool

• Find and download the appropriate XML schema for use in running the SbeTool. Link to SBE message schema for communication between the LMM and the Matching Engine: CTS-Base-sbe.xml.
• (SBE uses an XML schema to specify messages, headers, and other elements. The system is typed and users can create new types from primitives. An example schema can be found here).
• Command:
  $ java -jar sbe-all-1.20.3.jar MarketSbePayloads.xml

Step 3: SbeTool Output

• Running the SbeTool will create the appropriate Java source files. These files represent the types and messages declared in the schema.
• You may also find the generated files in here.

Step 4: Baseline Package Error Handling

• Warning: You may encounter errors related to the package declaration. One of the fist lines in each of the generated files is the baseline package declaration: package baseline;
• Commenting out this line in each generated file should fix the package related errors.

Step 5: Validating the SbeTool

• To validate the SbeTool, write the directBuffer.byteBuffer() to a text file.
• In Windows, use the command Format-Hex on the text file to view the encoded message in hex format.
• In Linux, the command hexdump can be used.

Other SBE links:

• real-logic simple-binary-encoding examples:.
• Simple-binary-encoding:.
• 5 Simple Binary Encoding gotchas:.

Difficuly with SBE and EML-CTS

• Variable String Encoding An issue was identified and resolved involving how varStringEncoding is used within SBE schemas, particularly in relation to the EiResponseType composite and its responseDescription field.

Background:

String encoding was previously broken, resulting in corrupted bytes during serialization. To work around this, manual byte offset placement was required, as the SBE-Tool did not generate a method to handle string encoding for varStringEncoding fields.

Root Cause:

The SBE schema itself was the source of the problem. Specifically, the line:

<ref name="responseDescription" type="varStringEncoding"/>

• Incorrect Composite Field Declaration:

<composite name="EiResponseType" description="See EiResponseType.java">
    <ref name="createdDateTime" type="Instant" />
    <ref name="inResponseTo" type="RefIdType" />
    <type name="responseCode" primitiveType="uint64" />
    <ref name="responseDescription" type="varStringEncoding"/>
    <ref name="responseDetail" type="ResponseDetailType" />
</composite>

Although this compiles without error, it does not result in a usable method for encoding strings. The SBE-Tool does not interpret as defining a varStringEncoding; instead, it expects the field to be defined with a tag. The correct line should be: <data name="responseDescription" type="varStringEncoding" id="x"/>

Constraint: However, fields cannot be placed inside a such as EiResponseType. They must instead be defined within a like EiCreatedTenderPayload. As a result, the schema must be updated to move the responseDescription field out of the composite and directly under the parent message. This change affects only the field's location in the byte buffer—it has no functional consequence. The field is still correctly mapped back to the payload object.

• Correct schema placement:

<sbe:message name="EiCreatedTenderPayload" id="6" description="See EiCreatedTenderPayload.java">
    <field name="counterPartyId" id="1" type="ActorIdType"/>
    <field name="inResponseTo" id="2" type="RefIdType"/>
    <field name="marketOrderId" id="3" type="MarketOrderIdType"/>
    <field name="partyId" id="4" type="ActorIdType"/>
    <field name="response" id="5" type="EiResponseType"/>
    <field name="tenderId" id="6" type="TenderIdType"/>
    <data  name="responseDescription" id="7" type="varStringEncoding"/>
</sbe:message>

Open Questions and Considerations:

Should the to responseDescription remain in the composite? It serves no purpose under the new design but does not interfere if left in place.

Any that uses a containing a varStringEncoding field must define that field independently under the . For example, if a payload includes response, it must also explicitly define responseDescription. Is this approach sustainable or considered best practice?

There are schema limitations when using : all fields must appear at the end of the message definition. This is required because variable-length data should be located last in the byte buffer for optimal alignment. An example can be seen in EiManagedTickerSubscriptionPayload, where the multicastListenReference field is defined as:

<data name="multicastListenReference" type="varStringEncoding" id="2"/>

<sbe:message name="EiManagedTickerSubscriptionPayload" id="10" description="See EiManagedTickerSubscriptionPayload.java">
    <field name="tickerType" id="1" type="TickerType"/>
    <field name="response" id="3" type="EiResponseType" />
    <field name="subscriptionActionTaken" id="4" type="SubscriptionActionType"/>
    <field name="subscriptionRequestId" id="5" type="RefIdType"/>
    <data  name="multicastListenReference" id="2" type="varStringEncoding"/>
    <data  name="responseDescription" id="6" type="varStringEncoding"/>
</sbe:message>

Although functionally valid, placing this field last in the schema causes its id to appear out of numeric order with earlier fields. This raises the question of whether to keep the current ID or shift IDs to maintain visual consistency.