Extensibility of the Template ID

[adkapur]

We just had a quick question about the extensibility of the template ID. This is the particular section in the specification which outlines backward compatibility:

Compatibility is only ensured under these conditions:

• Fields may be added to either the root of a message or to a repeating group, but in each case, they must be appended to end of a block.
• Existing fields cannot change data type or move within a message.
• A repeating group may be added, but only after existing groups and if there are no subsequent variable data elements at the end of the message.
• A variable data element may be added, but only after existing groups and data.
• Message header encoding cannot change.

Changes that break those constraints require consumers to update to the current schema used by publishers. In general, metadata changes such as name or description corrections do not break compatibility so long as wire format does not change.

If an enumeration is being used in a particular template ID then do we need to assign a new template ID if additional values are being added to the enumeration? Does this qualify as changing data type of an existing field?

For example NewOrderSingle has template ID 99 with sideEnum containing values 1 & 2 and if we were to add an additional value to this enumeration for cross (8) then will a new template ID be required or it is possible to just update the schema version and utilize the sinceVersion attribute?

    <enum name="sideEnum" encodingType="enumEncoding">
        <validValue name="Buy" description="Buy">1</validValue>
        <validValue name="Sell" description="Sell">2</validValue>
    </enum>

<sbe:message name="NewOrderSingle" id="99" blockLength="54" semanticType="D">
    <field name="ClOrdID" id="11" type="idString" description="Customer Order ID"
           offset="0" semanticType="String"/>
    <field name="Account" id="1" type="idString" description="Account mnemonic"
           offset="8" semanticType="String"/>
    <field name="Symbol" id="55" type="idString" description="Security ID" 
          offset="16" semanticType="String"/>
    <field name="Side" id="54" type="sideEnum" description="Side" offset="24" 
          semanticType="char"/>
    <field name="TransactTime" id="60" type="timestampEncoding" 
          description="Order entry time" offset="25" semanticType="UTCTimestamp"/>
    <field name="OrderQty" id="38" type="qtyEncoding" description="Order quantity"
           offset="33" semanticType="Qty"/>
    <field name="OrdType" id="40" type="ordTypeEnum" description="Order type" 
          offset="37" semanticType="char"/>
    <field name="Price" id="44" type="optionalDecimalEncoding" 
          description="Limit price" offset="38" semanticType="Price"/>
    <field name="StopPx" id="99" type="optionalDecimalEncoding" 
          description="Stop price" offset="46" semanticType="Price"/>
</sbe:message>

[donmendelson]

Added enum value is not explicitly mentioned in section 5 of the spec. However, the metadata provides for versioning of enum values, just as it provides for versioning of fields. The schema entity <validValue> has sinceVersion attribute to tell in which schema version the enum value was added.

sinceVersion - Documents the version of a schema in which a value was added

There is no problem going from old producer to new consumer. However, if the producer is of a later version than the consumer, and it sends a newly added enum value, then the consumer will consider it an invalid value.

I would expect the decoder to pass the exception up to the application layer for appropriate handling. For example, if OrdType value J=Market If Touched is added to a schema, and the consumer does not recognize it, then the application should return an order rejection with reason order type not supported, even if it does not know what "J" represents. Note that this is not strictly a versioning problem, however. It should have the same exception handling even if "J" was never added to the enum but was sent in error.

[adkapur]

Hi Don,

A couple of questions regarding this particular statement --> "A repeating group may be added, but only after existing groups and if there are no subsequent variable data elements at the end of the message"

a) Does this rule apply to individual fields as well described in this statement? --> "Fields may be added to either the root of a message or to a repeating group, but in each case, they must be appended to end of a block". Couldn't individual fields be added to the root of a message regardless of whether there are variable data elements at the end of a message?

b) What happens if we add a repeating group after existing groups and there are variable data elements at the end of the message? Will this break compatibility and force all consumers to upgrade to the latest schema version?