C++ Segmentation fault printing XML sub group (parent message prints fine)

[cpp77]

I'm processing an MDIncrementalRefreshBook46 message. It contains two XML sub-groups: NoMDEntries followed-by NoOrderIDEntries.

I create the MDIncrementalRefreshBook46 object and print it. I then iterate through the second XML sub-group and read the referenceID and OrderUpdateAction:

mktdata::MDIncrementalRefreshBook46 msg;
msg.wrapForDecode(bytes, offset, blockLength, mktdata::MessageHeader::sbeSchemaVersion(), bufferSize);
std::cout << msg << std::endl;

mktdata::MDIncrementalRefreshBook46::NoOrderIDEntries& mboEntries = msg.noOrderIDEntries();
while(mboEntries.hasNext())
{
    mktdata::MDIncrementalRefreshBook46::NoOrderIDEntries& mboEntry = mboEntries.next();
    const uint8_t referenceID = mboEntry.referenceID();
    std::cout << "referenceID: " << referenceID << std::endl;
    const mktdata::OrderUpdateAction::Value updateAction = mboEntry.orderUpdateAction();
    std::cout << "updateAction: " << updateAction << std::endl;
}

The parent message prints fine:

{"Name": "MDIncrementalRefreshBook46", "sbeTemplateId": 46, "TransactTime": 1592154621498375773, "MatchEventIndicator": ["lastQuoteMsg"], "NoMDEntries": [{"MDEntryPx": {"mantissa": 302375000000000}, "MDEntrySize": 10, "SecurityID": 19074, "RptSeq": 1, "NumberOfOrders": 1, "MDPriceLevel": 1, "MDUpdateAction": "New", "MDEntryType": "Bid"}], "NoOrderIDEntries": [{"OrderID": 646981700454, "MDOrderPriority": 8786193103, "MDDisplayQty": 10, "ReferenceID": 1, "OrderUpdateAction": "New"}]}

but if I try to retrieve the referenceID I get a null/0 (should be 1) and when reading the OrderUpdateAction I get an invalid value and an exception throw in OrderUpdateAction::get().

This could be misuse on my part, but I can't see what I'm doing wrong?

[mjpt777]

Printing is designed to be done on the whole messages only.

[cpp77]

Thanks for your reply. Unfortunately logging was only one side effect. If I remove the print and try to retrieve the OrderUpdateAction, or the ReferenceID, I still get corrupt bytes.

(I will edit my original post to highlight this)

[mjpt777]

@cpp77 If you can provide a full example then we can look at it.

[cpp77]

Code attached. Instructions:

1. Unzip single cpp file
2. Edit top 2 lines to point to your header locations
3. Compile. I used clang++ -Wall -std=c++17 sbe.cpp -o sbe
4. Run. I used gdb --args ./sbe "/path/to/market/data/file"

I reproduce with the MDP3 file from Sunday 14th June 2020:
20200614-PCAP_310_0_*_0-20200614

You will see the application terminate with:

Received template other than Refresh book
Seq num: 836
Received template other than Refresh book
Seq num: 837
Received template other than Refresh book
Seq num: 838
Received RefreshBook46 template
msg: {"Name": "MDIncrementalRefreshBook46", "sbeTemplateId": 46, "TransactTime": 1592154621498375773, "MatchEventIndicator": ["lastQuoteMsg"], "NoMDEntries": [{"MDEntryPx": {"mantissa": 302375000000000}, "MDEntrySize": 10, "SecurityID": 19074, "RptSeq": 1, "NumberOfOrders": 1, "MDPriceLevel": 1, "MDUpdateAction": "New", "MDEntryType": "Bid"}], "NoOrderIDEntries": [{"OrderID": 646981700454, "MDOrderPriority": 8786193103, "MDDisplayQty": 10, "ReferenceID": 1, "OrderUpdateAction": "New"}]}
terminate called after throwing an instance of 'std::runtime_error'
  what():  unknown value for enum OrderUpdateAction [E103]

Thank you in advance.

[mjpt777]

Can you boil this down to a smaller example? This code goes beyond the use of SBE.

Best to submit a test case as a patch.

[cpp77]

Hi, unfortunately I don't think I can make a smaller example because the problem only occurs on that particular template. So I need to read the file, read each packet, skip over the other templates until I encounter the MDIncrementalRefreshBook46.

I could create a file containing only the bytes of the problematic message (but then I could introduce an artifact).

Let me know what you need. I'm happy to provide (if i can).

[mjpt777]

If you cannot isolate to a simple test then I would have to spend time understanding your code and data which could take up quite a bit of my day. I could do this if you were on a support contract as a customer, otherwise I've no idea how well you know how to use SBE.

[mjpt777]

On a quick scan I can see you are not using the version support correctly.

[cpp77]

Hi, that is possible but given the attributes are all correct (below) surely the problem lies with the next part of the code?

{"Name": "MDIncrementalRefreshBook46", "sbeTemplateId": 46, "TransactTime": 1592154621498375773, "MatchEventIndicator": ["lastQuoteMsg"], "NoMDEntries": [{"MDEntryPx": {"mantissa": 302375000000000}, "MDEntrySize": 10, "SecurityID": 19074, "RptSeq": 1, "NumberOfOrders": 1, "MDPriceLevel": 1, "MDUpdateAction": "New", "MDEntryType": "Bid"}], "NoOrderIDEntries": [{"OrderID": 646981700454, "MDOrderPriority": 8786193103, "MDDisplayQty": 10, "ReferenceID": 1, "OrderUpdateAction": "New"}]}

If I had abused SBE, I would expect the bytes to be corrupt and the above values be wrong?

So surely the problem must be limited to the below code:

MDIncrementalRefreshBook46::NoOrderIDEntries& mboEntries = msg.noOrderIDEntries();

while(mboEntries.hasNext())
{
    mktdata::MDIncrementalRefreshBook46::NoOrderIDEntries& mboEntry = mboEntries.next();
    const uint8_t referenceID = mboEntry.referenceID();
    std::cout << "referenceID: " << referenceID << std::endl;
    const mktdata::OrderUpdateAction::Value updateAction = mboEntry.orderUpdateAction();
    std::cout << "updateAction: " << updateAction << std::endl;
}

[mjpt777]

Can you at least post your SBE schema?

[cpp77]

Sure, I have attached Schema. It should be the production one from the CME ftp
SchemaForSBE.txt

[mjpt777]

This is too large a scope. If it cannot be narrowed down to a repeatable example or test then we cannot further investigate without a support contract.

[cpp77]

This potential issue regards decoding one particular application message. Therefore to reach the message type one must iterate through the file, decoding each PCAP header, packet header and SBE header. This is what the code I attached 2 weeks ago did.

I can't see how this can be narrowed down but I'm genuinely open to suggestions?

[mjpt777]

Does it work if you have that single message encoded in a buffer on its own?

[mjpt777]

Closing due to no follow up or isolated testcase.

[cpp77]

I eventually discovered the problem whilst processing another message and getting the same issue. It never occurred that getters would need to be called in a particular order (to match the XML schema). Some getter methods change the state of the buffer, so if the getters aren't called in the correct order later getters read corrupted data.

This requirement puts a lot of emphasis on the client. We're processing 10-12 different messages, each with between 10 and 50 fields, meaning we have to check a lot of methods are called in the correct order.

I think it would be better if the getters were const and simply return strong types by casting the bytes without modifying any state. This way the ordering wouldn't matter. This should be possible as I wrote my own MDIncrementalRefreshBook46 class when I didn't know the cause of this issue.

@mjpt777 Thank you for your prompt responses throughout.

[mjpt777]

For the getters of strings and groups to be const SBE would be necessary to build up an index for the variable length items. This would have a significant cost. If you choose to use something so fast then you need to accept how it works.

[cpp77]

"For the getters of strings and groups to be const SBE would be necessary to build up an index for the variable length items"

RE strings, from the CME XML schema:

<field name="Symbol" id="55" type="Symbol" description="Instrument Name or Symbol " offset="35" semanticType="String"/>
<field name="SecurityID" id="48" type="Int32" description="Unique instrument ID" offset="55" semanticType="int"/>

Surely the length of a string is just the difference between offsets?

RE groups:

• The XML states where the group begins
• The XML states how large each group element is
• The message says how many elements in the group

All the offsets and sizes are known. I'm unsure why an index needed?

[mjpt777]

Fields are fixed length but the repeating groups and var data fields are not.

Surely the length of a string is just the difference between offsets?

This is like saying "just". :-)

Take the time to understand the implementation and investigate what happens with the position when progressing over repeating group, which can be nested, and var data.