Encoding `plog` Files

Lyle Johnson edited this page Jan 27, 2017 · 1 revision

Encoding plog Files

The first actual transcoding example will write the data back out to another PLog file. This is useful in combination with filters, because one truncate a larger PLog into a smaller one, write out a new PLog with only chosen types, or subsample a limited range of records, to create a smaller and more specific PLog file from a larger one.

Truncation

➜  polysync-transcode ibeo.25.plog --slice :2 plog -o ibeo.2.plog
➜  polysync-transcode ibeo.2.plog --type ps_log_record
ps_log_record x2

The first command created a new file ibeo.2.plog that contains a duplicate of the header ps_log_header from ibeo.25.plog, plus its first two records. This is because the filter argument '--slice :2' selects the records from the beginning of the file and stops after the second one. This can be checked with a binary comparison:

➜  wc -c ibeo.2.plog
28216 ibeo.2.plog
➜  cmp ibeo.2.plog ibeo.25.plog --bytes 28216

The first command just computes how many bytes to compare (which is the entire file ibeo.2.plog) and the second one is a standard command line tool to check binary equality.

Model Diagnostics

Polysync-transcode provides valuable introspection tools to create and diagnose sensor data models.

➜  polysync-transcode ibeo.25.plog --slice 2 dump
log_record {
    index: 2
    size: 5510
    prev_size: 26840
    timestamp: 1464470621735352
    msg_header {
        ...
    } 
    ps_byte_array_msg {
        ...
    } 
    ibeo.header {
        magic: 0xaffec0c2
        ...
        data_type: 0x2280
        time: 0xdaf49729f8d76ab6
    } 
    raw: [ 70 6c 91 2 0 0 0 0 f8 ab 90 2 ... 0 0 88 57 2c 5c ff 7f 0 0 2 0 ] (5450 elements)
} 

Arg, what is up with that last field raw: ...? The first two records in the file have no raw fields, just fully decoded records of type ibeo.vehicle_state and ibeo.scan_data, respectively. (Hint: the transcoder will read raw bytes like this whenever it cannot determine a better model.) Diagnose the problem:

➜  polysync-transcode ibeo.25.plog --slice 2 dump --loglevel debug1
...
transcode[debug1]: loading descriptions from "../share/ibeo.toml"
transcode[debug1]: loading descriptions from "../share/sensor.toml"
transcode[debug1]: loading descriptions from "../share/core.toml"
...
transcode[verbose]: index: 2 size: 5510 prev_size: 26840 timestamp: 1464470621735352 
detector[debug1]: ps_byte_array_msg matched from parent "msg_header"
detector[debug1]: ibeo.header matched from parent "ps_byte_array_msg"
detector[debug1]: type not detected, returning raw sequence
log_record {
    ...

Turning on the debug1 loglevel shows us that the type descriptions are loaded from TOML files. While parsing record #2, we see the types ps_byte_array_msg, and then ibeo.header were "matched", then the "type was not detected" so we got the raw sequence instead of a decoded type.

What if this result is surprising, and you think that ibeo type 0x2280 should be fully decoded because the description actually exists? Check more detail:

➜  polysync-transcode ibeo.25.plog --slice 2 dump --loglevel debug2
...
detector[debug2]: ibeo.vehicle_state not matched: parent "msg_header" != "ibeo.header"
detector[debug2]: ibeo.scan_data not matched: parent "msg_header" != "ibeo.header"
detector[debug2]: ibeo.header not matched: parent "msg_header" != "ps_byte_array_msg"
detector[debug2]: ps_image_data_msg: mismatched { type: 16 != 9 }
detector[debug2]: ps_can_frame_msg: mismatched { type: 16 != 9 }
detector[debug1]: ps_byte_array_msg matched from parent "msg_header"
decoder[debug2]: decoding "ps_byte_array_msg" at offset 28256
...
detector[debug2]: ibeo.vehicle_state not matched: parent "ps_byte_array_msg" != "ibeo.header"
detector[debug2]: ibeo.scan_data not matched: parent "ps_byte_array_msg" != "ibeo.header"
detector[debug2]: ps_image_data_msg not matched: parent "ps_byte_array_msg" != "msg_header"
detector[debug2]: ps_byte_array_msg not matched: parent "ps_byte_array_msg" != "msg_header"
detector[debug2]: ps_can_frame_msg not matched: parent "ps_byte_array_msg" != "msg_header"
detector[debug1]: ibeo.header matched from parent "ps_byte_array_msg"
decoder[debug2]: decoding "ibeo.header" at offset 28272
...
detector[debug2]: ibeo.vehicle_state: mismatched { data_type: 0x2280 missing from description }
detector[debug2]: ibeo.scan_data: mismatched { data_type: 0x2280 missing from description }
detector[debug2]: ibeo.header not matched: parent "ibeo.header" != "ps_byte_array_msg"
detector[debug2]: ps_image_data_msg not matched: parent "ibeo.header" != "msg_header"
detector[debug2]: ps_byte_array_msg not matched: parent "ibeo.header" != "msg_header"
detector[debug2]: ps_can_frame_msg not matched: parent "ibeo.header" != "msg_header"
detector[debug1]: type not detected, returning raw sequence

Here, we see the transcoder searching for the type, including mismatched types. For more insight, the detectors are defined, partly in share/ibeo.toml. In there, the ibeo.header type contains two detectors: one for ibeo.vehicle_state and another for ibeo.scan_data. In the above debug output, the messages including not matched: parent are saying the message cannot match because the previous message, the "parent", is the wrong type. The other reason that a detector fails is because the field values do not match the (otherwise correct) parent. For instance, an ibeo.vehicle_state message always follows an ibeo.header, and for the match to succeed, two fields (magic and data_type), have to contain the exact Ibeo magic number 0xAFFEC0C2 and the exact data_type value 0x2807.

So now we see that the reason that record #2 fails to decode is because it has a field data_type with value 0x2280, which has no detector defined for it. To fix this, you would add a new detector to the ibeo.header section in the file ibeo.toml.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.