Add header metadata to trace files

[tgiuli]

Tracefiles should have metadata that includes the version number of the tracefile being generated, and any other metadata from the vehicle we're currently connected to, such as model, vehicle interface firmware version, etc.

Latest proposed format:

{"metadata": {
    "version": "v3.0",
    "vehicle_interface_id": "7ABF",
    "vehicle": {
        "make": "Ford",
        "model": "Mustang",
        "trim": "V6 Premium",
        "year": 2013
    },
    "description": "highway drive to work",
    "driver_name": "TJ Giuli",
    "vehicle_id": "17N1039247929"
}

{"name":"accelerator_pedal_position","value":0,"timestamp":1364314054.403000}
{"name":"engine_speed","value":0,"timestamp":1364314054.415000}
{"name":"torque_at_transmission…….

• All fields are optional, just leave out if not used
• Entire header JSON object is optional - if you don't plan to use any of the fields, don't include the header object at all. Trace file parsers should check if the first record has the metadata field.

[peplin]

I'm thinking it's most appropriate to make the trace files complete JSON objects, instead of this newline separate JSON object thing. One advantage of the current trace file format is that it's really tolerant to being interrupted - there's no process to closing the trace out except to stop writing to the file. That makes it really easy to just openxc-dump > trace.json but makes metatdata hardware. I think it'd be simple enough to add more formal logging support to the tools (with a way to detect and fix improperly closed traces when playing back).

[tgiuli]

Hmm, as we start to deal with bigger traces, we may want to leave the log files in a format that is easy to parse as a stream, rather than requiring a JSON parser to read through the file in its entirety before allowing a program to manipulate the data. Many traces may be too large to fit in memory, and in other cases we may want to break up the trace to parallelize computations (e.g., map reduce).

I definitely agree that we should also add more formal logging support.

[peplin]

There are JSON parsers that can do stream processing to handle things like the Twitter firehose, I'd consider using one of those.

[peplin]

@mgm1338 recommended the staxon parser.

[peplin]

A proposal from @zacnelson :

Add this as the first line in a trace file - it's just another small JSON object so any legacy parsers would just skip it as invalid:

{
                "trace header": {
                                "name": Zac Nelson,
                                "timestamp": XXXXX,
                                "kit_number": YY,
                                "vehicle-model": Mustang,
                                "vehicle-year": 2013,
                                "description": "highway drive to work"
                }
}

[peplin]

After some discussion:

• timestamp isn't necessary, the first record in the file has the timestamp

[peplin]

I propose moving the vehicle information to an object, e.g.:

"vehicle": {
    "make": "Ford",
    "model": "Mustang",
    "year": 2013
}

[peplin]

I propose dropping the name field - doesn't really add any information but does leak personal info.

[peplin]

I propose vehicle_interface_id instead of kit_number since the VIs we build separately won't be associated with any "kit".

Lastly, for our purposes the vehicle interface ID should be the MAC address of the VI's RN-42 because we know that will be unique.

With my last couple of proposals, the top of a trace would look like this:

{"metadata": {
    "vehicle_interface_id": "7ABF",
    "vehicle": {
        "make": "Ford",
        "model": "Mustang",
        "year": 2013
    },
    "description": "highway drive to work"
}
{"name":"accelerator_pedal_position","value":0,"timestamp":1364314054.403000}
{"name":"engine_speed","value":0,"timestamp":1364314054.415000}
{"name":"torque_at_transmission…….

[tgiuli]

@zacnelson @peplin , all good points. @peplin , I take your point about privacy issues, however, I think it would be a good idea to leave the name field in and then people within the community can make the personal decision about whether they would like to use that field or not. If someone uses OpenXC to conduct a large experiment across a fleet of vehicles with multiple drivers per vehicle (e.g., a cab company), having some way to uniquely identify the vehicle, the driver, and the vehicle interface would be good.

With that in mind, I propose we add a vehicle_id field. Folks could populate this with a VIN, a unique ID specific to their organization, or with nothing at all. Within our group at Ford, I propose that we agree to leave name blank or use a pseudonym and that we don't populate the vehicle_id field.

[zacnelson]

@tgiuli and @peplin I agree about the privacy issues with the "name" field. I do like TJ's suggestion of leaving the "name" field for others. Additionally, we can figure out who recorded what tracefiles by the MAC address used. As of right now each person has a specific MAC address.

[peplin]

Agree with all points, making some fields optional. I'm going to edit the top of this ticket to have the proposed format.

[zacnelson]

Where could we distinguish between different vehicle models. Like "Mustang GT" vs. "Mustang (v6)"? Could we create an additional field under "vehicle" that is something like "trim?" I think this field would be particularly necessary in order to know what different modules might be on CAN. I'm thinking mostly about the powertrain modules or infotainment signals that vary between "trims."

"vehicle": {
        "make": "Ford",
        "model": "Mustang",
        "year": 2013,
        "trim:" GT
    },

[peplin]

We agreed on adding a trim level for future machine processing, and also a version field that should be the version of the cantranslator firmware.

Final format:

{"metadata": {
    "version": "v3.0",
    "vehicle_interface_id": "7ABF",
    "vehicle": {
        "make": "Ford",
        "model": "Mustang",
        "trim": "V6 Premium",
        "year": 2013
    },
    "description": "highway drive to work",
    "driver_name": "TJ Giuli",
    "vehicle_id": "17N1039247929"
}

{"name":"accelerator_pedal_position","value":0,"timestamp":1364314054.403000}
{"name":"engine_speed","value":0,"timestamp":1364314054.415000}
{"name":"torque_at_transmission…….