Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
DART data retrievals not per packet_time #944
I'm starting to populate a DART database with telemetry, and it's not retrieving data by time the way I expected.
In each packet, I have a PACKET_TIME item to override the default RECEIVED_TIMESECONDS. PACKET_TIME populates PACKET_TIMESECONDS and PACKET_TIMEFORMATTED accordingly. These all appear distinct from RECEIVED_TIMESECONDS, and work fine in packet viewer, screens, telemetry grapher, etc.
I'm expecting DART retrievals to be based on PACKET_TIME, however when doing so (with Telemetry Extractor or Grapher), the returned data corresponds to the time range that data was ingested into DART, not the time in PACKET_TIME.
I noticed that dart_common.rb's query_decom_reduced method filters rows via the "time" column. Looking at the table in pgAdmin, the time column does indeed appear to contain received times (or something very close). Note that I can see PACKET_TIME-based seconds and formatted items in later columns (i2,i3,i4).
When dart_packet_log_writer populates that time column, it uses packet.packet_time.dup.utc.iso8601(6), which seems right, but printing that value (or just packet.packet_time) to log right before it gets loaded into @ple_data yields RECEIVED_TIME instead (or something very close).
Is this as intended? or is PACKET_TIME just not accessible when dart_packet_log_writer tries to use it?
After a little more investigating, it appears that the culprit may be DartInterfaceThread's handle_packet method. This overrides the base class handle_packet method, which would normally fully identify/define each packet. DartInterfaceThread's stripped down version does not populate the packet items list, and DartPacketLogWriter later only does a cursory check on the target and packet name. Without PACKET_TIME on the items list, the .packet_time method instead returns the received time.
Here's a bit of a hack for dart_packet_log_writer.rb... Instead of:
Now, I'm really not sure if going through the whole identify_and_define_packet method for every packet is economical or advisable, but it makes sure PACKET_TIME is on the items list and I can now retrieve data from my DART database by packet time instead of local receive/ingest time.
Will have to keep evaluating this to make sure there are no other unintended repercussions. If there's a better way to do this I'm all ears!