Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DART data retrievals not per packet_time #944

spcxplorer opened this issue Feb 7, 2019 · 3 comments · Fixed by #1103

DART data retrievals not per packet_time #944

spcxplorer opened this issue Feb 7, 2019 · 3 comments · Fixed by #1103


Copy link

@spcxplorer spcxplorer commented Feb 7, 2019

I'm starting to populate a DART database with telemetry, and it's not retrieving data by time the way I expected.

In each packet, I have a PACKET_TIME item to override the default RECEIVED_TIMESECONDS. PACKET_TIME populates PACKET_TIMESECONDS and PACKET_TIMEFORMATTED accordingly. These all appear distinct from RECEIVED_TIMESECONDS, and work fine in packet viewer, screens, telemetry grapher, etc.

I'm expecting DART retrievals to be based on PACKET_TIME, however when doing so (with Telemetry Extractor or Grapher), the returned data corresponds to the time range that data was ingested into DART, not the time in PACKET_TIME.

I noticed that dart_common.rb's query_decom_reduced method filters rows via the "time" column. Looking at the table in pgAdmin, the time column does indeed appear to contain received times (or something very close). Note that I can see PACKET_TIME-based seconds and formatted items in later columns (i2,i3,i4).

When dart_packet_log_writer populates that time column, it uses packet.packet_time.dup.utc.iso8601(6), which seems right, but printing that value (or just packet.packet_time) to log right before it gets loaded into @ple_data yields RECEIVED_TIME instead (or something very close).

Is this as intended? or is PACKET_TIME just not accessible when dart_packet_log_writer tries to use it?

  • OS: Windows 7
  • COSMOS Version 4.3.0

This comment has been minimized.

Copy link

@spcxplorer spcxplorer commented Feb 12, 2019

After a little more investigating, it appears that the culprit may be DartInterfaceThread's handle_packet method. This overrides the base class handle_packet method, which would normally fully identify/define each packet. DartInterfaceThread's stripped down version does not populate the packet items list, and DartPacketLogWriter later only does a cursory check on the target and packet name. Without PACKET_TIME on the items list, the .packet_time method instead returns the received time.

Here's a bit of a hack for dart_packet_log_writer.rb... Instead of:

@ple_data << "(#{target_id},#{packet_id},'#{packet.packet_time.dup.utc.iso8601(6)}',#{@packet_log_id},#{@file_size},#{@meta_id},#{@is_tlm},true)"


identified_packet = Cosmos::System.telemetry.identify_and_define_packet(packet)

@ple_data << "(#{target_id},#{packet_id},'#{identified_packet.packet_time.dup.utc.iso8601(6)}',#{@packet_log_id},#{@file_size},#{@meta_id},#{@is_tlm},true)"

Now, I'm really not sure if going through the whole identify_and_define_packet method for every packet is economical or advisable, but it makes sure PACKET_TIME is on the items list and I can now retrieve data from my DART database by packet time instead of local receive/ingest time.

Will have to keep evaluating this to make sure there are no other unintended repercussions. If there's a better way to do this I'm all ears!


This comment has been minimized.

Copy link

@jasonatball jasonatball commented Feb 13, 2019

@spcxplorer thanks for looking into this and proposing a possible solution. I'm going to defer to @ryanatball on this one as he is the primary DART author and I'm not sure what he was thinking here.


This comment has been minimized.

Copy link

@jmthomas jmthomas commented Nov 23, 2019

@ryanatball do you think #1090 will address this issue?

ryanatball added a commit that referenced this issue Dec 27, 2019
ryanatball added a commit that referenced this issue Dec 30, 2019

closes #944. Fix packet_time in ingestor. Also improve System mutexing
@ryanmelt ryanmelt added this to the v4.4.1 milestone Dec 31, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
4 participants
You can’t perform that action at this time.