New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dart updates for ingest speed, correct time zone, tlmgrapher crash #779

Merged
merged 4 commits into from May 15, 2018

Conversation

Projects
None yet
3 participants
@ryanatball
Member

ryanatball commented Apr 27, 2018

and more

Still needs:

  1. DART size optimization. DART grows way faster than the raw data right now. Need to eliminate some indexes and fields if possible.

  2. New dart util needs to complete capability to remove files of data. To do this efficiently may need to add a packet_log_id to decom/reduced tables.

  3. Investigate if string limit of 191 is wasting space

  4. Decom needs to be faster. I increased the default num workers to 2, but it needs to run faster in general.

@ryanatball ryanatball requested review from jasonatball and donaldatball Apr 27, 2018

@ryanatball

This comment has been minimized.

Member

ryanatball commented Apr 27, 2018

  1. doesn't appear to be an issue. Varchars don't use the full space.
@codecov-io

This comment has been minimized.

codecov-io commented Apr 27, 2018

Codecov Report

Merging #779 into master will decrease coverage by 0.3%.
The diff coverage is 43.42%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #779      +/-   ##
==========================================
- Coverage   85.65%   85.34%   -0.31%     
==========================================
  Files         159      160       +1     
  Lines       15041    15215     +174     
==========================================
+ Hits        12884    12986     +102     
- Misses       2157     2229      +72
Impacted Files Coverage Δ
lib/cosmos/io/json_drb_object.rb 96.66% <100%> (+0.15%) ⬆️
lib/cosmos/top_level.rb 84.91% <100%> (ø) ⬆️
lib/cosmos/packets/packet_config.rb 98.96% <100%> (+0.02%) ⬆️
lib/cosmos/interfaces.rb 100% <100%> (ø) ⬆️
lib/cosmos/io/json_drb.rb 96.79% <20%> (-1.9%) ⬇️
lib/cosmos/interfaces/dart_status_interface.rb 22% <22%> (ø)
lib/cosmos/interfaces/serial_interface.rb 73.33% <0%> (-5.84%) ⬇️
lib/cosmos/io/serial_driver.rb 90% <0%> (-2.86%) ⬇️
...smos/tools/cmd_tlm_server/cmd_tlm_server_config.rb 100% <0%> (ø) ⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c62f78...ba43cb8. Read the comment docs.

@@ -6,6 +6,7 @@ DECLARE_TARGET INST
DECLARE_TARGET INST INST2
DECLARE_TARGET EXAMPLE
DECLARE_TARGET TEMPLATED
DECLARE_TARGET DART

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

This basically makes DART a part of the Demo now. Don't we need to modify the Gemfile to make sure we're including all the DART dependencies. I think you have it commented out right now.

This comment has been minimized.

@ryanatball

ryanatball May 14, 2018

Member

This adds a DART target, but it doesn't bring in all the gem requirements. I'll add DONT_CONNECT to the interface so that it doesn't add noise if DART is not running or used.

@@ -0,0 +1,2 @@
COMMAND DART CLEARERRORS BIG_ENDIAN "Clears error counters and messages"
APPEND_ID_PARAMETER PACKETID 8 INT 1 1 1 "Packet Id"

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

PACKET_ID

@@ -0,0 +1,40 @@
TELEMETRY DART STATUS BIG_ENDIAN "DART Status"
APPEND_ID_ITEM PACKETID 8 INT 1 "Packet Id"

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

PACKET_ID

@@ -0,0 +1,7 @@
# Ignored Parameters
# IGNORE_PARAMETER parameter_name
IGNORE_PARAMETER PACKETID

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

PACKET_ID

# Ignored Items
# IGNORE_ITEM item_name
IGNORE_ITEM PACKETID

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

PACKET_ID

@@ -146,8 +165,22 @@ def import(filename, force)
meta_id = ple.id
end
end
if ple_data_count >= 1000

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

Clever optimization

@ple_data << "('#{entry_time}','#{entry_time}',#{target_id},#{packet_id},'#{packet.received_time.dup.utc.to_s(:db)}',#{@packet_log_id},#{@file_size},#{@meta_id},#{@is_tlm},true)"
else
@ple_data << ",('#{entry_time}','#{entry_time}',#{target_id},#{packet_id},'#{packet.received_time.dup.utc.to_s(:db)}',#{@packet_log_id},#{@file_size},#{@meta_id},#{@is_tlm},true)"
end

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

Same comment as above.

super()
@first = true
@status_packet = System.telemetry.packet('DART', 'STATUS').clone
@status_packet.write('PACKETID', 1)

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

PACKET_ID

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

Can't you also just do a restore_defaults here?

This comment has been minimized.

@ryanatball

ryanatball May 14, 2018

Member

Only commands have defaults

@first = true
@status_packet = System.telemetry.packet('DART', 'STATUS').clone
@status_packet.write('PACKETID', 1)
@clear_errors_command = System.commands.packet('DART', 'CLEARERRORS')

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

I didn't notice this before but I'd prefer CLEAR_ERRORS here with an underscore.

canceled = false
unless @first
if @query_time
sleep_time = 20.0 - (Time.now - @query_time)

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

I think the 20 should be a config parameter to the interface. Someone is going to want to speed up or slow this down. Perhaps you make a minimum limit on it. Not sure how long it takes but you're doing a bunch of DB stuff. Maybe 5s minimum?

@ryanatball

This comment has been minimized.

Member

ryanatball commented May 14, 2018

Comments incorporated along with some other changes. ready for final review.

start_time = first_packet.received_time
end_time = last_packet.received_time
rescue
if size == 128 or size == 0

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

What is this magic number?

This comment has been minimized.

@ryanatball

ryanatball May 15, 2018

Member

Its the size of the normal COSMOS file header. I just checked in a change to remove the magic...

ensure
reader.close
end
num_deleted = PacketLogEntry.where("time >= ? and time <= ? and packet_log_id = ?", start_time - 1.minute, end_time + 1.minute, packet_log.id).delete_all

This comment has been minimized.

@jasonatball

jasonatball May 14, 2018

Collaborator

What's with the +- 1 minute

This comment has been minimized.

@ryanatball

ryanatball May 15, 2018

Member

This just provides some tolerance incase the timestamps in the log were a little out of order. The packet_log_id constraint makes sure nothing extra is deleted.

@ryanatball ryanatball merged commit bb8fb1b into master May 15, 2018

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@ryanatball ryanatball deleted the dart_updates branch May 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment