Skip to content

Commit

Permalink
Fixes #1592: making report::expire faster and without errors
Browse files Browse the repository at this point in the history
  • Loading branch information
orrabin authored and dLobatog committed Aug 7, 2014
1 parent 4c4f30b commit 4215def
Showing 1 changed file with 7 additions and 36 deletions.
43 changes: 7 additions & 36 deletions app/models/report.rb
Expand Up @@ -117,44 +117,15 @@ def <=>(other)
def self.expire(conditions = {})
timerange = conditions[:timerange] || 1.week
status = conditions[:status]
cond = "created_at < \'#{(Time.now.utc - timerange).to_formatted_s(:db)}\'"
cond += " and status = #{status}" unless status.nil?
# using find in batches to reduce the memory abuse
# trying to be smart about how to delete reports and their associated data, so it would be
# as fast as possible without a lot of performance penalties.
count = 0
Report.find_in_batches(:conditions => cond, :select => :id) do |reports|
report_ids = reports.map &:id
Log.delete_all({:report_id => report_ids})
count += Report.delete_all({:id => report_ids})
end
# try to find all non used logs, messages and sources

# first extract all information from our logs
all_reports, used_messages, used_sources = [],[],[]
Log.find_in_batches do |logs|
logs.each do |log|
all_reports << log.report_id unless log.report_id.blank?
used_messages << log.message_id unless log.message_id.blank?
used_sources << log.source_id unless log.source_id.blank?
end
end
cond = "reports.created_at < \'#{(Time.now.utc - timerange).to_formatted_s(:db)}\'"
cond += " and reports.status = #{status}" unless status.nil?

all_reports.uniq! ; used_messages.uniq! ; used_sources.uniq!
Message.joins(:logs).joins(:reports).where(cond)

This comment has been minimized.

Copy link
@ohadlevy

ohadlevy Aug 7, 2014

Member

whats the point in leaving this lines? what is using the query output?

also, wont this never deletes entries from source/message that has no logs?

Source.joins(:logs).joins(:reports).where(cond)
Log.joins(:report).where(:report_id => Report.where(cond)).delete_all

# reports which have logs entries
used_reports = Report.where(:id => all_reports).pluck(:id)

orphaned_logs = all_reports - used_reports
Log.where(:report_id => orphaned_logs).delete_all unless orphaned_logs.empty?

all_messages = Message.pluck(:id)
orphaned_messages = all_messages - used_messages
Message.where(:id => orphaned_messages).delete_all unless orphaned_messages.empty?

all_sources = Source.pluck(:id)
orphaned_sources = all_sources - used_sources
Source.where(:id => orphaned_sources).delete_all unless orphaned_sources.empty?
count = 0
count += Report.where(cond).delete_all

logger.info Time.now.to_s + ": Expired #{count} Reports"
return count
Expand Down

0 comments on commit 4215def

Please sign in to comment.