Skip to content

Commit

Permalink
Reduce memory bloating in FileMappedDict when reading metrics.
Browse files Browse the repository at this point in the history
During the read path (metrics export) we open all existing metric files
and read through them in order to aggregate metrics across different
processes. When reading non-empty files we end up parsing file content
twice: first during FileMappedDict initialisation, then again in the
caller site (`all_values`).

This commit refactors FileMappedDict so that while the `@positions` map
is populated at creation time, the actual metric values are read only
when explicitly requested. This avoids the memory bloat of unpacking
file content twice.
  • Loading branch information
Cristian Greco committed Oct 7, 2019
1 parent af29066 commit 8a8c92b
Showing 1 changed file with 12 additions and 16 deletions.
28 changes: 12 additions & 16 deletions lib/prometheus/client/data_stores/direct_file_store.rb
Original file line number Diff line number Diff line change
Expand Up @@ -215,11 +215,7 @@ def initialize(filename, readonly = false)

if @used > 0
# File already has data. Read the existing values
with_file_lock do
read_all_values.each do |key, _, pos|
@positions[key] = pos
end
end
with_file_lock { populate_positions }
else
# File is empty. Init the `used` counter, if we're in write mode
if !readonly
Expand All @@ -230,10 +226,14 @@ def initialize(filename, readonly = false)
end
end

# Yield (key, value, pos). No locking is performed.
# Return a list of key-value pairs
def all_values
with_file_lock do
read_all_values.map { |k, v, p| [k, v] }
@positions.map do |key, pos|
@f.seek(pos)
value = @f.read(8).unpack('d')[0]
[key, value]
end
end
end

Expand Down Expand Up @@ -309,22 +309,18 @@ def init_value(key)
@positions[key] = @used - 8
end

# Yield (key, value, pos). No locking is performed.
def read_all_values
# Read position of all keys. No locking is performed.
def populate_positions
@f.seek(8)
values = []
while @f.pos < @used
padded_len = @f.read(4).unpack('l')[0]
encoded = @f.read(padded_len).unpack("A#{padded_len}")[0]
value = @f.read(8).unpack('d')[0]
values << [encoded.strip, value, @f.pos - 8]
key = @f.read(padded_len).unpack("A#{padded_len}")[0].strip
@positions[key] = @f.pos
@f.seek(8, :CUR)
end
values
end
end
end
end
end
end


0 comments on commit 8a8c92b

Please sign in to comment.