Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metrics: reduce memory footprint by over 60% #1251

Merged
merged 3 commits into from Nov 9, 2017

Conversation

@jberry-suse
Copy link
Contributor

jberry-suse commented Nov 9, 2017

  • d1b6125:
    metrics: rework to store points as named tuple and write in batches.

    Savings of around 400MB per 10,000 requests. Using a named tuple was the
    original approach for this reason, but the influxdb interface requires
    dict()s and it seemed silly to spend time converting them. Additionally,
    influxdb client already does batching.

    Unfortunately, with the amount of data processed for Factory that will
    continue to grow this approach is necessary. The dict() final structures
    are buffered up to ~1000 before being written and released. Another
    benefit of the batching is that influxdb does not allocate memory for the
    entire incoming batch.

  • 6a25987:
    metrics: rework request pagination to provide as generator.

    This prevents the memory required to process to be enough to load all
    parsed request element trees at once. Instead only one page of requests
    is loaded at a time and the memory freed after processed. The end result
    is the memory consumption reduced by just over 20% (current Factory drops
    by around 2.5GB).

  • 0de7fb8:
    metrics: call ET.clear() to release unneeded memory used by search result.

    Roughly 1800MB per 10,000 requests saved.

Without these changes the ingest process will not run on production machine even with 10GB of RAM. This drop the required memory from around 12GB to 4.5GB. The small batches also reduces the influxdb process overhead during writing. Interestingly, as noted in commit message above, one of the improvements was the original way I approached this, but changed to match influxdb interface. :( Additionally, the need to call ElementTree.clear() is rather odd. The combined effect of these changes should make 10GB more than enough RAM for several years to come at which point the incoming requests could potentially be partitioned or perhaps a RAM increase will be irrelevant.

@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Nov 9, 2017

OBS is currently down which explains the osc build CI failure.

jberry-suse added 3 commits Nov 9, 2017
…sult.

Roughly 1800MB per 10,000 requests saved.
This prevents the memory required to process to be enough to load all
parsed request element trees at once. Instead only one page of requests
is loaded at a time and the memory freed after processed. The end result
is the memory consumption reduced by just over 20% (current Factory drops
by around 2.5GB).
Savings of around 400MB per 10,000 requests. Using a named tuple was the
original approach for this reason, but the influxdb interface requires
dict()s and it seemed silly to spend time converting them. Additionally,
influxdb client already does batching.

Unfortunately, with the amount of data processed for Factory that will
continue to grow this approach is necessary. The dict() final structures
are buffered up to ~1000 before being written and released. Another
benefit of the batching is that influxdb does not allocate memory for the
entire incoming batch.
@jberry-suse jberry-suse force-pushed the jberry-suse:metrics-memory-reduction branch from d1b6125 to 26883b1 Nov 9, 2017
@coveralls

This comment has been minimized.

Copy link

coveralls commented Nov 9, 2017

Coverage Status

Coverage decreased (-0.04%) to 31.084% when pulling 26883b1 on jberry-suse:metrics-memory-reduction into 3e191ca on openSUSE:master.

@openSUSE openSUSE deleted a comment from coveralls Nov 9, 2017
@jberry-suse jberry-suse merged commit f927c57 into openSUSE:master Nov 9, 2017
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@jberry-suse jberry-suse deleted the jberry-suse:metrics-memory-reduction branch Nov 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.