Memory usage: Observing paging and swapping

[dhbarman]

Proposal

Use case. Why is this important?

Better system performance.

Nice to have' is not a good use case :)

Bug Report

What did you do?

Nothing

What did you expect to see?

No paging and swapping as memory available is high.

What did you see instead? Under which circumstances?

Paging and swapping are observed on Prometheus nodes while memory is still available.

Environment

• System information:

  Linux 4.1.12-124.15.2.el7uek.x86_64 x86_64

• Prometheus version:

  prometheus --version
  prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c)
  build user: root@149e5b3f0829
  build date: 20180314-14:15:45
  go version: go1.10

[krasi-georgiev]

why do you think Prometheus is causing this? AFAIK It is the OS that controls Paging and swapping.

At compaction or rules evaluation it might use more RAM so maybe this is when the OS decides to SWAP.

[dhbarman]

If OS controls the behavior, shouldn't we see consistent or uniform behavior ? We see occasional high swapping and paging. I am collecting more data which I will upload here.

[dhbarman]

Attaching 3 graphs all from the same window of time:

1. Virtual memory: Paging and swapping
2. Memory Available and Swap Used %
3. Disk Performance %

[krasi-georgiev]

I imagine this happens at compaction when Prometheus needs more memory to expand and merge the blocks.

[dhbarman]

Is there anything we can do to avoid such spikes ? Or is it expected ? Does it have any correlation with incoming queries ?

[krasi-georgiev]

Is there anything we can do to avoid such spikes ? Or is it expected ?

It is to be expected.

Does it have any correlation with incoming queries ?

yes any heavy queries will greatly increase memory usage.

[dhbarman]

Is there no mitigation step? Do you suggest that we should increase swap memory allocation? Do you have any code change in prometheus which will improve its memory footprint?

[krasi-georgiev]

decrease load , increase available memory. Anything along these lines should help.
There are constant attempts to improve the memory usage.

Since this is not a bug I would suggest closing this and moving all additional questions to the user mailing list.

no crash, no panic = user mailing list 😉

[krasi-georgiev]

@dhbarman did you get this resolved so I can close the issue?

[dhbarman]

@krasi-georgiev Have not resolved it completely. We will keep monitoring our servers. We plan to upgrade to a newer version of Prometheus too. You may close the ticket for now.

[krasi-georgiev]

ok, thanks. Feel free to add more updates.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.