Prometheus Memory increasing frequently

[sanjay2916]

I have deployed prometheus(1.8.0) with alertmanager(0.5.0) and i have done below setting while deploying

      - '-storage.local.retention=72h'
      - '-storage.local.path=/home'
      - '-config.file=/etc/prometheus/prometheus.yml'
      - '-alertmanager.url=http://localhost:9093'
      - '-storage.local.target-heap-size=512123083'

prometheus memory keep on increasing and it never decreasing and I need to run prometheus with 512MB heapsize and pod memory limit 1024MB without crashing prometheus server with constant memory

prometheus logs:

kubectl logs prometheus-deployment-0 prometheus
time="2018-06-21T12:57:04Z" level=info msg="Starting prometheus (version=1.8.0, branch=HEAD, revision=3569eef8b1bc062bb5df43181b938277818f365b)" source="main.go:87"
time="2018-06-21T12:57:04Z" level=info msg="Build context (go=go1.9.1, user=root@bd4857492255, date=20171006-22:12:46)" source="main.go:88"
time="2018-06-21T12:57:04Z" level=info msg="Host details (Linux 4.14.48-coreos-r2 #1 SMP Thu Jun 14 08:23:03 UTC 2018 x86_64 prometheus-deployment-0 (none))" source="main.go:89"
time="2018-06-21T12:57:04Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:254"
time="2018-06-21T12:57:04Z" level=info msg="Listening on :9090" source="web.go:341"
time="2018-06-21T12:57:04Z" level=info msg="Loading series map and head chunks..." source="storage.go:428"
time="2018-06-21T12:57:04Z" level=info msg="0 series loaded." source="storage.go:439"
time="2018-06-21T12:57:04Z" level=info msg="Server is Ready to receive requests." source="main.go:230"
time="2018-06-21T12:57:04Z" level=info msg="Starting target manager..." source="targetmanager.go:63"
time="2018-06-21T12:57:04Z" level=info msg="Using pod service account via in-cluster config" source="kubernetes.go:105"
time="2018-06-21T12:57:04Z" level=info msg="Using pod service account via in-cluster config" source="kubernetes.go:105"
time="2018-06-21T12:57:04Z" level=info msg="Using pod service account via in-cluster config" source="kubernetes.go:105"
time="2018-06-21T13:02:04Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
time="2018-06-21T13:02:04Z" level=info msg="Done checkpointing in-memory metrics and chunks in 387.927908ms." source="persistence.go:665"

[sanjay2916]

i have set the heapsize to 512MB but prometheus server taking higher memory than heap memory

[krasi-georgiev]

1.8.0? - wow , why not use the latest 2.3 ?
Don't think anyone would want to troubleshoot such an old version.

[sanjay2916]

while deploying prometheus with 2.3 i am getting the below error:

level=error ts=2018-06-22T09:52:18.439015663Z caller=manager.go:483 component="rule manager" msg="loading groups failed" err="yaml: unmarshal errors:\n line 7: cannot unmarshal !!str ALERT N... into rulefmt.RuleGroups"

[krasi-georgiev]

It looks as if this is actually a question about usage and not development.

no crash, no panic = support question 😉

To make support question, and all replies, easier to find, I suggest you move this to our user mailing list.

If you haven't looked already you might find your answer in the official docs and examples or by searching in the users or devs groups.
The #prometheus IRC channel is also a great place to mix up with the community and ask questions (don't be afraid to answers few while waiting).

Once your questions have been answered, please add a link to the solution to help other Prometheans in trouble reaching this from a search 👍

[sanjay2916]

kubectl logs alertmanager-deployment-0

level=info ts=2018-06-24T09:29:18.918038222Z caller=main.go:136 msg="Starting Alertmanager" version="(version=0.14.0, branch=HEAD, revision=30af4d051b37ce817ea7e35b56c57a0e2ec9dbb0)"
level=info ts=2018-06-24T09:29:18.918139388Z caller=main.go:137 build_context="(go=go1.9.2, user=root@37b6a49ebba9, date=20180213-08:16:42)"
level=error ts=2018-06-24T09:29:18.91818351Z caller=main.go:141 msg="Unable to create data directory" err="mkdir data/: permission denied"

[krasi-georgiev]

still not a bug 😄

please move to the user mailing list.
https://groups.google.com/forum/#!forum/prometheus-users

[sanjay2916]

@krasi-georgiev : i have deployed prometheus with 2.3 version but i am facing the same issue and any way to restrict memory utilization of prometheus in 2.3 version

[simonpasquier]

@sanjay2916 you should definitely move your question to the user mailing list. Prometheus 2.x doesn't have any flag to control the memory usage.

[sanjay2916]

@krasi-georgiev : thanks