Auto scaling issue with Prometheus container

[prasenforu]

I am using Prometheus in kubernetes cluster using PVC (Persistent Volume Claim).

As single container its working fine but when I was trying to do autoscaling.

Its going to "CrashLoopBackOff" and in container logs I am getting following error

time="2017-04-11T13:23:52Z" level=info msg="Starting prometheus (version=1.1.1, branch=release-1.0, revision=ab312a075f810e2ed124783c46d68674af07193)" source="main.go:73"
time="2017-04-11T13:23:52Z" level=info msg="Build context (go=go1.6.3, user=root@8ab14ddb4898, date=20160907-09:37:01)" source="main.go:74"
time="2017-04-11T13:23:52Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:221"
time="2017-04-11T13:23:52Z" level=error msg="Could not lock /prometheus/DIRTY, Prometheus already running?" source="persistence.go:180"
time="2017-04-11T13:23:52Z" level=error msg="Error opening memory series storage: resource temporarily unavailable" source="main.go:158"

[sdurrheimer]

Does those auto-scaled prometheus use the same PVC ?

[prasenforu]

yes.

And in latest version also, same issue...

time="2017-04-11T15:02:49Z" level=info msg="Starting prometheus (version=1.5.2, branch=master, revision=bd1182d29f462c39544f94cc822830e1c64cf55b)" source="main.go:75"
time="2017-04-11T15:02:49Z" level=info msg="Build context (go=go1.7.5, user=root@1a01c5f68840, date=20170210-16:23:28)" source="main.go:76"
time="2017-04-11T15:02:49Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:248"
time="2017-04-11T15:02:49Z" level=error msg="Could not lock /prometheus/DIRTY, Prometheus already running?" source="persistence.go:198"
time="2017-04-11T15:02:49Z" level=error msg="Error opening memory series storage: resource temporarily unavailable" source="main.go:182"

[gouthamve]

So you are trying to run two Prometheus on the same data directory. If you want to scale you need to give each Prometheus instance a different directory.

In Prometheus scaling is done via Federation: https://prometheus.io/docs/operating/federation/

[prasenforu]

That's the challenge.

Because container are immutable. For any reason it may die. Till that time another container start my monitoring data lost.

If I have another container running with same data then there is nothing to lose of monitoring data.

I can not use different folder for other container

Please advise.

[gouthamve]

For HA of Prometheus you can run two "independent" (As in on different folders/machines) Prometheuses which scrape the same targets. If one containers goes down, you can switch to the other without any loss.

See: https://prometheus.io/docs/introduction/faq/#can-prometheus-be-made-highly-available

[prasenforu]

That mean when two containers running, both have same data.

In kubernetes cluster we are not doing any manual switching, that is taken care by kubernetes itself.

I think due to present of database inside prometheus container its not allowing scale up with same pvc.

If my understanding is correct then need to rethink on this.

[gouthamve]

As both Prometheuses are scraping the same targets, they hold more or less the same data.

[sdurrheimer]

Prometheus act as a stateful application, two instance can't use the same data directory.

[fabxc]

You either want to look at using StatefulSets, which allow templating of PVCs that are instantiated dynamically or try out the prometheus-operator which handles a lot of those things for you.

[prasenforu]

By the way in prometheus container which folders need setup for Persistent Volume.

I use /var folder from container.

[empyrean987]

@prasenforu did you ever solve this issue?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.