Kubernetes pod scaling

[prasenforu]

Recently we tried container monitoring with Prometheus.

Is there any approach where we can scale container/pod using cpu or memory metric in prometheus and alertmanager?

[gouthamve]

You could write an alert at which you want to scale (ex: CPU > 80% FOR 10mins).
Run Alertmanager and use a webhook receiver to write your scaling logic.

Usage related questions are best limited to: https://groups.google.com/forum/#!forum/prometheus-users

[prasenforu]

Will it work when it recover it should do scale down.

Can you please refer any blog or mail thread on this subject.

[juliusv]

On Tue, Mar 28, 2017 at 4:22 PM, prasenforu ***@***.***> wrote: Will it work when it recover it should do scale down.

Yes, you can also configure the webhook notifier to send notifications on resolved alerts. See https://prometheus.io/docs/alerting/configuration/# <webhook_config>

Can you please refer any blog or mail thread on this subject.

For example, https://www.robustperception.io/audio-alerting-with-prometheus/

[prasenforu]

Thanks for response.

I am more interested as follows,

When cpu high it should trigger one webhook.
Which I am able to do by setting receiver webhook.

But when it resolved it should trigger another webhook.

Which I am not able to configure.

Plesae advice.

[juliusv]

We don't support different receivers for firing/resolved for the same alert, but you could emulate it with two alerting rules, one that fires when something is bad and one that fires on the opposite condition (everything is good), and then notify different webhooks for each when they are firing. For example:

ALERT InstanceDown up{job="myservice"} == 0
[...]

ALERT AllInstancesUp avg(up{job="myservice"}) == 1
[...]

Not sure how great that pattern is though :)

[juliusv]

Closing, as questions are for the users mailing list, not GitHub issues. See https://prometheus.io/community/.

[prasenforu]

If I create another alert for scale down.

Like cpu < 70%

But in that scenario it is allways true if I do then it will scale down to zero than mean no pod running.

My question any alternative approach or does it support alert as range.

CPU > 40% but CPU < 70%

in a single alert.

[andrewhowdencom]

@prasenforu If you ask your question on the mailing list I may be able to help further. (Paste a link here once you have)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.