scaling to 0 removes alert rules #35
Description
I am trying to get the benefits of serverless, with microservices by autoscaling based on queue length. When there are no messages in the queue, the service can be scaled down to 0.
I'm using @thomasjpfan's https://github.com/thomasjpfan/docker-scaler, though Jenkins job would be the same, and a rabbitmq prometheus exporter. This allows me to scale without instrumenting the service specifically.
~ $ docker service logs voelhgexsur4
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal | 2018/02/20 07:21:32 Scale service down: rethink-denormalizer_service
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal | 2018/02/20 07:21:32 scale-service success: Scaling rethink-denormalizer_service from 1 to 0 replicas (min: 0)
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal | 2018/02/20 07:21:32 Alertmanager received message: Scaling rethink-denormalizer_service from 1 to 0 replicas (min: 0)
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal | 2018/02/20 07:31:32 Scale service down: rethink-denormalizer_service
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal | 2018/02/20 07:31:32 scale-service success: Scaling rethink-denormalizer_service from 1 to 0 replicas (min: 0)
monitor_scaler.1.voelhgexsur4@ip-172-31-0-9.ec2.internal | 2018/02/20 07:31:32 Alertmanager received message: Scaling rethink-denormalizer_service from 1 to 0 replicas (min: 0)
It works great.
Problem is, this bring the service down in the eyes of DFM, and correctly, for DFP.
In DFM this is an issue because now, because the alerts needed to scale back up when there are messages in the queue again have been removed as part of the service removed event.