Controller Scaling pods out of service without health pods

[sstarcher]

deis limits:set --cpu cmd=200000

Will of course cause the pods to not be able to schedule. Upon doing this the controller waits for the new pods to become healthy and after the timeout, while the pods are still pending it will shutdown the functioning pods.

INFO [APP]: config APP-1b1f384 updated
INFO [APP]: sstarcher changed nothing
INFO [APP]: quay.io/image/image:latest exists in the target registry. Using image for release 32 of app APP
INFO deploy APP-v32-cmd, img quay.io/image/image:latest, cmd ""
INFO scaling release APP-v32-cmd to 4 out of final 4
INFO adding 30s on to the original 120s timeout to account for the initial delay specified in the readiness probe
INFO waiting for 4 pods in APP namespace to be in services (150 timeout)
INFO waited 20s and 0 pods are in service
INFO waited 10s and 0 pods are in service
INFO waited 30s and 0 pods are in service
INFO waited 20s and 0 pods are in service
INFO waited 40s and 0 pods are in service
INFO waited 30s and 0 pods are in service
INFO waited 50s and 0 pods are in service
INFO waited 40s and 0 pods are in service
INFO waited 60s and 0 pods are in service
INFO waited 50s and 0 pods are in service
INFO waited 70s and 0 pods are in service
INFO waited 60s and 0 pods are in service
INFO waited 80s and 0 pods are in service
INFO waited 70s and 0 pods are in service
INFO waited 90s and 0 pods are in service
INFO waited 80s and 0 pods are in service
INFO waited 100s and 0 pods are in service
INFO waited 90s and 0 pods are in service
INFO waited 110s and 0 pods are in service
INFO waited 100s and 0 pods are in service
INFO waited 120s and 0 pods are in service
INFO waited 110s and 0 pods are in service
INFO waited 130s and 0 pods are in service
INFO waited 120s and 0 pods are in service
INFO waited 140s and 0 pods are in service
INFO waited 130s and 0 pods are in service
INFO waited 150s and 0 pods are in service
INFO waited 140s and 0 pods are in service
INFO waited 160s and 0 pods are in service
INFO 0 out of 4 pods in namespace APP are in service
INFO scaling old release APP-v31-cmd from original 4 to 0
INFO Not scaling RC APP-v31-cmd in Namespace APP to 0 replicas. Already at desired replicas
INFO Not scaling RC APP-v31-cmd in Namespace APP to 0 replicas. Already at desired replicas
10.245.65.10 "POST /v2/apps/APP/config/ HTTP/1.1" 201 628 "Deis Client v2.0.0-dev"
10.245.39.145 "GET /v2/apps/APP/config/ HTTP/1.1" 200 628 "Deis Client v2.0.0-dev"
INFO waited 170s and 0 pods are in service
INFO waited 180s and 0 pods are in service
INFO waited 190s and 0 pods are in service
INFO waited 200s and 0 pods are in service
INFO waited 210s and 0 pods are in service
INFO waited 220s and 0 pods are in service
INFO waited 230s and 0 pods are in service
INFO waited 240s and 0 pods are in service
INFO waited 250s and 0 pods are in service
INFO waited 260s and 0 pods are in service
INFO waited 270s and 0 pods are in service
INFO waited 280s and 0 pods are in service
INFO waited 290s and 0 pods are in service
INFO waited 300s and 0 pods are in service
INFO waited 310s and 0 pods are in service
INFO [APP]: config APP-2e15ce8 updated
INFO [APP]: sstarcher changed limits for cpu
INFO [APP]: quay.io/image/image:latest exists in the target registry. Using image for release 33 of app APP
INFO deploy APP-v33-cmd, img quay.io/image/image:latest, cmd ""
INFO scaling release APP-v33-cmd to 4 out of final 4
INFO adding 30s on to the original 120s timeout to account for the initial delay specified in the readiness probe
INFO waiting for 4 pods in APP namespace to be in services (150 timeout)
INFO waited 320s and 0 pods are in service
INFO waited 10s and 0 pods are in service
INFO waited 330s and 0 pods are in service
INFO waited 20s and 0 pods are in service
INFO waited 340s and 0 pods are in service

[sstarcher]

The above is the same or a similar problem to #706

[sstarcher]

I recently recreated this problem in v2.1 prior versions cpu limits were in milicores and after updating to 2.1 the new setup is by default in full cores causing all pods to be pending, causing an outage.

INFO 0 out of 3 pods in namespace public are in service
INFO scaling old release public-v10-cmd from original 4 to 1
INFO scaling RC public-v10-cmd in Namespace public from 4 to 1 replicas
INFO adding 10s on to the original 120s timeout to account for the initial delay specified in the readiness probe
INFO waiting for 1 pods in public namespace to be in services (130 timeout)
INFO 1 out of 1 pods in namespace public are in service
INFO waiting for 3 pods in public namespace to be terminated (60s timeout)
INFO 3 pods in namespace public are terminated
INFO scaling release public-v11-cmd to 4 out of final 4
INFO scaling RC public-v11-cmd in Namespace public from 3 to 4 replicas
INFO waiting for 4 pods in public namespace to be in services (120 timeout)
INFO waited 10s and 0 pods are in service
INFO waited 20s and 0 pods are in service
INFO waited 30s and 0 pods are in service
INFO waited 40s and 0 pods are in service
INFO waited 50s and 0 pods are in service
INFO waited 60s and 0 pods are in service
INFO waited 70s and 0 pods are in service
INFO waited 80s and 0 pods are in service
INFO waited 90s and 0 pods are in service
INFO waited 100s and 0 pods are in service
INFO waited 110s and 0 pods are in service
INFO 0 out of 4 pods in namespace public are in service
INFO scaling old release public-v10-cmd from original 4 to 0
INFO scaling RC public-v10-cmd in Namespace public from 1 to 0 replicas
INFO waiting for 1 pods in public namespace to be terminated (60s timeout)
INFO 1 pods in namespace public are terminated
INFO Not scaling RC public-v10-cmd in Namespace public to 0 replicas. Already at desired replicas

[helgi]

#792 has not yet been merged which would have prevented the outage. There is an edge case that I need to work out.

Can't speak to the milli vs full core issue. @bacongobbler think you looked at something similar to this recently?

[sstarcher]

@helgi the change to the milli vs full was an expected change that @bacongobbler

Just making sure to keep the ticket updated incase any of the info above was useful.

[bacongobbler]

Can't speak to the milli vs full core issue. @bacongobbler think you looked at something similar to this recently?

deis/workflow-cli#75

[helgi]

I believe this was fixed by #978 which should be part of the 2.4 release that is about to drop - Let us know if you still run into this problem after that.