Cloudformation fails to apply new stacks due max certificates

[ese]

Even including only 25 or less in the cloudformation stack this one fails to apply changes internally.

Once the certificate limit is reached KIAC is able to create a new one ALB stack but if the old one need to change some of their certificates it fails to update. It's common deal with this situation since every new ingress could fit their certificate in the full ALB pushing someother to the new one with spare space.

example:
Imagine a limit of 3 certs
you have one ALB with A[a,b,c] an a new ingress with [d] certificate is created in the cluster.
In the next iteration of the KIAC, detects 2 stacks and start to distribute the ingresses in this stacks with A[d,a,b] and B[c]. This leads make the upgrade of the A stack fails so I guess cloudformation is trying first to add new certs and then delete others.

In addition even having the update stack failed and rollbacked, KIAC updates ingresses statuses with the stacks. In this case statuses of c will point to B while the cert is still in A making a working service to fail

Also once you have the stack in UPDATE_ROLLBACK_COMPLETE status it is not ever touched anymore by KIAC

kube-ingress-aws-controller/worker.go

Lines 51 to 53 in 14fb070

	if !item.certsEqual() && item.stack.IsComplete() {
	return update
	}

This make it unusable for more than 25 certificates

[mikkeloscar]

Thanks for reporting this. We did some changes like sorting certificates to work around bugs in Cloudformation. I suspect this might have broken the original logic for handling the cases where the limit is reached for one ALB.

[ese]

More detail on this as it was tried to handle with #184 reducing max certificates to 24, this still lets stack a brittle condition, specially in a high dynamic environments where ingresses are being added and deleted dinamically in an automated way.

What happens

Once you completed an ALB with 24 certificates (and having 1 spare space for future changes), adding ingresses (or deleting) with 2 new certificates in the same time frame between two polling loops, the controller could lead to a broken stack.

As in the first comment imagine a max of 3 certificates per ALB, you have configured your max to 2 and you have one ALB with [c,d] certifcates.
Then 2 new ingresses appear in the cluster with [a,b] certificates and controller start a new loop.
The sort algorithm from the controller sort at the beginning the new certificates [a,b] so send a cloudformation stack update to first ALB with [a,b] which lead cloudformation to fail becouse there are more new certicates to add (2) than spare space (1) in the stack.
Also controllar is creating a new cloudformation stack for a new ALB with [c,d]

What to expect

Whatever the number of ingresses and related certificates changes in the cluster, controller manage properly the ALBs to end with usable infrastructure

Conclusion

Relying on cloudformation as it seems to works it is a little bit tricky hamdle this.

• 1. do changes one by one in the cloudformation stacks. This could delay applyings too much. Also too much certificate dance between ALB could lead unnecesary downtimes while DNS and other stuff changes are applied.
• 2. Once a certificate lay on one ALB dont move from it in future updates. Pretty much stable in terms of certicate dance but you need to implement extra logic to handle current state and desire state. Also you have take in account when several ingresses dissapear which could lead to move several certifacte to full ALB
• 3. Be less efficiency and only add certs to an existing stack when enough spare space is in the ALB for new certificates, if there is no enough spare space just create new stacks.

[szuecs]

https://github.com/zalando-incubator/kube-ingress-aws-controller/releases/tag/v0.7.1