Downtime during migration to V2

[tibin-mfl]

I am trying to upgrade alb ingress controller v1.1.8 to new LB controller v2.0.0, as per this doc -> https://kubernetes-sigs.github.io/aws-load-balancer-controller/guide/upgrade/migrate_v1_v2/

I tried upgrading in my dev cluster and we saw that we are facing downtime during the upgrade process.

Result of the test during the upgrade process 👇

Test-1:

fortio load -a -c 50 -qps 500 -t 300s http://xxx-2048game-2048ingr-6fa0-xxx.ap-south-1.elb.amazonaws.com 
Code 200 : 148983 (99.3 %)
Code 503 : 1017 (0.7 %)

Test-2:

fortio load -a -c 50 -qps 500 -t 200s http://xxx-2048game-2048ingr-6fa0-xxx.ap-south-1.elb.amazonaws.com 
Code 200 : 87031 (87.0 %)
Code 503 : 12969 (13.0 %)

Currently, during the upgrade process before the targets are registered to the new target group, the target group is updated in ALB listeners, so where there are 0 targets in the new target group the ALB listener will be pointing to that target group, that is causing the downtime.
This shouldn’t be the case ALB listener should be updated only once the targets are registered to the new target group.

[kishorj]

There was an issue in the v2.0.0 release which caused downtime during the v1 to v2 migration. The issue has been resolved in the v2.0.1 patch release.

[kishorj]

The latest helm chart uses v2.0.1 release. If you have applied the yaml manifests instead, you can download the latest manifests from https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/main/docs/install/v2_0_1_full.yaml

[M00nF1sh]

you'll need to run helm repo update to get the last helm chart :D

[alfredkrohmer]

@kishorj @M00nF1sh Are you referring this fix? 8ac0f32

If yes, then the documentation here should be updated as well. Thank the flying spaghetti monster, I was already planning a huge migration because of this 🤯 If you want, I can create PR to fix the docs.

[kishorj]

@devkid, 8ac0f32 contains the fix.

You are welcome to submit a PR for the doc fix.

[tibin-mfl]

@kishorj Tried migration with the 2.0.1, it is not having downtime as it's not recreating target group.

I just wanted to know 1 info, currently, the targetPort is 1 itself, was there any use case of making the targetPort to the actual listening port of the service? In the future will I require to update this targetPort for any use-case?

[tibin-mfl]

@kishorj @devkid

I had planned for 0 downtime migration from v1.1.8 to v2.0.0 in this way

• Add group name annotation to all existing ingresses. (This will create new ALB and Target Group)
• Once the LB is provisioned, point the traffic domain to the new ALB
• After the traffic is stopped on old ALB, Uninstall existing AWSALBIngressController
(As domain will be cached in some of the clients until all the requests are stopped to the LB we should not delete the old albingresscontroller)
• Delete old ALB resources manually.

The above steps were working without issues.
In the above scenario, I was updating all my ingresses and I was deploying the new LB controller.
But I wanted to deploy the controller first and then update my ingresses 1 by 1.

This is the scenario what is happened
-> I have an ingress demo-ing, controller v1.1.8, ALB targetPort: 1
-> I deployed v2.0.1
-> Now both controller running and my ALB is intact ALB targetPort: 1 nothing updated in ALB
-> Now I updated my ingress demo-ing, added annotation group.name.
-> This deleted my old ALB with targetPort: 1 and created a new ALB with ALB targetPort: 80.
-> Also as old controller was running it recreated the ALB with ALB targetPort: 1

Expected Behavior.
-> After I deploy v2.0.1 controller, and add an annotation to my ingress
-> Old ALB should not be deleted.
-> We should have the option to delete this manually once all the traffic drained from this.

[kishorj]

@tibin-mfl The 0 downtime migration applies only for the existing ingress objects (with only the v1 supported annotations). If you modify the ingress objects with the new group annotation prior to migration, then the 0 downtime cannot be ensured and also you would have to cleanup the ALB resources manually.
Here is a simpler workflow

• Uninstall the AWSALBIngressController, keep the existing ingress objects intact
• Install AWSLoadBalancerController, wait until all resources get reconciled (a few minutes)

At this point, your setup should function the same way as before. Now if you'd like to use the new ingress group feature, you can first create new ingress objects with the desired group configuration. Once the ALB gets provisioned and functional, you can switch over your application to use the new ALB. When the old ingresses are no longer necessary, you can delete them. The controller will cleanup the necessary AWS resources if tagged properly, for details you can refer to https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/upgrade/migrate_v1_v2/.

You should not run the old AWSALBIngressController and the new AWSLoadBalancerController simultaneously in the same cluster.

[kishorj]

@kishorj Tried migration with the 2.0.1, it is not having downtime as it's not recreating target group.

I just wanted to know 1 info, currently, the targetPort is 1 itself, was there any use case of making the targetPort to the actual listening port of the service? In the future will I require to update this targetPort for any use-case?

We are making it consistent with the target port used for traffic routing in case numeric target ports are specified in the service spec. For 2.0.1 and later, existing target groups from v1 release will get preserved. No changes are necessary from the end user.

[M00nF1sh]

The TargetGroup's port is excluded from the criteria of whether to recreate TargetGroup, thus there won't be downtimes.
We now do our best effort to set the TargetGroup's port correctly. (even its value doesn't matter to this controller's functionality).

[tibin-mfl]

@tibin-mfl The 0 downtime migration applies only for the existing ingress objects (with only the v1 supported annotations). If you modify the ingress objects with the new group annotation prior to migration, then the 0 downtime cannot be ensured and also you would have to cleanup the ALB resources manually.
Here is a simpler workflow

• Uninstall the AWSALBIngressController, keep the existing ingress objects intact
• Install AWSLoadBalancerController, wait until all resources get reconciled (a few minutes)

At this point, your setup should function the same way as before. Now if you'd like to use the new ingress group feature, you can first create new ingress objects with the desired group configuration. Once the ALB gets provisioned and functional, you can switch over your application to use the new ALB. When the old ingresses are no longer necessary, you can delete them. The controller will cleanup the necessary AWS resources if tagged properly, for details you can refer to https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/upgrade/migrate_v1_v2/.

You should not run the old AWSALBIngressController and the new AWSLoadBalancerController simultaneously in the same cluster.

@kishorj
After the new controller is deployed and if I want the new ingress group featured enabled, what should be done?
For ex:
My ingress name is demo-ing and I uninstalled the v1 controller and deployed the v2 controller. Now I want to enable the ingress group feature, if I add an annotation in demo-ing then it will recreate ALB and Target group.
But I want both the ALB to be running until all the traffic is completed on ALB created by the v1 controller.

[M00nF1sh]

@tibin-mfl
changing groupName(including adding groupName) is considered as a disruptive action. The controller won't wait new ALB/TGs to be fully functional before delete the old ALB/TGs. (which is always true since there will be things like external DNS cname updates which is out of this controller's control).

If you want to maintain zero downtime while change groupName, you should create another Ingress with desired groupName instead. and delete old Ingress.