Load balancers are not torn down after cluster deletion

[d47853]

When the cluster is torn down as the stack is deleted, the load balancers and their target groups remain. This probably happens because the alb helm chart is torn down before the graphs have unregistered.

[m29827]

Uninstalling the gaffer Helm chart before destroying the cluster correctly deletes the application load-balancers and the associated target groups. Running cdk destroy after successfully uninstalls the alb Helm chart and destroys the remaining cluster components.

[ctas582]

Just trying to understand what is missing here - it sounds like the CDK construct that is used to deploy a Helm chart will correctly uninstall the chart when you run cdk destroy or delete the CloudFormation stack. So is the problem that the ALB chart is being uninstalled before the Gaffer chart is uninstalled? In which case a dependency link just needs adding.

[m29827]

Thanks for the comment - Yes that's right, I think adding the dependency would fix it but maybe it's not straight forward to do that dynamically at runtime? If not then that would result in a change to the gaffer chart that coupled it to the AWS ALB ingress controller - at the moment the Kind deployment of this chart uses an Nginx implementation. Maybe there's a better way, I'm a Helm novice!

[d47853]

So is the problem that the ALB chart is being uninstalled before the Gaffer chart is uninstalled?

Yes, though as @m29827 says the graphs are added dynamically by users via the REST API. At the moment all the graphs are deployed to the default namespace so it could be a case of uninstalling all helm releases in the default namespace before uninstalling the ingress controller. It might be possible to do this using a custom resource.

Though if not, then we should add a comment in the docs reminding administrators to uninstall all graphs before destroying the stack for this reason.

[ctas582]

Ahh I see - I incorrectly assumed that the project deployed 1 cluster per graph (not actually properly looked at the code :S). In which case everything would be deployed via a single CloudFormation stack, so you just need to add a DependsOn to the Gaffer Helm Chart that references the ALB Helm Chart.

As you are deploying multiple graphs, outside of CloudFormation, you will likely need to add a Custom Resource. That can be backed by a lambda function, which detects when it is being told to delete, and calls an API endpoint to destroy/uninstall all the deployed graphs.

In addition to ALBs, you will probably also find that EBS volumes are currently being left behind too.

[d47853]

Merged into develop