Running leader election on Karpenter running outside of cluster

[garvinp-stripe]

Description

What problem are you trying to solve?
I am currently running Karpenter "outside" of the cluster, specifically Karpenter itself isn't able to discover API server via K8s Services however we have other method of service discovery to find the API server. We provide Karpenter with a kubeconfig and uses the KUBECONFIG env var. When attempting to run Karpenter with this configuration we run into

 failed to setup manager: unable to find leader election namespace: not running in-cluster, please specify LeaderElectionNamespace

it seems like the controller runtime that implements leader election allows us to run outside but we need to provide the leaderelectionnamespace
https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/leaderelection/leader_election.go#L116

I don't see any configuration that allows us to specify this at this time.

How important is this feature to you?
This is pretty important since we won't be able to run Karpenter in leader elected mode.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[abeer-stripe]

I was able to figure out the solution for us here, and it seems it would work in the general case as well. The solution is just passing LeaderElectionNamespace as system.Namespace() here:

https://github.com/aws/karpenter-core/blob/d34a21d5648d0abd91c58fd60b5a5344b7308498/pkg/operator/operator.go#L110-L112

system.Namespace() is already used here in the same file - https://github.com/aws/karpenter-core/blob/d34a21d5648d0abd91c58fd60b5a5344b7308498/pkg/operator/operator.go#L96C71-L96C90

What this does is it avoid this part of the code in controller-runtime when running leader-election.
https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/leaderelection/leader_election.go#L74

I can raise a PR if this seems agreeable.

[ellistarn]

SGTM.