[release-1.24] Wait for kubelet port to be ready before setting

[brandond]

Proposed Changes

When using k3s with karpenter, we have been seeing issues that we are not able to exec and log into pods that lands into karpenter nodes.

When digging further, we have found out that every time we made exec and log requests, k3s agent returns the following log

Tunnel authorizer checking dial request for 127.0.0.1:10250"
Mar 09 17:36:42 ip-10-0-0-125 k3s[15315]: time="2023-03-09T17:36:42Z" level=error msg="Remotedialer proxy error" error="connect not allowed"

When k3s-agent starts, we've seen that it was setting kubelet port to 0, which is not correct.

Tunnel authorizer set Kubelet Port 0

I think what's happening in here is that when using k3s agent with karpenter, the node resource was precreated by karpenter. So agent boots up, it already sees the node but at this point the kubelet port might not be populated back since k3s-agent is still starting. So it just sets the zero port.

Types of Changes

In order to solve this bug, we need to make sure the we only set the port when port is ready(great than 0). This will prevent k3s agent from validating the wrong port and rejecting exec and log request.

Verification

Testing this with aws-karpenter

Testing

Linked Issues

• [release-1.24] Tunnel authorizer sets port before Kubelet is ready #7055

User-Facing Change

The agent tunnel authorizer now waits for the kubelet to be ready before reading the kubelet port from the node object.

Further Comments