[Port v2.9] cluster deploy running excessively and taking a long time to execute

[moio]

This is a port of #39597

Rancher Server Setup

• Rancher version: 2.9.x
• Installation option: any
• Proxy/Cert Details: irrelevant

Information about the Cluster

• Kubernetes version: irrelevant
• Cluster Type (Local/Downstream): issue is seen in local cluster with at least one disconnected cluster

User Information

• What is the role of the user logged in? irrelevant

Describe the bug
With a disconnected downstream cluster, the clusterdeploy handler takes several seconds to time out. This is fine with only a few downstream clusters disconnected, but when the number of disconnected clusters goes over 50 or so, the worker queues will get saturated.

Reason for the slow time out is that the handler may execute code that reaches out to the disconnected downstream cluster - typically clusterDeploy.getNodeAgentImage which tries to list DaemonSets. That alone can take up to ~30 seconds due to how TunnelServer's remotedialer mechanism works.

To Reproduce
See #39597 (comment)

Result

Expected Result
Little to no executions of cluster deploy handler while creating users. Cluster deploy handler should not probably not exceed an execution time of 1 second.

Screenshots
N/A
Additional context
N/A

[moio]

closing per conversation with @git-ival