metrics server install should use a "management" node pool

[atimgraves]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

The metrics server is key to services like the auto scalers. Thus is needs to be highly available.
It has been proposed that the cluster autoscaler PR ( #576 ) be updated to make the node pool it creates more general purpose for key cluster management functions (e.g. autoscaler, admission controllers etc.)
Given the metrics server is a key function thbis issue requests that it b deployed into the "management" node pool mentioned above

New or Affected Resource(s)

The metrics server installer should be configured with node selectors and tolerations to ensure it's added to the management node pool. Note that this is something that could be implemented as part of issue #631 (use the metrics server helm chart)

It may be advisable to also ensure (or at least provide the option) that the metrics server is deployed with multiple replicas as part of the same work

Potential Terraform Configuration

References

#576 (cluster auto scaler and management node pool)
#631 (Metrics server helm chart)

[devoncrouse]

Global and pool-specific node_labels added to 5.x in #640

Global and pool-specific node_taints pending in #754

These should be usable with extensions' Helm config to control scheduling as desired.