Kubectl is returning 'Unable to connect to the server' and 'couldn't get current server API group list' all of a sudden

[DevOpsAzurance]

Running the following steps to install Helm, Kubectl and Kubelogin on self-hosted runner agent:

- task: HelmInstaller@0
  displayName: 'Install Helm'
  inputs:
    helmVersion: '2.14.1'
    installKubectl: true

- task: KubeloginInstaller@0
  inputs:
    kubeloginVersion: 'latest'

We then run login for the managed identity:

$newKubeConfigPath = "$($env:SYSTEM_DEFAULTWORKINGDIRECTORY)/$($env:BUILD_BUILDID)/config"
$env:KUBECONFIG=$newKubeConfigPath
Write-host "##vso[task.setvariable variable=KUBECONFIG;]$newKubeConfigPath"

az login --identity --username "ID"
az aks get-credentials --format azure -n ${{ parameters.aks_name  }} -g ${{ parameters.aks_name  }} --file "$newKubeConfigPath"
kubelogin convert-kubeconfig -l azurecli --kubeconfig $newKubeConfigPath

In a separate job, we are getting the following output:

kubelogin version 
git hash: v0.1.0/0fcd072d45250a50cde855cf50204ad2dc784095
Go version: go1.20.12
Build time: 2023-12-28T19:01:42Z
Platform: linux/amd64
Using KUBECONFIG:
	/usr/EveDORunnersLinux1/_work/10/s/186674/config
Current Configuration:
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: [https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io:443](https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io/)
  name: MyCluster
contexts:
- context:
    cluster: MyCluster
    user: clusterUser_MyCluster_MyCluster
  name: MyCluster
current-context: MyCluster
kind: Config
preferences: {}
users:
- name: clusterUser_MyCluster_MyCluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - get-token
      - --login
      - azurecli
      - --server-id
      - [GUID]
      command: kubelogin
      env: null
      installHint: |2

        kubelogin is not installed which is required to connect to AAD enabled cluster.

        To learn more, please go to https://azure.github.io/kubelogin/
      interactiveMode: IfAvailable
      provideClusterInfo: false
E0222 19:16:55.203802  610079 memcache.go:265] couldn't get current server API group list: Get "[https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io:443/api?timeout=32s"](https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io/api?timeout=32s%22): dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
E0222 19:16:55.239220  610079 memcache.go:265] couldn't get current server API group list: Get "[https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io:443/api?timeout=32s"](https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io/api?timeout=32s%22): dial tcp: lookup MyClusterktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
E0222 19:16:55.314833  610079 memcache.go:265] couldn't get current server API group list: Get "[https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io:443/api?timeout=32s"](https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io/api?timeout=32s%22): dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
E0222 19:16:55.371566  610079 memcache.go:265] couldn't get current server API group list: Get "[https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io:443/api?timeout=32s"](https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io/api?timeout=32s%22): dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
E0222 19:16:55.413369  610079 memcache.go:265] couldn't get current server API group list: Get "[https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io:443/api?timeout=32s"](https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io/api?timeout=32s%22): dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
Unable to connect to the server: dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
E0222 19:16:55.929616  610103 memcache.go:265] couldn't get current server API group list: Get "[https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io:443/api?timeout=32s"](https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io/api?timeout=32s%22): dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
E0222 19:16:55.969947  610103 memcache.go:265] couldn't get current server API group list: Get "[https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io:443/api?timeout=32s"](https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io/api?timeout=32s%22): dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
E0222 19:16:56.017596  610103 memcache.go:265] couldn't get current server API group list: Get "[https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io:443/api?timeout=32s"](https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io/api?timeout=32s%22): dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
E0222 19:16:56.059858  610103 memcache.go:265] couldn't get current server API group list: Get "[https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io:443/api?timeout=32s"](https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io/api?timeout=32s%22): dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
E0222 19:16:56.093238  610103 memcache.go:265] couldn't get current server API group list: Get "[https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io:443/api?timeout=32s"](https://MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io/api?timeout=32s%22): dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
Unable to connect to the server: dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host
Unable to connect to the server: dial tcp: lookup MyCluster-ktlmhe07.privatelink.eastus2.azmk8s.io on 127.0.0.53:53: no such host

It just started happening all of a sudden.

[weinong]

has it worked before? It appears to me that 1) you are using private AKS and 2) you are trying to connect to the private cluster whose fqdn is private. if your self-hosted runners are on a peered vnet, you may use az aks get-credentials --public-fqdn