title | description | ms.date | ms.reviewer | ms.service | ms.custom |
---|---|---|---|---|---|
Resolve az aks command invoke failures |
Resolve az aks command invoke failures in Azure CLI when you try to access a private Azure Kubernetes Service (AKS) cluster. |
10/25/2022 |
chiragpa, andbar, haitch, v-leedennis |
azure-kubernetes-service |
sap:Connectivity, devx-track-azurecli |
This article describes how to resolve az aks command invoke failures in Microsoft Azure CLI so that you can successfully connect to any Azure Kubernetes Service (AKS) cluster, especially to a private AKS cluster.
Other connection methods need to use extra configuration components, as shown in the following table.
Connection methods | Extra configuration component |
---|---|
Virtual network | Virtual private network (VPN) |
Peered network | Azure ExpressRoute |
Private endpoint | Jumpbox |
The az aks command invoke
Azure CLI command is an alternative way of connecting to a cluster that doesn't require extra configuration components.
When you run the az aks command invoke
command, Azure CLI automatically creates a command-<ID>
pod in the aks-command
namespace to access the AKS cluster and retrieve the required information.
-
The Kubernetes kubectl tool. To install kubectl by using Azure CLI, run the az aks install-cli command.
The following table lists common az aks command invoke
error messages. Each error message has a link to the section that describes why the error is occurring, and how to fix it.
Error message | Link |
---|---|
Operation returned an invalid status 'Not Found' | Cause 1: The pod can't be created because of node or resource constraints |
Failed to run command in managed cluster due to kubernetes failure. details: admission webhook "validation.gatekeeper.sh" denied the request: <policy-specific-message> | Cause 2: Azure Policy doesn't allow the pod creation |
Error from server (Forbidden): namespaces is forbidden: User "<ID>" cannot list resource "<resource>" in API group "" at the cluster scope | Cause 3: Required roles aren't granted |
Failed to connect to MSI. Please make sure MSI is configured correctly. Get Token request returned: Response [400]; |
Cause 4: There's a Cloud Shell issue |
The operation returns a Not Found
status because the command-<ID>
pod can't reach a successful state, such as Running
. (In many cases, the pod stays in the Pending
state.) In this case, the nodes aren't able to schedule the pod. This scenario can have different causes, such as the following causes:
- Resource constraints
- Nodes that have a
NotReady
orSchedulingDisabled
state - Nodes that have taints that the pod can't tolerate
- Other causes
Make sure that the command-<ID>
pod can be scheduled and run.
If you have specific Azure policies, the az aks command invoke
command can fail because of a disallowed configuration in the command-<ID>
pod. For example, you might have an Azure policy that requires a read-only root file system or other specific configuration.
We recommend that you exempt the aks-command
namespace for the associated Azure policies that don't allow the pod creation. For more information about exemption, see Understand scope in Azure Policy
To exempt an Azure Policy:
-
In the Azure portal, search for and select Policy.
-
In the Policy navigation pane, locate the Authoring section, and then select Assignments.
-
In the table of assignments, find the row that contains the Assignment name that you want to change, and then select the name of the assignment.
-
In the policy assignment page for that assignment, select Edit assignment.
-
Select the Parameters tab.
-
Clear the Only show parameters that need input or review option.
-
In the Namespace exclusions box, add the aks-command namespace to the list of namespaces to be excluded.
Alternatively, if the policy isn't a built-in policy, you can check the configuration of the command-<ID>
pod, and adjust the policy as necessary. To explore the pod's YAML configuration, run the following command:
kubectl get pods command-<ID> --namespace aks-command --output yaml
To use the az aks command invoke
command, you must have access to the following roles on the cluster:
Microsoft.ContainerService/managedClusters/runCommand/action
Microsoft.ContainerService/managedClusters/commandResults/read
If you don't have these roles, the az aks command invoke
command can't retrieve the required information.
Add the Microsoft.ContainerService/managedClusters/runCommand/action
and Microsoft.ContainerService/managedClusters/commandResults/read
roles.
The az aks command invoke
command isn't processed as expected when it's run directly in the Azure Cloud Shell environment. This is a known issue in Cloud Shell.
In Cloud Shell, run the az login command before you run az aks command invoke
.
Run the az aks command invoke
command on a local computer or any virtual machine (VM) that has Azure CLI installed.
[!INCLUDE Third-party disclaimer]
[!INCLUDE Azure Help Support]