Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set-AzAksCluster fails to upgrade cluster #17376

Closed
aelij opened this issue Mar 7, 2022 · 2 comments · Fixed by #17776
Closed

Set-AzAksCluster fails to upgrade cluster #17376

aelij opened this issue Mar 7, 2022 · 2 comments · Fixed by #17776
Assignees
@wyunchi-ms
Labels
AKS bug This issue requires a change to an existing behavior in the product in order to be resolved. P1

Comments

@aelij
Copy link
Member

aelij commented Mar 7, 2022

Description

Error:

The identity ids must not be null or empty for 'UserAssigned' identity type.

This looks very similar to #14583

Issue script & Debug output

DEBUG: 10:28:12 - SetAzureRmAks begin processing with ParameterSet 'InputObjectParameterSet'.
WARNING: Upcoming breaking changes in the cmdlet 'Set-AzAksCluster' :
Set-AzAks will be removed in the next major release. Please use Set-AzAksCluster instead of Set-AzAks
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.
DEBUG: SharedTokenCacheCredential.GetToken invoked. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId: 
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] Azure region was not configured or could not be discovered. Not using a regional authority.
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12] Found 1 cache accounts and 0 broker accounts
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12] Returning 1 accounts
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] MSAL MSAL.NetCore with assembly version '4.30.1.0'. CorrelationId(<id>)
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] === AcquireTokenSilent Parameters ===
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] LoginHint provided: False
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] Account provided: True
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] ForceRefresh: False
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] 
=== Request Data ===
Authority Provided? - True
Scopes - https://management.core.windows.net//.default
Extra Query Params Keys (space separated) - 
ApiId - AcquireTokenSilent
IsConfidentialClient - False
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - <id>

DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] === Token Acquisition (SilentRequest) started:

        Authority Host: login.microsoftonline.com
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] Azure region was not configured or could not be discovered. Not using a regional authority.
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] Access token is not expired. Returning the found cache entry. [Current time (03/07/2022 08:28:12) - Expiration Time (03/07/2022 09:29:31 +00:00) - Extended Expiration Time (03/07/2022 09:29:31 +00:00)]
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] Returning access token found in cache. RefreshOn exists ? False
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] Fetched access token from host login.microsoftonline.com. 
DEBUG: (False) MSAL 4.30.1.0 MSAL.NetCore Microsoft Windows 10.0.22000 [03/07/2022 08:28:12 - ] === Token Acquisition finished successfully. An access token was returned with Expiration Time: 03/07/2022 09:29:31 +00:00 and Scopes https://management.core.windows.net//user_impersonation https://management.core.windows.net//.default
DEBUG: SharedTokenCacheCredential.GetToken succeeded. Scopes: [ https://management.core.windows.net//.default ] ParentRequestId:  ExpiresOn: 2022-03-07T09:29:31.0000000+00:00
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/subid/resourceGroups/name/providers/Microsoft.ContainerService/managedClusters/name?api-version=2021-05-01

Headers:
x-ms-client-request-id        : <id>
Accept-Language               : en-US

Body:



DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-ratelimit-remaining-subscription-reads: 11999
x-ms-correlation-request-id   : <id>
x-ms-request-id               : <id>
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
Server                        : nginx
Date                          : Mon, 07 Mar 2022 08:28:11 GMT

Body:
{
  "id": "/subscriptions/subid/resourcegroups/name/providers/Microsoft.ContainerService/managedClusters/name",
  "location": "norwayeast",
  "name": "name",
  "type": "Microsoft.ContainerService/ManagedClusters",
  "properties": {
    "provisioningState": "Succeeded",
    "powerState": {
      "code": "Running"
    },
    "kubernetesVersion": "1.22.6",
    "dnsPrefix": "name",
    "fqdn": "name.hcp.norwayeast.azmk8s.io",
    "azurePortalFQDN": "name.portal.hcp.norwayeast.azmk8s.io",
    "agentPoolProfiles": [
      {
        "name": "system",
        "count": 3,
        "vmSize": "Standard_DS3_v2",
        "osDiskSizeGB": 100,
        "osDiskType": "Ephemeral",
        "kubeletDiskType": "OS",
        "vnetSubnetID": "/subscriptions/subid/resourceGroups/name/providers/Microsoft.Network/virtualNetworks/name/subnets/aks",
        "maxPods": 30,
        "type": "VirtualMachineScaleSets",
        "availabilityZones": [
          "1",
          "2",
          "3"
        ],
        "provisioningState": "Succeeded",
        "powerState": {
          "code": "Running"
        },
        "orchestratorVersion": "1.22.6",
        "mode": "System",
        "enableEncryptionAtHost": true,
        "osType": "Linux",
        "osSKU": "Ubuntu",
        "nodeImageVersion": "AKSUbuntu-1804gen2containerd-2022.02.15",
        "enableFIPS": false
      }
    ],
    "windowsProfile": {
      "adminUsername": "azureuser",
      "enableCSIProxy": true
    },
    "servicePrincipalProfile": {
      "clientId": "msi"
    },
    "addonProfiles": {
      "ingressApplicationGateway": {
        "enabled": true,
        "config": {
          "applicationGatewayId": "/subscriptions/subid/resourceGroups/name/providers/Microsoft.Network/applicationGateways/name",
          "effectiveApplicationGatewayId": "/subscriptions/subid/resourceGroups/name/providers/Microsoft.Network/applicationGateways/name"
        },
        "identity": {
          "resourceId": "/subscriptions/subid/resourcegroups/name-aks/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingressapplicationgateway-name",
          "clientId": "<id>",
          "objectId": "<id>"
        }
      },
      "omsagent": {
        "enabled": true,
        "config": {
          "logAnalyticsWorkspaceResourceID": "/subscriptions/<id>/resourceGroups/adp-devexp01-logs/providers/Microsoft.OperationalInsights/workspaces/adp-devexp01" 
        },
        "identity": {
          "resourceId": "/subscriptions/subid/resourcegroups/name-aks/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-name",
          "clientId": "<id>",
          "objectId": "<id>"
        }
      }
    },
    "nodeResourceGroup": "name-aks",
    "enableRBAC": true,
    "networkProfile": {
      "networkPlugin": "azure",
      "loadBalancerSku": "Standard",
      "loadBalancerProfile": {
        "managedOutboundIPs": {
          "count": 1
        },
        "effectiveOutboundIPs": [
          {
            "id": "/subscriptions/subid/resourceGroups/name-aks/providers/Microsoft.Network/publicIPAddresses/<id>"
          }
        ]
      },
      "serviceCidr": "10.0.0.0/16",
      "dnsServiceIP": "10.0.0.10",
      "dockerBridgeCidr": "172.17.0.1/16",
      "outboundType": "loadBalancer"
    },
    "aadProfile": {
      "managed": true,
      "adminGroupObjectIDs": null,
      "enableAzureRBAC": true,
      "tenantID": "<id>"
    },
    "maxAgentPools": 100,
    "identityProfile": {
      "kubeletidentity": {
        "resourceId": "/subscriptions/subid/resourcegroups/name-aks/providers/Microsoft.ManagedIdentity/userAssignedIdentities/name-agentpool",
        "clientId": "<id>",
        "objectId": "<id>"
      }
    },
    "autoUpgradeProfile": {
      "upgradeChannel": "node-image"
    },
    "podIdentityProfile": {}
  },
  "identity": {
    "type": "UserAssigned",
    "userAssignedIdentities": {
      "/subscriptions/subid/resourceGroups/name/providers/Microsoft.ManagedIdentity/userAssignedIdentities/name-aks": {        
        "clientId": "<id>",
        "principalId": "<id>"
      }
    }
  },
  "sku": {
    "name": "Basic",
    "tier": "Paid"
  }
}


DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
PUT

Absolute Uri:
https://management.azure.com/subscriptions/subid/resourceGroups/name/providers/Microsoft.ContainerService/managedClusters/name?api-version=2021-05-01

Headers:
x-ms-client-request-id        : <id>
Accept-Language               : en-US

Body:
{
  "properties": {
    "kubernetesVersion": "1.23.3",
    "dnsPrefix": "name",
    "agentPoolProfiles": [
      {
        "name": "system",
        "count": 3,
        "vmSize": "Standard_DS3_v2",
        "osDiskSizeGB": 100,
        "vnetSubnetID": "/subscriptions/subid/resourceGroups/name/providers/Microsoft.Network/virtualNetworks/name/subnets/aks",
        "maxPods": 30,
        "osType": "Linux",
        "type": "VirtualMachineScaleSets",
        "mode": "System",
        "orchestratorVersion": "1.22.6",
        "availabilityZones": [
          "1",
          "2",
          "3"
        ],
        "tags": {},
        "nodeLabels": {},
        "nodeTaints": []
      }
    ],
    "windowsProfile": {
      "adminUsername": "azureuser"
    },
    "servicePrincipalProfile": {
      "clientId": "msi"
    },
    "addonProfiles": {
      "ingressApplicationGateway": {
        "enabled": true,
        "config": {
          "applicationGatewayId": "/subscriptions/subid/resourceGroups/name/providers/Microsoft.Network/applicationGateways/name",
          "effectiveApplicationGatewayId": "/subscriptions/subid/resourceGroups/name/providers/Microsoft.Network/applicationGateways/name"
        }
      },
      "omsagent": {
        "enabled": true,
        "config": {
          "logAnalyticsWorkspaceResourceID": "/subscriptions/<id>/resourceGroups/adp-devexp01-logs/providers/Microsoft.OperationalInsights/workspaces/adp-devexp01" 
        }
      }
    },
    "nodeResourceGroup": "name-aks",
    "enableRBAC": true,
    "networkProfile": {
      "networkPlugin": "azure",
      "serviceCidr": "10.0.0.0/16",
      "dnsServiceIP": "10.0.0.10",
      "dockerBridgeCidr": "172.17.0.1/16",
      "loadBalancerSku": "Standard",
      "loadBalancerProfile": {
        "managedOutboundIPs": {
          "count": 1
        },
        "effectiveOutboundIPs": [
          {
            "id": "/subscriptions/subid/resourceGroups/name-aks/providers/Microsoft.Network/publicIPAddresses/<id>"
          }
        ]
      }
    },
    "aadProfile": {
      "adminGroupObjectIDs": [],
      "tenantID": "<id>"
    },
    "identityProfile": {
      "kubeletidentity": {
        "resourceId": "/subscriptions/subid/resourcegroups/name-aks/providers/Microsoft.ManagedIdentity/userAssignedIdentities/name-agentpool",
        "clientId": "<id>",
        "objectId": "<id>"
      }
    },
    "privateLinkResources": []
  },
  "identity": {
    "type": "UserAssigned",
    "userAssignedIdentities": {}
  },
  "location": "norwayeast",
  "tags": {}
}


DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
BadRequest

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-failure-cause            : gateway
x-ms-request-id               : <id>
x-ms-correlation-request-id   : <id>
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
Date                          : Mon, 07 Mar 2022 08:28:12 GMT

Body:
{
  "error": {
    "code": "MissingIdentityIds",
    "message": "The identity ids must not be null or empty for 'UserAssigned' identity type."
  }
}


Set-AzAksCluster: test.ps1:30:3
Line |
  30 |    Set-AzAksCluster -InputObject $cluster -KubernetesVersion $Kubernet …
     |    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The identity ids must not be null or empty for 'UserAssigned' identity type.

DEBUG: AzureQoSEvent: Module: Az.Aks:3.1.2; CommandName: Set-AzAksCluster; PSVersion: 7.2.1; IsSuccess: False; Duration: 00:00:02.5855748; Exception: The identity ids must not be null or empty for 'UserAssigned' identity type.;
DEBUG: Finish sending metric.
DEBUG: 10:28:15 - SetAzureRmAks end processing.

Environment data

Name                           Value
----                           -----
PSVersion                      7.2.1
PSEdition                      Core
GitCommitId                    7.2.1
OS                             Microsoft Windows 10.0.22000
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     2.7.3                 Az.Accounts                         {Add-AzEnvironment, Clear-AzContext, Clear-AzDefault, Connect-AzAccount…}
Script     3.1.2                 Az.Aks                              {Disable-AzAksAddOn, Enable-AzAksAddOn, Get-AzAksCluster, Get-AzAksNodePool…}

Error output

RequestId      : <id>
Message        : The identity ids must not be null or empty for 'UserAssigned' identity type.
ServerMessage  : MissingIdentityIds: The identity ids must not be null or empty for 'UserAssigned' identity type. (System.Collections.Generic.List`1[Microsoft.Rest.Azure.CloudError])
ServerResponse : {BadRequest}
RequestMessage : {PUT https://management.azure.com/subscriptions/<id>/resourceGroups/name/providers/Microsoft.ContainerService/managedClusters/name?api-version=2021-05-01}
InvocationInfo : {Set-AzAksCluster}
Line           :   Set-AzAksCluster -InputObject $cluster -KubernetesVersion $KubernetesVersion -ControlPlaneOnly:$controlPlaneOnly

Position       : At test.ps1:30 char:3
                 +   Set-AzAksCluster -InputObject $cluster -KubernetesVersion $Kubernet …
                 +   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
StackTrace     :    at Microsoft.Azure.Management.ContainerService.ManagedClustersOperations.BeginCreateOrUpdateWithHttpMessagesAsync(String resourceGroupName, String resourceName,
                 ManagedCluster parameters, Dictionary`2 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.ContainerService.ManagedClustersOperations.CreateOrUpdateWithHttpMessagesAsync(String resourceGroupName, String resourceName, ManagedCluster      
                 parameters, Dictionary`2 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.ContainerService.ManagedClustersOperationsExtensions.CreateOrUpdateAsync(IManagedClustersOperations operations, String resourceGroupName, String  
                 resourceName, ManagedCluster parameters, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.ContainerService.ManagedClustersOperationsExtensions.CreateOrUpdate(IManagedClustersOperations operations, String resourceGroupName, String       
                 resourceName, ManagedCluster parameters)
                    at Microsoft.Azure.Commands.Aks.SetAzureRmAks.<>c__DisplayClass28_0.<ExecuteCmdlet>b__0()
                    at Microsoft.Azure.Commands.Aks.KubeCmdletBase.RunCmdLet(Action action)
                    at Microsoft.Azure.Commands.Aks.SetAzureRmAks.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
HistoryId      : 6
@aelij aelij added bug This issue requires a change to an existing behavior in the product in order to be resolved. needs-triage This is a new issue that needs to be triaged to the appropriate team. labels Mar 7, 2022
@ghost ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Mar 7, 2022
@dingmeng-xue
Copy link
Member

Thanks for reporting. It is a bug and will be fixed.

@dingmeng-xue dingmeng-xue self-assigned this Mar 12, 2022
@wyunchi-ms
Copy link
Contributor

Hi @aelij , we are fixing this issue now. You can use Set-AzAksCluster -ResourceGroupName $resourceGroupName -Name $name -KubernetesVersion $version instead of Get-AzAksCluster -ResourceGroupName $resourceGroupName -Name $name | Set-AzAksCluster -KubernetesVersion $version as a workaround.

@wyunchi-ms wyunchi-ms linked a pull request Apr 12, 2022 that will close this issue
[Aks] Fixed the issue that identity cannot be piped into Set-AzAksCluster #17776
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wyunchi-ms wyunchi-ms

Labels
AKS bug This issue requires a change to an existing behavior in the product in order to be resolved. P1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Aks] Fixed the issue that identity cannot be piped into Set-AzAksCluster Azure/azure-powershell
3 participants
@aelij @wyunchi-ms @dingmeng-xue