[BUG]: Received Error 7000222 After Executing a Release Pipeline Using Kubernetes Service Connection Type

[sohlae]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

HelmDeploy

Task version

0

Issue Description

I currently have a pipeline that uses a Kubernetes service connection in Azure DevOps. This service connection has been created using the Azure Subscription authentication method.

2 days ago the client secret has expired which has caused the pipeline to fail and return the error message below.

Error: Failed to fetch credentials for the cluster l***1. Reason Could not fetch access token for Azure. Status code: invalid_client, status message: Error(s): 7000222 - Timestamp: 2025-06-26 00:10:02Z - Description: AADSTS7000222: The provided client secret keys for app '***' are expired.

After deleting the expired client secret and generating a new one from the App Registrations page in the Azure Portal, we reran the pipeline hoping that it will use the newly-generated secret, however another error was returned.

Error(s): 7000215 - Timestamp: 2025-06-26 13:29:29Z - Description: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '***'.

After hours of investigating the problem, we were able to identify what was causing the issue. See below for the details of our investigation.

Investigation

Our pipeline is using a task called HelmDeploy@0 to deploy a Docker image to an AKS instance in Azure. This is just one of the steps and the one that performs the actual deploy (helm install) is also using this task.

- task: HelmDeploy@0
     displayName: 'Helm add stable repo'
     inputs:
       azureSubscriptionEndpoint: $(azureSubscriptionEndpointPrd) # Value is Subscription Name (Subscription ID)
       azureResourceGroup: $(azureResourceGroupPrd)
       kubernetesCluster: $(kubernetesClusterPrd)
       command: repo
       arguments: 'add stable https://charts.helm.sh/stable --debug' 

When we changed the task to use kubernetesServiceConnection instead of azureSubscriptionEndpoint it worked, that is, it was able to use the same service connection and using the newly-generated client secret resulting in the pipeline completing successfully.

- task: HelmDeploy@0
     displayName: 'Helm add stable repo'
     inputs:
       connectionType: 'Kubernetes Service Connection'
       kubernetesServiceConnection: 'name-of-the-service-connection'
       command: repo
       arguments: 'add stable https://charts.helm.sh/stable --debug'

My questions right now are

1. Why is it that when we used azureSubscriptionEndpoint, it continued to use the old client secret? Is there an internal logic in HelmDeploy@0 that uses a cached secret whenever this field is used?
2. How did using kubernetesServiceConnection solve the problem when it essentially does the same thing as azureSubscriptionEndpoint?

Any guidance on this topic would be much appreciated!

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu 22.04

Relevant log output

##[error]Error: Failed to fetch credentials for the cluster name-of-cluster. Reason Could not fetch access token for Azure. Status code: invalid_client, status message: Error(s): 7000215 - Timestamp: 2025-06-26 16:48:59Z - Description: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '***'. Trace ID: 55f0a02e-de17-4dab-8640-e8984b031600 Correlation ID: 5bf99f85-1a5a-414c-bd70-55dfc7705883 Timestamp: 2025-06-26 16:48:59Z - Correlation ID: 5bf99f85-1a5a-414c-bd70-55dfc7705883 - Trace ID: 55f0a02e-de17-4dab-8640-e8984b031600. Make sure the service connection is assigned an appropriate Azure RBAC role: https://aka.ms/azdo-aks-rm-access.

Full task logs with system.debug enabled

 ##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Verbose - Canonical Authority: https://login.microsoftonline.com/***/
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Verbose - Did not find cloud discovery metadata in the config... Attempting to get cloud discovery metadata from the hardcoded values.
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Verbose - Found cloud discovery metadata from hardcoded values.
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Returning result from authorityUpdateCloudDiscoveryMetadata
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Executing function authorityUpdateEndpointMetadata
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Verbose - Attempting to get endpoint metadata from authority configuration
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Verbose - Did not find endpoint metadata in the config... Attempting to get endpoint metadata from the hardcoded values.
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Returning result from authorityUpdateEndpointMetadata
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Item key: authority-metadata-***-login.windows.net
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Getting cache key-value store
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Setting cache key value store
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Returning result from authorityResolveEndpointsAsync
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Verbose - Replacing tenant domain name *** with id {tenantid}
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [cfa75612-4d37-4091-b72d-e6b19396e86c] : @azure/msal-node@2.10.0 : Info - Building oauth client configuration with the following authority: https://login.microsoftonline.com/***/oauth2/v2.0/token.
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [cfa75612-4d37-4091-b72d-e6b19396e86c] : @azure/msal-node@2.10.0 : Verbose - Client credential client created
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Getting in-memory cache
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Getting cache key-value store
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Verbose - Replacing tenant domain name *** with id {tenantid}
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Item key: server-telemetry-***
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Getting cache key-value store
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Verbose - Replacing tenant domain name *** with id {tenantid}
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [cfa75612-4d37-4091-b72d-e6b19396e86c] : @azure/msal-common@14.13.0 : Info - Sending token request to endpoint: https://login.microsoftonline.com/***/oauth2/v2.0/token
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Item key: throttling.{"clientId":"***","authority":"[https://login.microsoftonline.com/***","scopes":["https://management.azure.com//.default"],"authenticationScheme":"Bearer"}](https://login.microsoftonline.com/***%22,%22scopes%22:[%22https://management.azure.com//.default%22],%22authenticationScheme%22:%22Bearer%22%7D)
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Getting cache key-value store
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Item key: server-telemetry-***
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Getting cache key-value store
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Item key: server-telemetry-***
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Getting cache key-value store
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Setting cache key value store
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Item key: server-telemetry-***
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Getting cache key-value store
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Item key: server-telemetry-***
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Getting cache key-value store
##[debug][Thu, 26 Jun 2025 16:48:59 GMT] : [] : @azure/msal-node@2.10.0 : Trace - Setting cache key value store
##[debug]task result: Failed
##[error]Error: Failed to fetch credentials for the cluster ls-ms-core-prd-ne-aks-01. Reason Could not fetch access token for Azure. Status code: invalid_client, status message: Error(s): 7000215 - Timestamp: 2025-06-26 16:48:59Z - Description: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '***'. Trace ID: 55f0a02e-de17-4dab-8640-e8984b031600 Correlation ID: 5bf99f85-1a5a-414c-bd70-55dfc7705883 Timestamp: 2025-06-26 16:48:59Z - Correlation ID: 5bf99f85-1a5a-414c-bd70-55dfc7705883 - Trace ID: 55f0a02e-de17-4dab-8640-e8984b031600. Make sure the service connection is assigned an appropriate Azure RBAC role: https://aka.ms/azdo-aks-rm-access.
##[debug]Processed: ##vso[task.issue type=error;source=TaskInternal;correlationId=e6fbbbc4-4ea7-4fd3-9ff1-386c83d127f3;]Error: Failed to fetch credentials for the cluster ls-ms-core-prd-ne-aks-01. Reason Could not fetch access token for Azure. Status code: invalid_client, status message: Error(s): 7000215 - Timestamp: 2025-06-26 16:48:59Z - Description: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '***'. Trace ID: 55f0a02e-de17-4dab-8640-e8984b031600 Correlation ID: 5bf99f85-1a5a-414c-bd70-55dfc7705883 Timestamp: 2025-06-26 16:48:59Z - Correlation ID: 5bf99f85-1a5a-414c-bd70-55dfc7705883 - Trace ID: 55f0a02e-de17-4dab-8640-e8984b031600. Make sure the service connection is assigned an appropriate Azure RBAC role: https://aka.ms/azdo-aks-rm-access.
##[debug]Processed: ##vso[task.complete result=Failed;]Error: Failed to fetch credentials for the cluster ls-ms-core-prd-ne-aks-01. Reason Could not fetch access token for Azure. Status code: invalid_client, status message: Error(s): 7000215 - Timestamp: 2025-06-26 16:48:59Z - Description: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '***'. Trace ID: 55f0a02e-de17-4dab-8640-e8984b031600 Correlation ID: 5bf99f85-1a5a-414c-bd70-55dfc7705883 Timestamp: 2025-06-26 16:48:59Z - Correlation ID: 5bf99f85-1a5a-414c-bd70-55dfc7705883 - Trace ID: 55f0a02e-de17-4dab-8640-e8984b031600. Make sure the service connection is assigned an appropriate Azure RBAC role: https://aka.ms/azdo-aks-rm-access.
Finishing: Helm add stable repo 

Repro steps

trigger:
  branches:
    include:
    - none

pool:
  vmImage: 'ubuntu-22.04'

variables:
- group: Business-Prd

stages:
- stage: Docker_Build
  # Omitted for brevity...

- stage: Prd_Deploy_To_AKS
  displayName: Prd - Deploy to AKS
  dependsOn:
  - Prd_Docker_To_ACR
  jobs:
  - deployment: deployToAKS
    displayName: Deploy to AKS
    environment:
      name: $(deploymentEnvironmentPrd)
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self  # Garantir que o código seja clonado

          - task: HelmInstaller@1
            displayName: 'Install Helm version'
            inputs:
              helmVersionToInstall: latest

          - task: HelmDeploy@0
            displayName: 'Helm add stable repo'
            inputs:
              azureSubscriptionEndpoint: $(azureSubscriptionEndpointPrd)
              azureResourceGroup: $(azureResourceGroupPrd)
              kubernetesCluster: $(kubernetesClusterPrd)
              command: repo
              arguments: 'add stable https://charts.helm.sh/stable --debug'

[EshitaD]

Hi @sohlae, have you updated your secret in the ARM service connection after generating the new secret?