RunPipelineParameters: template_parameters includes mandatory parameters but their values are not found; incorrectly raises AzureDevOpsServiceError

[detmoo]

Intro

I am calling the run_pipeline method for the pipelines_client of version 7.1.

The Pipeline I am targeting includes some mandatory parameters. And these are being included within the RunPipelineParameters object that is passed to the client method. However, for certain of these mandatory parameters, their values are not found and AzureDevopsServiceError is raised (instead of beginning the pipeline run).

Please can you inspect the following evidence and advise or investigate further.

Minimal Repro

Reproducing the error assumes you have a pipeline available. In this case the pipeline_id = "168", but will vary according to your organisation, etc. The following repro demonstrates that the AzureDevopsServiceError which gets raised is not logically consistent with the template_parameters that were input, and as a result a straight Exception is raised instead (with the anomalous parameters included in that error message). The repro expects a valid PAT to be set in your environment.

import os

from azure.devops.connection import Connection
from azure.devops.exceptions import AzureDevOpsServiceError
from azure.devops.v7_1.pipelines import RunPipelineParameters
from msrest.authentication import BasicAuthentication


# settings according to your organisation
_client = 'clients_v7_1.get_pipelines_client'
base_url = "https://dev.azure.com/myOrganisation"
azdo_project_name = "myProject"
pipeline_id = "168"


def get_auth():
    """Returns the credentials object using PAT or CLI."""
    auth = None
    pat = os.environ.get('AZURE_DEVOPS_PAT', None)
    if pat:
        auth = BasicAuthentication('', pat)
    else:
        raise Exception("auth skipped because AZURE_DEVOPS_PAT not set")
    return auth


# derive the correct service client as set as a callable object
client_getter = Connection(base_url=base_url, creds=get_auth())
for stub in _client.split('.'):
    client_getter = getattr(client_getter, stub)
client = client_getter()


# these are the parameters unique to my pipeline (including all mandatory parameters)
template_parameters = {'datacentreId': '38bda748-e055-493b-89ee-4c1b9adfa6aa',
                       'serverId': '71b18772-e389-4d25-9355-9bc90dc1ed8e',
                       'nicId': 'a7374982-b992-402e-9c9f-dcfa46a3ad43',
                       'serviceConnection': '3232_AnsibleRoleTestDatacentre_DEV.PoolServer_x',
                       'targetIps': 'xxxxxx',
                       'hostName': 'AnsibleRoleTestDatacentre_DEV.PoolServer_x',
                       'ansibleControllerUser': 'xxxxxx',
                       'ansibleControllerHost': 'DevOpsTestDatacentreDEV.BD-Build-Agent',
                       'ansibleControllerIp': 'xxxxxx'
                       }

# we create the RunPipelineParameters instance
pipeline_parameters = RunPipelineParameters(template_parameters=template_parameters)

# try to run the pipeline and manage the exception
try:
    run = client.run_pipeline(pipeline_parameters, azdo_project_name, pipeline_id)
except AzureDevOpsServiceError as e:
    print("This is the erroneous error message:\n\n", e, "\n")
    falseErrors = []
    for key in template_parameters.keys():
        if f"A value for the '{key}' parameter must be provided." in e.message:
            falseErrors.append(key)
    if falseErrors:
        raise Exception(f"These parameters were passed as template_parameters but values were not found: {falseErrors}")

Output

Following is the output of the minimal repro. As shown in the final error message, some parameters were passed but not found, and so AzureDevopsServiceError raised falsely.

Traceback (most recent call last):
  File "C:\Users\Oliver\AppData\Roaming\JetBrains\PyCharmCE2021.2\scratches\run_pipeline_fail_repro.py", line 46, in <module>
    run = client.run_pipeline(pipeline_parameters, azdo_project_name, pipeline_id)
  File "C:\Users\Oliver\repos\bowdata\bowdata.toolkits.azdo\.venv\lib\site-packages\azure\devops\v7_1\pipelines\pipelines_client.py", line 263, in run_pipeline
    response = self._send(http_method='POST',
  File "C:\Users\Oliver\repos\bowdata\bowdata.toolkits.azdo\.venv\lib\site-packages\azure\devops\client.py", line 107, in _send
    response = self._send_request(request=request, headers=headers, content=content, media_type=media_type)
  File "C:\Users\Oliver\repos\bowdata\bowdata.toolkits.azdo\.venv\lib\site-packages\azure\devops\client.py", line 71, in _send_request
    self._handle_error(request, response)
  File "C:\Users\Oliver\repos\bowdata\bowdata.toolkits.azdo\.venv\lib\site-packages\azure\devops\client.py", line 273, in _handle_error
    raise AzureDevOpsServiceError(wrapped_exception)
azure.devops.exceptions.AzureDevOpsServiceError: A value for the 'datacentreId' parameter must be provided.
A value for the 'serverId' parameter must be provided.
A value for the 'nicId' parameter must be provided.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Oliver\AppData\Roaming\JetBrains\PyCharmCE2021.2\scratches\run_pipeline_fail_repro.py", line 54, in <module>
    raise Exception(f"These parameters were passed as template_parameters but values were not found: {falseErrors}")
Exception: These parameters were passed as template_parameters but values were not found: ['datacentreId', 'serverId', 'nicId']

[detmoo]

On attempting the run the pipeline with the same parameters via the GUI, it is clear the pipeline YAML itself was malformed, and there is as such no issue with the azure-devops-python-api.

The error message seen when attempting to run via the GUI does match the error message returned by the azure-devops-python-api _handle_error method. It is arguably not a good error message, but again, no issue with azure-devops-python-api and hence closing the issue.