Skip to content

test(cicd): ADO pipeline YAML test only covers the azd runner despite its name #395

Description

Problem

tests/unit/test_cicd.py::test_azure_devops_pipelines_stay_valid_yaml_for_every_eval_runner does not do what its name says.

The name promises coverage of every eval runner. The body only exercises the azd runner. The cloud and local execution modes never get an Azure DevOps pipeline generated in the test, so the YAML those branches emit is never parsed.

This was found during the critical review of #391.

Why it matters

#391 fixed a real bug: the eval gate installed azd but never gave it credentials, on both GitHub Actions and Azure DevOps. That bug was invisible partly because the ADO generation path has thin coverage. A test whose name claims full runner coverage but silently covers one runner is worse than no test, because it suppresses the instinct to add one.

Proposed fix

Parametrize the test over the three execution modes (azd, cloud, local) and assert for each:

  1. The generated ADO pipeline is valid YAML.
  2. When the mode needs Azure credentials, the pipeline actually contains the credential step.
  3. When the mode does not need azd, the pipeline does not install it.

If some modes genuinely cannot produce an ADO pipeline today, make that explicit with an assertion that the generator refuses, rather than leaving the case untested.

Acceptance criteria

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions