Perform unit/integration test for pipelines (Azure DevOps)
Unit testing CI/CD pipelines is a challenge. Test frameworks for pipelines are almost non-existent or at least very scarce. Teams often develop pipelines using trial-and-error and they test along the way. Some of the challenges are:
- During testing the pipeline, a wrong version of an app was deployed by accident.
- Code from a feature branch was accidentally tagged with a release version tag.
- You need to wait for 15 minutes before the last step you just changed is executed.
- The number of commits is very high because of the trial-and-error nature of developing and testing pipelines.
- Temporary code, added to the pipeline specifically for testing is not removed.
- Temporary disabled or commented code is not enabled / uncommented anymore.
- The pipeline code contains switches or conditions specifically for testing the pipeline.
- The overview with regular application pipeline runs is cluttered with a zillion test runs.
- Asserts are difficult to incorporate in a pipeline, and if possible, they decrease readability.
This library is used to perform unit- and integration tests on (YAML) pipelines. At the moment, only
Azure DevOps pipelines are supported.
Assume that your application and pipeline code reside in a repository called "myrepo" in the Azure DevOps project "MyApp".
Development on the (Java) app is straightforward.
With the junit-pipeline libray, it becomes possible to test the YAML pipeline code.
Testing the pipeline code is performed by JUnit tests. In these tests, the original pipeline code is manipulated according to your needs.
Assume, you want to mock your deployment task to prevent something to be deployed to your AWS account. With a few lines, the "AWSShellScript@1"
task is replaced by a script. Example code:
String inlineScript = "echo \"This is a mock script\"\n" +
"echo \"Mock-and-roll\"";
pipeline.mockStepSearchByIdentifier("AWSShellScript@1", inlineScript);To test the manipulated script, execute it like this:
pipeline.startPipeline();
The junit-pipeline library connects with the Azure DevOps test project (in the figure below, represented by "UnitTest"), and pushes the code to your test repository ("myrepo-test", in this example), after which the pipeline is executed. If the repository and/or the pipeline in the test project do not exists, they are automatically created for you. The illustration below shows how it works in concept.
In addition, all external repositories defined in the pipeline are cloned and also pushed to the Azure DevOps test project. External repositories are
used to define pipeline templates that are included in your pipeline, but reside in a different repository than the pipeline itself.
The junit-pipeline library takes care that the main pipeline refers to the
cloned copies of these repositories instead to the original ones, so external templates can also be manipulated.
Unfortunately, testing a pipeline within the IDE is not possible. You need an Azure DevOps unit test project for this. Create a test project
using this link: Create a project in Azure DevOps
The properties file is located in src/main/resources. It contains the properties for your project. Some important properties:
- source.path - Contains the location (directory) of the main Git repository on your computer. This is the repository in which you develop your app and the associated pipeline. In the example case, this repository is called "myrepo".
- target.path - Contains the location (directory) of the Git repository used to test the pipeline. You are not actively working in this repo. It is only used for the junit-pipeline framework to communicate with the Azure DevOps test project. Before you start, this directory must not exist.
- target.organization - The name of your organization as defined in Azure DevOps. This will be included in the Azure DevOps API calls.
- target.project - The name of the test project. In the example case it is called "UnitTest".
- source.base.path.external - The local directory of the external repositories on the workstation, defined in the "repositories" section of the main pipeline YAML file. You are not actively working in this repo.
- target.base.path.external - The location (local directory) containing external Git repositories; this location is used communicate with the Azure DevOps test project.
- source.repository.name - The name of the main repository
- target.repository.name - The name of the repository used in the Git repository used for testing. Example: If a source repository with the name "myrepo" is used, the target.repository.name used for testing the pipeline can be called "myrepo-test". target.repository.name should preferably not be equal to source.repository.name.
- azdo.user - User used in the Azure DevOps API calls. Can be the default name 'UserWithToken'.
- azdo.pat - The PAT (Personal Access Token) used in the Azure DevOps API calls.
See Use personal access tokens how to create a PAT.
Make sure this PAT is authorized to clone repositories in other Azure DevOps projects in the organization (other than the test project). - git.commit.pattern - Defines the type of files pushed to the local- and remote test repo (this is a subset of the files from the main repo)
- pipelines.api - Name of the Azure DevOps base Pipeline API; do not change this value.
- pipelines.api.runs - Name of a specific Azure DevOps Pipeline API; do not change this value.
- pipelines.api.version - Version of the Azure DevOps Pipeline API; only change if it is really needed (e.g., if a new version of the API is released).
- git.api - Name of the Azure DevOps base Git API; do not change this value.
- git.api.repositories - Name of a specific Azure DevOps Git API; do not change this value.
- git.api.version - Version of the Azure DevOps Git API; only change if it is really needed (e.g., if a new version of the API is released).
- build.api - Name of the Azure DevOps base Build API; do not change this value.
- build.api.version - Version of the Azure DevOps Build API; only change if it is really needed (e.g., if a new version of the API is released).
- build.api.poll.frequency - The result of a pipeline run is retrieved, using an Azure DevOps API. This API is called with a frequency determined by build.api.poll.frequency (in seconds).
- build.api.poll.timeout - The timeout value of polling the result of the pipeline run. If the final result is not retrieved yet, the polling stops after a number of seconds, defined by build.api.poll.timeout.
- project.api - Name of the Azure DevOps base Project API; do not change this value.
- project.api.version - Version of the Azure DevOps Project API; only change if it is really needed (e.g., if a new version of the API is released).
- variable.groups.api - The Azure DevOps API used to retrieve the list of variable groups in the project.
- variable.groups.api.version - Version of this API.
- variable.groups.validate - Defines whether the variable groups in the YAML files must be validated. Default is true.
- environments.api - The Azure DevOps API used to retrieve the list of environments in the project.
- environments.api.version - Version of this API.
- environments.validate - Defines whether the environments in the YAML files must be validated. Default is true.
- error.continue - If true, the junit-.pipeline framework continues after an error is detected (e.g., if the pipeline YAML file or a template file is incorrect). Note, that this can result in unpredictable results. If false, the framework stops with the test as soon as an error is detected.
- templates.external.include - If true (= default), templates of other repositories are als included. This means that if a method is executed, for example to skip a step, this also applies to these templates. If set to false, the templates in other repositories are just as-is.
The property file is stored in the resources folder.
After the properties file has been created, the junit-pipeline library must be added to the pom.xml of your project. Example:
<dependency>
<groupId>io.github.hvmerode</groupId>
<artifactId>junit-pipeline</artifactId>
<version>1.2.13</version>
</dependency>This repository already contains a sample unit test file called PipelineUnit.java. We take this file as an example.
If you want to check out a demo project, please take a look at 'hello-pipeline'.
Pipeline unit tests are defined in a unit test Java class. Before tests are executed, a new AzDoPipeline Java object must be instantiated. Its constructor requires two arguments, a property file and the main pipeline YAML file. Example:
AzDoPipeline pipeline = new AzDoPipeline("junit_pipeline_my.properties", "./pipeline/pipeline_test.yml");The junit_pipeline_my.properties file in this example contains my personal properties, but you can use
the junit_pipeline.properties file in the resources folder and customize it to your needs.
The file ./pipeline/pipeline_test.yml is the main pipeline file. It can be stored in any folder of the code repository.
Its path is relative to the root of the repository. The main pipeline file may contain references to other yamlTemplate files
in the repository. The junit-pipeline frameworks takes these templates into account in pipeline manipulation.
Before the pipeline code is pushed to the Azure DevOps unit test project, and started, it is possible to execute
custom code. This code is provided as a list of 'hooks'. The unit test file PipelineUnit.java shows an example; test 3.
This repository also contains a few standard hooks:
- DeleteJUnitPipelineDependency - Deletes the junit-pipeline dependency from the pom.xml, before it is pushed to the Azure DevOps unit test project.
- DeleteTargetFile - Deletes a single file before it is pushed to the Azure DevOps unit test project. It can be used to remove the file that includes the pipeline unit tests, if you don't want it to run it in the test project.
- FindReplaceInFile - Find and replace a string in a given file; either replaces the first occurence or all occurences.
This hook can be used to fix some inconveniences in the target yaml files in the Azure DevOps test project.
The junit-pipeline library contains a set of methods - used in unit tests - to manipulate the pipeline. Let's go over a few of them:
Note, that this is only a subset of the methods available.
public void mockStepSearchByIdentifier (String stepIdentifier, String inlineScript)Example:
- task: AWSShellScript@1
inputs:
awsCredentials: $(aws_connection)
regionName: $(aws_region)
scriptType: 'inline'
inlineScript: |
#!/bin/bash
set -ex
export cdk=`find $(Pipeline.Workspace)/. -name 'cdk*.jar'`
export app=`find $(Pipeline.Workspace)/. -name 'app*.jar'`
echo "Deploying stack"
cdk deploy --app '${JAVA_HOME_11_X64}/bin/java -cp $cdk com.org.app.Stack' \
-c env=${{ parameters.environment }} \
-c app=$app
--all \
--ci \
--require-approval never
displayName: 'Deploy to AWS'
Calling in Java:
String inlineScript = "echo \"This is a mock script\"\n" + "echo \"Mock-and-roll\"";
pipeline.mockStepSearchByIdentifier ("AWSShellScript@1", inlineScript);results in:
- script: |-
echo "This is a mock script"
echo "This is line 2"
public void skipStageSearchByIdentifier (String stageIdentifier)Skip a stage. The result is, that the stage is completely removed from the output pipeline yaml file, which basically is the same as skipping it.
Example:
- stage: my_stage displayName: 'This is my stage'
Calling in Java:
pipeline.skipStageSearchByIdentifier ("my_stage")==> The stage with identifier "my_stage" is skipped
public void skipJobSearchByIdentifier (String jobIdentifier)Example:
- job: my_job displayName: 'This is my job'
Calling in Java:
pipeline.skipJobSearchByIdentifier ("my_job")==> The job with identifier "my_job" is skipped
public void skipTemplateSearchByIdentifier (String templateIdentifier)Example:
- template: templates/stages/template-stages.yml
parameters:
parameter_1: value_1
parameter_2: value_2
Calling in Java:
pipeline.skipTemplateSearchByIdentifier ("templates/stages/template-stages.yml")==> The template with identifier "templates/stages/template-stages.yml" is skipped
public void overrideVariable (String variableName, String value)variables: - name: myVar value: myValue
Calling in Java:
pipeline.overrideVariable ("myVar", "myNewValue")results in:
variables: - name: myVar value: myNewValue
This method does not replace variables defined in a Library (variable group).
public void overrideTemplateParameter (String parameterName, String value)Example:
- yamlTemplate: step/mytemplate.yml
parameters:
tag: $(version)
To replace the version with a fixed value (2.1.0), call:
pipeline.overrideTemplateParameter ("tag", "2.1.0"). This results in:
- yamlTemplate: step/mytemplate.yml
parameters:
tag: 2.1.0
public void overrideParameterDefault (String parameterName, String defaultValue)Example:
- name: myNumber type: number default: 2 values: - 1 - 2 - 4
Calling in Java:
pipeline.overrideParameterDefault ("myNumber", "4") results in:
- name: myNumber type: number default: 4 values: - 1 - 2 - 4
public void overrideLiteral (String literalToReplace, String newValue, boolean replaceAll)Example:
- task: AzureWebApp@1
displayName: Azure Web App Deploy
inputs:
azureSubscription: $(azureSubscription)
appName: samplewebapp
Calling in Java:
pipeline.overrideLiteral ("$(azureSubscription)", "1234567890") results in
- task: AzureWebApp@1
displayName: Azure Web App Deploy
inputs:
azureSubscription: 1234567890
appName: samplewebapp
If replaceAll is 'true' all occurences of literal in both the main YAML and the templates are replaced.
If replaceAll is 'false' the first occurence of literal in both the main YAML and the templates are replaced.
public void overrideCurrentBranch (String newBranchName, boolean replaceAll)Example:
Assume the following condition:
and(succeeded(), eq(variables['Build.SourceBranchName'], 'main'))
After applying
pipeline.overrideCurrentBranch ("myFeature")it becomes:
and(succeeded(), eq('myFeature', 'main'))
If replaceAll is 'true', all occurences in both the main YAML and the templates are replaced.
If replaceAll is 'false', the first occurence in both the main YAML and the templates are replaced.
public void setVariableSearchStepByDisplayName (String displayValue,
String variableName,
String value,
boolean insertBefore)Example:
- task: AzureRMWebAppDeployment@4
displayName: Azure App Service Deploy
inputs:
appType: webAppContainer
ConnectedServiceName: $(azureSubscriptionEndpoint)
WebAppName: $(WebAppName)
DockerNamespace: $(DockerNamespace)
DockerRepository: $(DockerRepository)
DockerImageTag: $(Build.BuildId)
After applying
pipeline.setVariableSearchStepByDisplayName ("Azure App Service Deploy", "WebAppName", "newName")a script is inserted just before the AzureRMWebAppDeployment@4 (note, that 'insertBefore' is omitted; default is 'true'). When running the pipeline, the value of "WebAppName" is set with the value "newName"
pwsh: 'Write-Host "echo ##vso[task.setvariable variable=WebAppName]newName"'
In addition to the setVariableSearchStepByDisplayName, the methods setVariableSearchStepByIdentifier and
setVariableSearchTemplateByIdentifier are supported.
public void assertVariableEqualsSearchStepByDisplayName (String displayValue,
String variableName,
String compareValue,
boolean insertBefore)Example:
After calling
pipeline.assertVariableEqualsSearchStepByDisplayName ("Deploy the app", "myVar", "123")the value of variable 'myVar' is compared with '123', just before the step with displayName
"Deploy the app" is executed. If you want to validate just after execution of the step, call
assertVariableEqualsSearchStepByDisplayName ("Deploy the app", "myVar", "123", false).
public void assertFileExistsSearchStepByDisplayName (String displayValue,
String fileName,
boolean insertBefore)public void mockPowerShellCommandSearchStepByDisplayName (String displayValue,
String command,
String[] commandOutput)The commandOutput argument contains the return value of the mocked command. This is a json string. This method actually has two signatures. One, in which the commandOutput is a String, and one in which the commandOutput is an array of Strings. In the latter case, this is used if a PowerShell script contains multiple instances of the same command. By using an array of Strings, each command has a different return value.
In junit_pipeline version 1.1.5. only the 'Invoke-RestMethod' is supported
Example:
- pwsh: |
$Url = "https://server.contoso.com:8089/services/search/people/export"
$result = Invoke-RestMethod -Method 'Post' -Uri $url -OutFile output.csv
Write-Output "Result: $($result.element)"
displayName: 'Invoke-RestMethod step 2 of 2'
When calling
pipeline.mockPowerShellCommandSearchStepByDisplayName("Invoke-RestMethod step 2 of 2",
"Invoke-RestMethod",
"{\"element\" : \"value_1\"}");the pre-processor inserts a new "pwsh" script before the "pwsh" script with 'displayName' "Invoke-RestMethod step 2 of 2", containing code to mock the 'Invoke-RestMethod', and adds a reference to this code in the script "Invoke-RestMethod step 2 of 2". When executing the pipeline, the 'Invoke-RestMethod' returns the json, specified in the mockPowerShellCommandSearchStepByDisplayName() method.
The mockPowerShellCommandSearchStepByDisplayName() method works with both "pwsh" scripts and "PowerShell@2" tasks.
Note, that the junit-pipeline library also contains a Bash version of this method, called mockBashCommandSearchStepByDisplayName(), which works with "script", "bash", and "Bash@3" tasks. It mocks the Bash commands:
- curl
- wget
- ftp
public AzDoPipeline insertTemplateSearchSectionByDisplayName (String sectionType,
String displayValue,
String templateIdentifier,
Map<String, String> parameters,
boolean insertBefore)The startPipeline method has a few representations:
- startPipeline() - Starts the pipeline with the default branch (in most cases, this is the master branch).
- startPipeline(String branchName) - Starts the pipeline with a given branch, for example a feature branch.
- startPipeline(String branchName, List hooks) - Starts the pipeline with a given branch but before the pipeline starts, the list with 'hooks' is executed.
- startPipeline(String branchName, List hooks, boolean dryRun) - Performs all actions but does not start the pipeline in Azure DevOps. Use this boolean to minimize the exexution time (A free Azure DevOps account includes 1 Microsoft-hosted job with 1,800 minutes per month).
The result of a pipeline run is retrieved using:
pipeline.getRunResult();It is also possible to retrieve the run result of each individual stage, job, or step:
pipeline.getRunResult().getStageResultSearchByName("simpleStage");- Tests cannot be executed in parallel. Because the target repository is updated for each test, the next test must wait before the previous one is completed.
- Templates residing in external repositories (GitHub and other Azure DevOps projects) are taken into account, but:
- The ref parameter is not (yet) fully implemented. Only the format "refs/heads/branch" is supported; the pattern "refs/tags/tag" is not yet supported .
- If a remote external repository is updated, the update is not automatically included in the test; first delete the corresponding local directory; this enables the creation of q new clone of the external repository. For example, if an external repository is called 'Templates', 2 local directories are created, 'Templates' and 'Templates-source'; delete them both.
- An external GitHub repository is assumed to be public; no credentials are used to access the GitHub repository.
- If the pipeline makes use of a resource in the test project for the first time, it needs manual approval first; for example, a variable group or an Environment. The Azure DevOps API returns an HTTP status 400.
- If unknown service connections are used, if the updated pipeline code is not valid YAML anymore, or if a manual approval
of a resource is required, the AzDo API returns an HTTP status code 400.
- An Azure DevOps "on..failure" / "on..success" construction is translated to "true..failure" / "true..success". It may be an issue in snakeyaml.
- Temporary fix is replacing the literal "true:" with "'on:'" (with single quotes)
- A task with an input parameter 'template:' is handled as if it is a yamlTemplate (although it isn't); processing
is still fine though (gives a warning), but it should not be treated as a yamlTemplate. Alternative is to change the
warning and give the recommendation that, although it is correct, it may lead to confusion.
- In case of a build error, try to determine the details of the error. Check:
- Whether all service endpoints in the yaml files exist, using https://dev.azure.com/mycorp-com/UnitTest/_apis/serviceendpoint/endpoints?endpointNames=endpointName1,endpointName2,endpointName3 or https://dev.azure.com/mycorp-com/UnitTest//_apis/serviceendpoint/endpoints?api-version=7.1-preview.4
- Whether an approval is pending: See https://learn.microsoft.com/en-us/rest/api/azure/devops/approvalsandchecks/approvals/query?view=azure-devops-rest-7.2&tabs=HTTP ???
- Whether a resource exist (how)
- Add a custom condition to a stage, job, or step
- Test on Linux; some filesystem methods in Utils may not work properly.
- Support "refs/tags/tag" and "refs/refname" for external repositories with templates.
- Skip stage, job, or step deletes it from the yaml file. Add alternative to skip it but still have it visible in the pipeline (e.g. add condition: eq(true,false)).
- Look into the usage of CD events (https://cdevents.dev/ and https://github.com/cdevents/spec)
- Dynamically create service connections, which refer to the locally running HTTP server.
- Clone variable group from original Azure DevOps project into the Azure DevOps test project.
- Log YAML line numbers in method Utils.validatePipelineFile() according to yaml-line-numbers.md
- Add option to continue on error for all steps.
- Possibility to replace a step with another step.
- Add unit tests to the junit-pipeline code itself.
- Add methods to add, update or remove conditions in stages or jobs. Use the overrideLiteral method, if possible.
- Check whether the output pipeline is a valid pipeline (valid yaml and valid Azure DevOps pipeline). This is a 'nice-to-have'.
Provide PropertyUtils reference to the AzDoPipeline constructor; this is used to deviate from the properties fileMake manipulation of external template repos optionalIn case of a build error, try to determine the details of the error. Check:Validate whether the variable groups used are valid: See https://dev.azure.com/mycorp-com/UnitTest/_apis/distributedtask/variablegroupsValidate whether the environments used are valid, using: https://dev.azure.com/mycorp-com/UnitTest/_apis/distributedtask/environments
Implement method setVariableSearchStepByIdentifier- ${{ if }} construction in stages or jobs gives a validation errorPossibility to replace a step with a yamlTemplate file (the yamlTemplate file could serve as a mock file).Make the task inserted by assertFileExistsSearchStepByDisplayName of type 'pwsh', because it can run on both Windows and Linux.Add links to the timeline log to open the pipeline in a browser.'Assert variable' adds a task with a condition. Result is skipped (true) or failed (false). For clearness reasons this must be succeeded (true) or failed (false). Make sure that the task works on both Linux and Windows.Publish pipeline unit test report. Make use of the timeline api (https://learn.microsoft.com/en-us/rest/api/azure/devops/build/timeline/get?view=azure-devops-rest-5.1) Report contains:- Which stages, jobs, and tasks are executed for a certain pipeline run?- How long did each stage, job, and task run?- What is the status of each stage, job, and task?- The report must be created locally by the junit-plugin, showed in the log and/or created as HTML file (optional).Method constructAssertStep() needs rework. The text is incorrect in case of AssertNotEmpy, for example "AssertNotEmpty: variable 'releaseVersion' with value '' is not equal to ''".Reset trigger to none; this prevents that pipelines are executed twice; one time because the repo is updated and one time because it is explicitly started by the junit-pipeline framework.Add option to pipeline.mockStep to display a name (the inline script shows as CmdLine in Azure DevOps).mockBashCommandSearchStepByDisplayName / mockPowerShellCommandSearchStepByDisplayName: Add additional counter argument to pinpoint the right command (if the same command occurs multiple times in one step; default the first one is picked). This makes it possible to define different commandOutput strings per command.Create local HTTP server that receives HTTP(S) requests send by the pipeline. The HTTP server runs on the Azure DevOps agent and intercepts requests from the pipeline (eg. sent using curl). The override literal method should alter the endpoint defined in the pipeline. This feature must also provide the option to return a specific HTTP status code and custom response.Some of the methods add a script task to the yaml. Currently this is a bash type of script, so it is assumed that the Azure DevOps agent is a Linux agent.If the first run of a pipeline is not 'master' but another branch, the pipeline does not run. The first runs must be 'master'.Scripts added in some methods must be Azure DevOps agent agnostic; this means that inserted tasks must either be Linux or Windows scripts. Currently, Linux agents are assumed.If a new AzDoPipeline object is created with a different .yml file, the pipeline in Azure DevOps still uses the original .yml file; the pipeline must use the provided file.Check whether the input pipeline and templates are a valid Azure DevOps pipeline YAML filesOnly YAML templates in the same repository are taken into account. Templates in other repositories (identified with a @ behind the yamlTemplate name) are ignored.
TODO: Option to incorporate other resources (repositories) and manipulate the templates in these repos also.Copying files from the main local repo to the test local repo involves exclusion of files, using an exclusion list. This list is currently hardcoded
and contains "idea, target, .git and class". This should be made configurable in the junit_pipeline.properties file.Sometimes you get the error "org.eclipse.jgit.api.errors.RefAlreadyExistsException: Ref myFeature already exists". This happens if a branch already exists (the checkout wants to create it again). Just ignore this error.With the introduction of tests running in multiple branches, it is not possible to run multiple tests in one go. Second test fails because cloning/checkout is not possible somehowThe updated pipeline code is pushed to the default branch in the test project (master); pushing to other branches is not possible.The project id of the Azure DevOps test project must be configured manually and is not (yet) derived automatically.Add an assert step; check a variable on a certain value using a condition. Exit with 1 if the condition is not met.This step can be added before or after a certain step using a pipeline method.
Check/assert output variables of a step.
Copyright (c) Henry van Merode.
Licensed under the MIT License.