Perl Helper Tools for Azure Pipelines
This repo contains a set of tools and Azure Pipelines templates designed to make it easy to test Perl projects with multiple versions of Perl across Linux, macOS, and Windows.
I consider this stuff to be at a beta level of stability. I'm trying to do all work in branches so that the master branch will always work with existing projects. However, I may make breaking changes to the way templates are used, so I encourage you to pin your use to a tag. See below for details on how to do that.
Creating a Service Connection
In order to use these templates in your pipelines you must create a "Service Connection" for your Azure project. Fortunately this is only needed once per Azure project, and a single project can contain many pipelines, each of which corresponds to a single code project (GitHub repo, Subversion repo, etc.).
Go to https://dev.azure.com/.
Click on the project that contains (or will contain) the pipelines which will use these templates.
Click on the gear icon in the lower left.
Click on "Service Connections".
Select "New service connection", then "GitHub".
For "Authentication method" select "Grant authorization".
For "OAuth Configuration" select "AzurePipelines".
Click on "Authorize". You may be prompted to log in to GitHub and/or to allow a third party application to access GitHub on your behalf. You will need to allow this, obviously.
The connection name can be anything you want. It is simply used to refer to
the connection in your
azure-pipelines.yml files. I suggest using
houseabsolute/ci-perl-helpers if you don't have a naming scheme you want to
Click on "Save".
If you have multiple Azure DevOps projects you will need to do this once per project. Note that a single project can contain many repos with many pipelines. You only need separate projects if you want separate ACLs.
Put this in your
resources: repositories: - repository: ci-perl-helpers type: github name: houseabsolute/ci-perl-helpers endpoint: houseabsolute/ci-perl-helpers stages: - template: templates/build.yml@ci-perl-helpers - template: templates/linux.yml@ci-perl-helpers parameters: test_xt: true use_default_perls: true - template: templates/macos.yml@ci-perl-helpers parameters: use_default_perls: true - template: templates/windows.yml@ci-perl-helpers parameters: use_default_perls: true
resources.repositories.endpoint value must match the name of the Service
Connection you created.
This will test your Perl project in the following scenarios:
- On Windows, using the latest stable version of Perl available via Berrybrew.
- On macOS, using the latest stable version of Perl.
- On Linux with the last stable release of each major Perl version starting from 5.8.9 up to the newest stable release (5.30.1 at the time this was written).
- On Linux with the latest dev release of Perl.
- On Linux with the current contents of the
bleadbranch of the github.com/Perl/perl5 repo. If tests fail when
proveis run then your pipeline will still pass, but a failure to compile your code will cause the pipeline to fail.
Pinning a Helpers Version
If you do not specify a
ref when referring to this repo, your build will
always pull the latest version of this project's templates from this repo's
master branch. To pin your project to a specific verson of these templates,
resources: repositories: - repository: ci-perl-helpers type: github name: houseabsolute/ci-perl-helpers ref: refs/tags/v0.1.0 endpoint: houseabsolute/ci-perl-helpers
There is one template for the build stage,
build.yml, and three test stage
All of these take the following common parameters:
||If you set this to a string it will be used as part of the cache key for the Perl installation used by this stage. Every time you change this key you will invalidate the old cache. In most cases you should not need to change this, but if your build fails in a confusing way you can try setting this to see if that fixes the problem. If it does, just leave the new key in place and the next build will use the new cache.|
||If you set this to a true value then the helper tools will spew out a lot more debugging information. Please set this to true and do a build before reporting issues with these tools. That way I can look at your failed build and have a better sense of what went wrong.|
||string||tag or branch of the
||This is the suffix used as part of the Docker tag for image that the build job runs on. You should not set this manually unless you have a very good reason. By default, this suffix is determined by looking at the version of the
The test stages share most of their parameters in common.
Choosing Perl Versions
There are a number of options for choosing which versions of Perl you want to
test with. When referring to Perl versions there are two different ways to do
so. You can pass a full version like
"5.28.2", or you can pass
just the major and minor version like
"5.28". If you just pass
major and minor then the helpers will automatically pick the highest patch
release for that minor series of releases.
You can also use the following strings:
latest- the most recent stable Perl release
dev- the most recent dev release
blead- a Perl will be built from the latest code in the perl5 git repository.
The templates accept the following parameters to determine which Perls to test with.
||If this is true, the stage will use whatever the default Perl versions are for that operating system. See the "Quick Start" section above for a description of each template's defaults.|
||array of strings||
||You can use this to provide an explicit list of Perl versions to test with.|
||You can set one or both of these to ask for a range of Perl versions. The range is inclusive of both the low and high ends. See below for more details.|
||boolean||false||If this is true then both unthreaded and threaded versions of each Perl will be tested. This parameter is not supported by the
||array of strings||
||An array of Perl versions for which test failures are not treated as a failure of the CI job. You can refer to Perl versions in this array using the same options as you do elsewhere.|
When selecting a range of Perls, the last patch release of a minor series will be selected. If you leave one end of the range unset, then it uses the default bounds for the range. The default lowest Perl is 5.8.9 and the default highest Perl is blead.
The following versions are available on Linux and macOS:
- Every stable release thereafter.
latest, the most recent stable release.
dev, the most recent dev release, such as 5.31.7.
blead, the latest commit to the perl5 git repository.
On Windows, the available versions are the same as those supported by Berrybrew. When a version is available in both 64- and 32-bit variants, the 64-bit variant will always be chosen.
All of the test stages allow you to run coverage and extended tests, to provide custom steps, and to install arbitrary packages.
||By default the test stages do not run tests with coverage enabled. You can use this parameter to enable a coverage test. If you set this to a non-empty string then coverage will be tested with the most recent stable release of Perl included in this stage. The value of this string determines the type of coverage report that is generated. See below for the allowed options.|
||number||1||Running tests under
||The version of Perl to use when running coverage tests. By default this will be the most recent stable version of Perl included in this stage will be used.|
||boolean||false||If this is true then the raw output from
||boolean||false||If this is true, then one of the test runs will be done with the
||A list of extra Perl packages to be installed before running tests. This list will be passed to
||array of steps||
||You can provide an arbitrary list of steps to be run at the start or end of the job that runs the tests.|
||array of strings||
||You can use this to pass a list of packages to be installed by the appropriate package manager (Apt, Brew, or Chocolatey).|
The following values are accepted for the
html- Generates a report as a set of HTML files.
clover- Generates a report in the format expected by the Atlassian Clover software.
codecov- Uploads coverage data to codecov.io. You must also set
CODECOV_TOKENas a pipeline variable. You almost certainly want to make this value secret. If your repository contains a
.codecov.ymlfile then this will be used when uploading the report.
coveralls- Uploads coverage data to coveralls.io. You must also set
COVERALLS_TOKENas a pipeline variable. You almost certainly want to make this value secret.
sonarqube- Generates a report in the format expected by SonarQube. See the
Devel::Cover::Report::SonarGenericdocs for details on how to have this automatically uploaded to SonarQube.
Installing Tools for Extended Tests
If you have a script at
./dev-bin/install-xt-tools.sh, this will be run
before running extended tests. This script can be used to do things like
installing non-Perl dependencies. The best place to install things is in
/usr/local/bin, which is already in the
PATH when tests are run.
How This Works
These tools consist of a set of Azure Pipeline templates, Perl scripts for various tasks, and a set of Docker images for Linux testing.
The Docker images contain two versions of Perl, one of which is used to run
the tools and build the distribution, and one of which is used to execute your
package's tests. This is useful for a few reasons. First, it me use modern
Perl idioms (like subroutine signatures) in the the tools. Second, it means
you can test on older Perls even if your tooling requires a newer Perl. For
Dist::Zilla requires Perl
5.14.0 but I have distros which use it and still support 5.8.9. It also means
that dependencies needed for building, for example
Dist::Zilla and its dependencies,
are not present when running tests. That means there's a better chance of
discovering missing prereqs.
The Pipeline itself has several stages. The Build stage contains a single
job. This job checks out your source and generates a tarball from it using
your build tooling. The helper tools can detect the use of dzil or minilla,
and will use them when appropriate. Otherwise the tools fall back to using
Build.PL and executing
make dist or
./Build dist. The resulting tarball is saved as a pipeline artifact.
Then there is one test stage for each of the supported operating systems, Linux, macOS, and Windows. These stages have several jobs. The first job in each stage dynamically generates a matrix of test jobs based on the parameters you provide. Each entry in the matrix turns into a separate job for a single Perl version.
Each matrix job downloads the tarball created in the Build stage. The job
extracts this tarball and executes the contained
as appropriate. The tests are then run using
If you asked for coverage testing, the appropriate
environment variable settings are used to invoke
Devel::Cover. All of the coverage
output is optionally saved as a build artifact. Some coverage reporters also
upload the report directly to a code coverage service. Finally, the test
prove is turned into JUnit XML and uploaded as a set of test
results, which lets you see a more detailed view of test failures in the Azure
Pipelines screen for each CI run.
See this repository's issues for todo items.