Datadog Continuous Testing for Azure DevOps

Overview

With the SyntheticsRunTests task, you can run Synthetic tests within your Azure Pipeline configuration and ensure all your teams using Azure DevOps can benefit from Synthetic tests at every stage of the software lifecycle.

For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.

Authentication

Service Connection

To connect to your Datadog site, Datadog recommends setting up a custom service connection when configuring the SyntheticsRunTests task.

You need to provide the following inputs:

• Datadog site: Your Datadog site. The possible values are listed in this table.
• Custom subdomain (default: app): The custom subdomain to access your Datadog organization. If your URL is myorg.datadoghq.com, the custom subdomain is myorg.
• API key: Your Datadog API key. This key is created in your Datadog organization.
• Application key: Your Datadog application key. This key is created in your Datadog organization.

API and Application keys

• API key: Your Datadog API key. This key is created in your Datadog organization and should be stored as a secret.
• Application key: Your Datadog application key. This key is created in your Datadog organization and should be stored as a secret.
• Datadog site: Your Datadog site. The possible values are listed in this table.
• Custom subdomain (optional): The custom subdomain to access your Datadog organization. If your URL is myorg.datadoghq.com, the custom subdomain is myorg.

Setup

To connect to your Datadog account, create a Datadog CI service connection in your Azure pipelines project. Once created, all you need is the name of the service connection in the tasks.

1. Install the Datadog Continuous Testing extension from the Visual Studio Marketplace in your Azure Organization.
2. Add your Datadog API and application keys in the Datadog CI service connection, or as secrets to your Azure Pipelines project.
3. In your Azure DevOps pipeline, use the SyntheticsRunTests task.

Your task can be simple or complex.

Simple usage

Example task using public IDs

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    publicIds: |
      abc-d3f-ghi
      jkl-mn0-pqr

Example task using existing synthetics.json files

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    files: 'e2e-tests/*.synthetics.json'

For an example test file, see this test.synthetics.json file.

Example task using pipeline secrets for authentication

- task: SyntheticsRunTests@1
  inputs:
    authenticationType: 'apiAppKeys'
    apiKey: '$(DatadogApiKey)'
    appKey: '$(DatadogAppKey)'
    datadogSite: '$(DatadogSite)'
    subdomain: 'myorg'

Complex usage

Example task using the testSearchQuery

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    testSearchQuery: 'tag:e2e-tests'

Example task using the testSearchQuery and variable overrides

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    testSearchQuery: 'tag:e2e-tests'
    variables: |
      START_URL=https://staging.website.com
      PASSWORD=$(StagingPassword)

Example task using a global configuration file with configPath

By default, the path to the global configuration file is datadog-ci.json. You can override this path with the config_path input.

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    configPath: './global.config.json'
    connectedService: 'my-datadog-ci-connected-service'

Inputs

For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.

Name	Description
apiKey	Your Datadog API key. This key is created in your Datadog organization and should be stored as a secret. Required when authenticationType == apiAppKeys
appKey	Your Datadog application key. This key is created in your Datadog organization and should be stored as a secret. Required when authenticationType == apiAppKeys
authenticationType	(Required) How to store and retrieve credentials. Must be either apiAppKeys or connectedService
batchTimeout	Specifies the timeout duration in milliseconds for the CI batch. When a batch times out, the CI job fails and no new test runs are triggered, but ongoing test runs complete normally. Default: 1800000 (30 minutes)
connectedService	The name of the Datadog CI service connection. Required when authenticationType == connectedService
configPath	The path to the global configuration file that configures datadog-ci. Default: datadog-ci.json
datadogSite	Your Datadog site. The possible values are listed in this table. Default: datadoghq.com
failOnCriticalErrors	Fail the CI job if a critical error that is typically transient occurs, such as rate limits, authentication failures, or Datadog infrastructure issues. Default: false
failOnMissingTests	Fail the CI job if the list of tests to run is empty or if some explicitly listed tests are missing. Default: false
failOnTimeout	Fail the CI job if the CI batch fails as timed out. Default: true
files	Glob patterns to detect Synthetic test configuration files, separated by new lines. Default: {,!(node_modules)/**/}*.synthetics.json
jUnitReport	The filename for a JUnit report if you want to generate one. Default: none
locations	Override the list of locations to run the test from, separated by new lines or commas. The possible values are listed in this API response. Default: none
publicIds	Public IDs of Synthetic tests to run, separated by new lines or commas. If no value is provided, tests are discovered in Synthetic test configuration files. Default: none
selectiveRerun	Whether to only rerun failed tests. If a test has already passed for a given commit, it is not rerun in subsequent CI batches. By default, your organization's default setting is used. Set it to false to force full runs when your configuration enables it by default. Default: none
subdomain	The custom subdomain to access your Datadog organization when authenticationType == apiAppKeys. If your URL is myorg.datadoghq.com, the custom subdomain is myorg. Default: app
testSearchQuery	Use a search query to select which Synthetic tests to run. Use the Synthetic Tests list page's search bar to craft your query, then copy and paste it. Default: none
variables	Override existing or inject new local and global variables in Synthetic tests as key-value pairs, separated by new lines or commas. For example: START_URL=https://example.org,MY_VARIABLE=My title. Default: none

Outputs

Name	Description
batchUrl	The URL of the batch.
criticalErrorsCount	The number of critical errors.
failedCount	The number of failed results.
failedNonBlockingCount	The number of failed non-blocking results.
passedCount	The number of passed results.
previouslyPassedCount	The number of previously passed results.
testsNotFoundCount	The number of not found tests.
testsSkippedCount	The number of skipped tests.
timedOutCount	The number of timed out results.
rawResults	The list of results, as a raw JSON string.

Further reading

Additional helpful documentation, links, and articles:

• Getting Started with Continuous Testing
• Continuous Testing and CI/CD Configuration
• Best practices for continuous testing with Datadog