Skip to content

Latest commit

 

History

History
195 lines (151 loc) · 7.78 KB

test.md

File metadata and controls

195 lines (151 loc) · 7.78 KB
Raw

Testing Dev Container Features

A built-in testing framework for Features is in active development. This command helps you iterate on self-authored Features.

The test command utilizes the CLI's build and exec commands to test Features in your local source tree. The command will look at the target path for mirrored src and test directories (example. Without any additional arguments, the test command will auto-generate a test for each Feature (pulling the source code directly from src/<FEATURE>), and exec test/<FEATURE>/test.sh inside of the running container.

For the test to pass, the container must (1) build and start successfully, and (2) execute the test.sh with a success (zero) exit code. Note that auto-generated tests will execute the given Feature with default options.

Additonally, 'scenarios' can be defined for each Feature to test more complicated cases. For example, you can use a scenario to test Feature options or >1 Feature in a container. You can find more information about writing scenario tests below.

The source code of the sub-command is here. An example of the command being used in CI can be found in the devcontainers/feature-starter repo and the devcontainers/features repo.

For more information on the test command, run devcontainer features test --help.

An example project structure can be found below.

.
├── README.md
├── src
│   ├── dotnet
│   │   ├── devcontainer-feature.json
│   │   └── install.sh
│   ├── oryx
│   │   ├── devcontainer-feature.json
│   │   └── install.sh
|   ├── ...
│   │   ├── devcontainer-feature.json
│   │   └── install.sh
├── test
│   ├── _global
│	│	├── scenarios.json
│   │   └── some_test_scenario.sh
│   ├── dotnet
│   │   └── test.sh
│   ├── oryx
|   |   ├── scenarios.json
|   |   ├── install_dotnet_and_oryx.sh
│   |   └── test.sh
|   ├── ...
│   │   └── test.sh
...

To run all the dotnet related tests from a repo structured above, the command would be:

devcontainer features test  -f dotnet -b ubuntu

Scenarios

Scenarios are an additional mode that augments the auto-generated test (that is asserted with the test/<FEATURE>/test.sh script).

Scenarios are snippets of devcontainer.json configuration. The scenario is a JSON object, where the key is the test name, and the object is a devcontainer.json.

The following example references the oryx Feature.

The following scenarios.json defines a single test scenario named install_dotnet_and_oryx. The scenario will install the dotnet and oryx Features in the target repo with the provided options.

test/oryx/scenarios.json
{
    "install_dotnet_and_oryx": {
        "image": "ubuntu:focal",
        "features": {
            "dotnet": {
                "version": "6",
                "installUsingApt": "false"
            },
            "oryx": {}
        }
    }
}

The test command will build a container with the config above, and then look for a .sh test file with the same name. The test will pass if the container builds successfully and the install_dotnet_and_oryx.sh shell script exits will a successful exit code (0).

test/install_dotnet_and_oryx.sh
#!/bin/bash

set -e

# Import test library for `check` command
source dev-container-features-test-lib

check "Oryx version" oryx --version
check "Dotnet is not removed if it is not installed by the Oryx Feature" dotnet --version

# Install platforms with oryx build tool
check "oryx-install-dotnet-2.1" oryx prep --skip-detection --platforms-and-versions dotnet=2.1.30
check "dotnet-2-installed-by-oryx" ls /opt/dotnet/ | grep 2.1

....
....

# Replicates Oryx's behavior for universal image
mkdir -p /opt/oryx
echo "vso-focal" >> /opt/oryx/.imagetype

mkdir -p /opt/dotnet/lts
cp -R /usr/local/dotnet/current/dotnet /opt/dotnet/lts
cp -R /usr/local/dotnet/current/LICENSE.txt /opt/dotnet/lts
cp -R /usr/local/dotnet/current/ThirdPartyNotices.txt /opt/dotnet/lts

....
....

# Report result
reportResults

Providing additional build files (Dockerfile, lifecycle scripts, etc...)

The pattern below can be used to provide additional files to the container build. The test command will look for a test/<FEATURE>/<SCENARIO_NAME> directory, and copy the contents into the hidden, intermediate .devcontainer folder used for the build. This can be used to provide additional files to the container, such as a Dockerfile or lifecycle scripts.

For example, given a scenario defined in a Feature's scenarios.json:

// ...
	"frowning_with_a_dockerfile": {
		"build": {
			"dockerfile": "Dockerfile"
		},
		"features": {
			"smile": {
				"shouldFrown": true
			}
		}
	}
// ...

The following file structure can be used to correctly "wire in" the Dockerfile for the build of this scenario:

.
├── src
│   └── smile
│       ├── devcontainer-feature.json
│       └── install.sh
└── test
    └── smile
        ├── frowning.sh
        ├── frowning_with_a_dockerfile  <------------ new folder   
        │   └── Dockerfile              <----- gets copied over to generated .devcontainer
        ├── frowning_with_a_dockerfile.sh
        ├── scenarios.json
        ├── smiling.sh
        └── test.sh

The intermediate .devcontainer folder will look similar to this. Notice that the Dockerfile will be correctly evaluated since it is placed into the .devcontainer folder. A full example is codified as a test in this repository.

/tmp/devcontainercli/container-features-test/1667949592814 $ tree -a
.
├── .devcontainer
│   ├── devcontainer.json
│   ├── Dockerfile
│   └── smile
│       ├── devcontainer-feature.json
│       └── install.sh
│
├── dev-container-features-test-lib
├── frowning.sh
├── frowning_with_a_dockerfile
│   └── Dockerfile
├── frowning_with_a_dockerfile.sh
├── scenarios.json
├── smiling.sh
└── test.sh

NOTE: The flags --global-scenarios-only, --skip-scenarios, and --skip-autogenerated can be passed to run a subset of tests.

Global Scenarios

The test/_global directory is a special directory that holds scenario tests not tied to a specific Feature. This directory is useful for scenarios that broadly tests several Features in a given repository.

The --global-scenarios-only can be passed to only run the global scenarios.

dev-container-features-test-lib

The dev-container-features-test-lib is convenience helper defined in the CLI that adds several bash functions to organize test asserts. Note that using this libary is not required.

check <LABEL> <cmd> [args...]

Description: Executes cmd and prints success/failed depending on exit code (0 === success) of cmd. Note: Use quotes to include whitespace in the label or individual arguments for the command. Example: check "python is available" python3 --version

reportResults

Prints results of check and checkMultiple