Skip to content

Latest commit

 

History

History
266 lines (205 loc) · 10.2 KB

test.md

File metadata and controls

266 lines (205 loc) · 10.2 KB
Raw

Testing Dev Container Features

A built-in testing framework for Features is in active development. This command helps you iterate on self-authored Features.

The test command utilizes the CLI's build and exec commands to test Features in your local source tree. The command will look at the target path for mirrored src and test directories (example. Without any additional arguments, the test command will auto-generate a test for each Feature (pulling the source code directly from src/<FEATURE>), and exec test/<FEATURE>/test.sh inside of the running container.

For the test to pass, the container must (1) build the generated dev container and start successfully, and (2) execute the test.sh with a success (zero) exit code. Note that auto-generated tests will execute the given Feature with default options.

There are additional 'modes' that run an additional set of tests per Feature if the required assertion script is present. The table below highlights other modes:

Mode Use Case Details CLI Flag
Auto-generated Auto-generates a barebones dev container with (1) the target Features, (2) the --base-image, and the (3) --remote-user) Quick way to assert the default behavior of a Feature --skip-autogenerated
Scenarios Define more complicated test scenarios. Use a scenario to test Feature options or >1 Feature in a container. Details --skip-scenarios
Duplicate Tests A mode that will generate a dev container installing the same Feature twice with different options Details --skip-duplicated

The source code of the sub-command is here. An example of the command being used in CI can be found in the devcontainers/feature-starter repo and the devcontainers/features repo.

For more information on the test command, run devcontainer features test --help.

An example project structure can be found below.

.
├── README.md
├── src
│   ├── dotnet
│   │   ├── devcontainer-feature.json
│   │   └── install.sh
│   ├── oryx
│   │   ├── devcontainer-feature.json
│   │   └── install.sh
|   ├── ...
│   │   ├── devcontainer-feature.json
│   │   └── install.sh
├── test
│   ├── _global
│	│	├── scenarios.json
│   │   └── some_test_scenario.sh
│   ├── dotnet
|   |   ├── duplicate.sh
│   │   └── test.sh
│   ├── oryx
|   |   ├── scenarios.json
|   |   ├── install_dotnet_and_oryx.sh
│   |   └── test.sh
|   ├── ...
│   │   └── test.sh
...

To run all the dotnet related tests from a repo structured above, the command would be:

devcontainer features test  -f dotnet --base-image ubuntu

Scenarios

Scenarios are an additional mode that augments the auto-generated test (that is asserted with the test/<FEATURE>/test.sh script).

Scenarios are snippets of devcontainer.json configuration. The scenario is a JSON object, where the key is the test name, and the object is a devcontainer.json.

The following example references the oryx Feature.

The following scenarios.json defines a single test scenario named install_dotnet_and_oryx. The scenario will install the dotnet and oryx Features in the target repo with the provided options.

test/oryx/scenarios.json
{
    "install_dotnet_and_oryx": {
        "image": "ubuntu:focal",
        "features": {
            "dotnet": {
                "version": "6",
                "installUsingApt": "false"
            },
            "oryx": {}
        }
    }
}

The test command will build a container with the config above, and then look for a .sh test file with the same name. The test will pass if the container builds successfully and the install_dotnet_and_oryx.sh shell script exits will a successful exit code (0).

test/install_dotnet_and_oryx.sh
#!/bin/bash

set -e

# Import test library for `check` command
source dev-container-features-test-lib

check "Oryx version" oryx --version
check "Dotnet is not removed if it is not installed by the Oryx Feature" dotnet --version

# Install platforms with oryx build tool
check "oryx-install-dotnet-2.1" oryx prep --skip-detection --platforms-and-versions dotnet=2.1.30
check "dotnet-2-installed-by-oryx" ls /opt/dotnet/ | grep 2.1

....
....

# Replicates Oryx's behavior for universal image
mkdir -p /opt/oryx
echo "vso-focal" >> /opt/oryx/.imagetype

mkdir -p /opt/dotnet/lts
cp -R /usr/local/dotnet/current/dotnet /opt/dotnet/lts
cp -R /usr/local/dotnet/current/LICENSE.txt /opt/dotnet/lts
cp -R /usr/local/dotnet/current/ThirdPartyNotices.txt /opt/dotnet/lts

....
....

# Report result
reportResults

Providing additional build files (Dockerfile, lifecycle scripts, etc...)

The pattern below can be used to provide additional files to the container build. The test command will look for a test/<FEATURE>/<SCENARIO_NAME> directory, and copy the contents into the hidden, intermediate .devcontainer folder used for the build. This can be used to provide additional files to the container, such as a Dockerfile or lifecycle scripts.

For example, given a scenario defined in a Feature's scenarios.json:

// ...
	"frowning_with_a_dockerfile": {
		"build": {
			"dockerfile": "Dockerfile"
		},
		"features": {
			"smile": {
				"shouldFrown": true
			}
		}
	}
// ...

The following file structure can be used to correctly "wire in" the Dockerfile for the build of this scenario:

.
├── src
│   └── smile
│       ├── devcontainer-feature.json
│       └── install.sh
└── test
    └── smile
        ├── frowning.sh
        ├── frowning_with_a_dockerfile  <------------ new folder   
        │   └── Dockerfile              <----- gets copied over to generated .devcontainer
        ├── frowning_with_a_dockerfile.sh
        ├── scenarios.json
        ├── smiling.sh
        └── test.sh

The intermediate .devcontainer folder will look similar to this. Notice that the Dockerfile will be correctly evaluated since it is placed into the .devcontainer folder. A full example is codified as a test in this repository.

/tmp/devcontainercli/container-features-test/1667949592814 $ tree -a
.
├── .devcontainer
│   ├── devcontainer.json
│   ├── Dockerfile
│   └── smile
│       ├── devcontainer-feature.json
│       └── install.sh
│
├── dev-container-features-test-lib
├── frowning.sh
├── frowning_with_a_dockerfile
│   └── Dockerfile
├── frowning_with_a_dockerfile.sh
├── scenarios.json
├── smiling.sh
└── test.sh

NOTE: The flags --global-scenarios-only, --skip-scenarios, and --skip-autogenerated can be passed to run a subset of tests.

Global Scenarios

The test/_global directory is a special directory that holds scenario tests not tied to a specific Feature. This directory is useful for scenarios that broadly tests several Features in a given repository.

The --global-scenarios-only can be passed to only run the global scenarios.

Duplicate-style Tests

When executing the command without the --skip-duplicated flag, each Feature with a duplicate.sh will generate a test installing a given Feature twice (with different options). This is useful for asserting that a Feature can be installed multiple times without conflict (is idempotent). Additionally, the options used for each distinct Feature are passed into the assertion script, should that be useful to write an assertion.

For example, the dotnet Feature above provided a duplicate.sh, therefore the test command will generate a dev container test case installing dotnet twice.

The generated dev container could look something like this:

devcontainer.json

{
    "image": "ubuntu",
    "features": {
        "./dotnet": {
            "version": "5",
            "installUsingApt": "false"
        },
        "./dotnet-0": {} // Default
    }
}

duplicate.sh

#!/bin/bash

set -e

# Optional: Import test library
source dev-container-features-test-lib

# The values of the randomized options will be set as environment variables.
if [ -z "${VERSION}" ]; then
	echo "Version of dotnet to install from randomized Feature not set!"
	exit 1
fi

if [ -z "${INSTALLUSINGAPT}" ]; then
	echo "Boolean to install using apt from randomized Feature not set!"
	exit 1
fi

# The values of the default options will be set as environment variables.
if [ -z "${VERSION__DEFAULT}" ]; then
	echo "Version of dotnet to install default Feature not set!"
	exit 1
fi

if [ -z "${INSTALLUSINGAPT__DEFAULT}" ]; then
	echo "oolean to install using apt from default Feature not set!"
	exit 1
fi

check "randomized version of dotnet installed"  bash -c "dotnet --list-sdks | ${VERSION}"
check "default version of dotnet installed"  bash -c "dotnet --list-sdks | ${VERSION__DEFAULT}"

# ...
# ...

# Report result
reportResults

dev-container-features-test-lib

The dev-container-features-test-lib is convenience helper defined in the CLI that adds several bash functions to organize test asserts. Note that using this libary is not required.

check <LABEL> <cmd> [args...]

Description: Executes cmd and prints success/failed depending on exit code (0 === success) of cmd. Note: Use quotes to include whitespace in the label or individual arguments for the command. Example: check "python is available" python3 --version

reportResults

Prints results of check and checkMultiple