[Tracking] devcontainer features test improvements

[joshspicer]

An issue collecting various improvements that can be made for the features test... command. The command is still experimental and under active development.

• remoteUser, containerUser, or a container with a default non-root user are failing with a permissions issue when included in a scenario
• Scenario tests appear to run as root even when -u specified in command line
• Feature values specified in scenario.json do not appear to be space escaped ("a b" ends up as "a")
• -f prevents other parameters after it from being used since it treats them all as part of the array. Makes it impossible to pass in the require parameters.
• Document all command functionality (ie: `dev-container-features-test-lib` docs? spec#113 (comment)) (devcontainer features test documentation #219)
• If there is a scenario.json file and no test.sh, the test will not run unless --skip-autogenerated is set. We should be smarter about this
• The one positional argument (path to collection) is required and doesn't use the default value (cwd)
• Support build.dockerfile - Currently errors since Dockerfile referenced is not copied to temp location
• Allow specifying a specific scenario to test via the CLI (helps if only one of several is failing)
• devcontainer features test: Exec'ing test scripts assumes /bin/bash #438
• Output results to stdout so you can pipe to a log file w/o getting the entire build output (appears to go to stderr right now)
• Be able to override a certain base image <-> feature combos in the autogenerated tests (Oryx - Clean up and shrink size features#116 (review))
• Feature Dependencies - installsAfter not respected #163 / scenarios.json does not support running with existing features #217
• Run same tests with multiple scenarios spec#114 (reply in thread)
• [features test command] add --docker-cli-path #334
• [feature] remove test containers after every run #409
• [question] How can I pass external args to tests? #448

[eitsupi]

Hello.
I developed a feature using the feature template and noticed that the test for option selection was not covered.

Is it possible to run multiple installations by referencing a pre-created options matrix (json file) when running an installation as a test?
Or the user may want to give options as CLI arguments to parallelize using the matrix on GitHub Actions.

Similarly, it would be useful to be able to specify the platform (CPU architecture).

edit: I understand that we can add scenarios.json to the folder for each feature, but it seems useless because when I set up the GitHub matrix, the tests in scenarios.json are run under all matrix conditions --skip-scenarios seems work fine.

[Clockwork-Muse]

Can we add another task? Currently, there isn't a way to supply --docker-path/--docker-compose-path to any of the features commands, which causes issues if you're using podman in ci scenarios.

[joshspicer]

@Clockwork-Muse Do you mean a way to specify the path to the docker/docker-compose CLIs? Yes, I'll convert your request into an issue 👍 (If you're interested, we'd also accept a PR on this)

[Clockwork-Muse]

Yes, that's what I meant.

[jcbhmr]

If there is a scenario.json file and no test.sh, the test will not run unless --skip-autogenerated is set. We should be smarter about this

👍 id very much like to see this so that i can do this in something like gh actions:

devcontainer features test -f ${{ matrix.id }}

and NOT fail like this:

as a workaround I'm currently doing this:

if [[ -e test/${{ matrix.id }}/test.sh ]]; then
  devcontainer features test -f ${{ matrix.id }}
else
  devcontainer features test -f ${{ matrix.id }} --skip-autogenerated
fi

accidentally made devcontainers/spec#294 🙄