test: Reqnroll BDD acceptance suite for the CLI (PROT-41)#5
Merged
Conversation
Mirror install's flag so uninstall can skip the PATH edit. Without it, uninstall unconditionally rewrites the user PATH on Windows, which makes it unsafe to exercise from automated tests. Guard PathManager.RemoveFromPath behind the flag.
Black-box acceptance tests that run the real built protostar binary via CliWrap and assert on stdout, exit code, and filesystem side effects, the way a user runs it. Reqnroll (SpecFlow's successor) plus xUnit. Covers --version, the default command, --help, and install/uninstall in a temp sandbox (--no-modify-path, so the suite never touches the real PATH). - test/Protostar.Cli.Acceptance: Features (Gherkin), Steps, Support (CliRunner, Sandbox) - ci.yml: build and run the suite on Ubuntu and Windows - gitignore Reqnroll-generated *.feature.cs (rebuilt every compile) - test/README documents how to run and how to extend to harness integrations
Member
|
Is CliWrap running our installed CLI? So if we want to write BDD tests about our configuration, will the tests change our settings? |
ljones491
approved these changes
Jun 1, 2026
Member
Author
It is running our CLI but it is sandboxed. I added the ability to provide the path to use for the configuration of protostar / other harnesses. The BDD will setup files for certain harnesses at a location and then pass that location to protostar who won't see a difference between them vs the actual installed location of something like claude. So no we are good. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a black-box acceptance test suite that drives the real built
protostarbinary the way a user runs it, plus the CI to run it on every PR.Resolves PROT-41.
test/Protostar.Cli.Acceptance/Features/*.feature— 5 Gherkin specs:--version, default command,--help, install, uninstall.Steps/CliSteps.cs— bindings using Cucumber expressions.Support/CliRunner.cs— locates the binary viaPROTOSTAR_BINor a repo-root walk.Support/Sandbox.cs— a throwaway temp dir per scenario. Install/uninstall pass--no-modify-path, so the suite never edits the machine PATH.feat(cli):--no-modify-pathonuninstallUninstall previously rewrote the Windows user PATH unconditionally, which made it unsafe to run under test. The flag mirrors
installand guards the PATH edit. Committed separately so release-please sees a clean feature.CI (
.github/workflows/ci.yml)pull_requestand pushes tomain.ubuntu-latestandwindows-latest(exercises the OS-specific binary name and install paths).dotnet build -c Releasethendotnet test -c Release. Actions pinned to SHAs per PROT-39.Verification
dotnet build -c Release: 0 warnings, 0 errors.dotnet test -c Release: 7/7 scenarios pass locally (~1s).Note on scope
The "already installed" idempotency branch only fires when the running exe equals the destination, which a black-box test using the framework-dependent build cannot reach (a copied single exe cannot self-run). The re-install scenario instead asserts a second install is safe and the binary is present. Once CI publishes a self-contained binary, that path becomes testable.