Problem
Base has good unit/contract coverage today: Python engines are covered with pytest, Bash command behavior is covered with BATS, and bin/base-test dogfoods the project test contract. What is still missing is a true integration layer that proves common Base workflows work end to end across an isolated workspace.
This matters more now because Base supports multiple installation layouts, including a Git checkout beside project repos and Homebrew-installed Base under Homebrew�s prefix.
Proposed approach
Add a hermetic integration test suite that creates a temporary developer environment instead of mutating the real machine:
- create a temporary
HOME, BASE_CACHE_DIR, and workspace root
- use the checked-out Base repo as the Base runtime under test
- create one or more fake project repos with
base_manifest.yaml
- run real
basectl commands through the public launchers
- stub external tools such as
brew, mise, IDE CLIs, and notification commands when a scenario needs them
- keep tests deterministic and network-free by default
Good first scenarios:
basectl --version and basectl version resolve the expected Base runtime
basectl projects list --workspace <tmp-workspace> discovers a sibling project
basectl setup <project> can reconcile a no-artifact project without touching the real user environment
basectl check <project> and basectl doctor <project> produce expected text and JSON output for the fake project
basectl test <project> runs a manifest-declared command from the project root and passes extra args after --
basectl update-profile --dry-run reports planned profile changes without writing real shell startup files
- Homebrew-layout simulation verifies that project discovery does not assume project repos live beside Brew-installed
BASE_HOME
Suggested implementation
- Add integration tests under
tests/integration/ or integration/.
- Prefer BATS for shell-level workflow tests because the public surface is command-oriented.
- Factor common setup helpers for temporary workspaces, fake
PATH tools, manifests, and assertions.
- Add an
integration job to .github/workflows/tests.yml after the first suite is stable.
- Keep full machine bootstrap, real Homebrew installs, and networked package installation out of the default PR path.
Relationship to future CI work
This complements the planned basectl ci work. The integration suite should validate today�s public commands first, while basectl ci can later add a purpose-built non-interactive CI command surface.
Acceptance criteria
- A new integration suite runs locally with one command and does not mutate the developer�s real home directory, shell profiles, Homebrew state, or project repos.
- The suite is network-free by default.
- GitHub Actions runs the integration suite on Ubuntu for PRs.
- The suite covers at least project discovery, project setup/check/doctor, project test delegation, and profile dry-run behavior.
- Documentation explains when to add unit tests versus integration tests.
- Follow-up issues are opened for macOS/Homebrew packaging smoke tests if those require slower or privileged runners.
Problem
Base has good unit/contract coverage today: Python engines are covered with pytest, Bash command behavior is covered with BATS, and
bin/base-testdogfoods the project test contract. What is still missing is a true integration layer that proves common Base workflows work end to end across an isolated workspace.This matters more now because Base supports multiple installation layouts, including a Git checkout beside project repos and Homebrew-installed Base under Homebrew�s prefix.
Proposed approach
Add a hermetic integration test suite that creates a temporary developer environment instead of mutating the real machine:
HOME,BASE_CACHE_DIR, and workspace rootbase_manifest.yamlbasectlcommands through the public launchersbrew,mise, IDE CLIs, and notification commands when a scenario needs themGood first scenarios:
basectl --versionandbasectl versionresolve the expected Base runtimebasectl projects list --workspace <tmp-workspace>discovers a sibling projectbasectl setup <project>can reconcile a no-artifact project without touching the real user environmentbasectl check <project>andbasectl doctor <project>produce expected text and JSON output for the fake projectbasectl test <project>runs a manifest-declared command from the project root and passes extra args after--basectl update-profile --dry-runreports planned profile changes without writing real shell startup filesBASE_HOMESuggested implementation
tests/integration/orintegration/.PATHtools, manifests, and assertions.integrationjob to.github/workflows/tests.ymlafter the first suite is stable.Relationship to future CI work
This complements the planned
basectl ciwork. The integration suite should validate today�s public commands first, whilebasectl cican later add a purpose-built non-interactive CI command surface.Acceptance criteria