Skip to content

Add hermetic integration tests for Base workflows #294

@codeforester

Description

@codeforester

Problem

Base has good unit/contract coverage today: Python engines are covered with pytest, Bash command behavior is covered with BATS, and bin/base-test dogfoods the project test contract. What is still missing is a true integration layer that proves common Base workflows work end to end across an isolated workspace.

This matters more now because Base supports multiple installation layouts, including a Git checkout beside project repos and Homebrew-installed Base under Homebrew�s prefix.

Proposed approach

Add a hermetic integration test suite that creates a temporary developer environment instead of mutating the real machine:

  • create a temporary HOME, BASE_CACHE_DIR, and workspace root
  • use the checked-out Base repo as the Base runtime under test
  • create one or more fake project repos with base_manifest.yaml
  • run real basectl commands through the public launchers
  • stub external tools such as brew, mise, IDE CLIs, and notification commands when a scenario needs them
  • keep tests deterministic and network-free by default

Good first scenarios:

  • basectl --version and basectl version resolve the expected Base runtime
  • basectl projects list --workspace <tmp-workspace> discovers a sibling project
  • basectl setup <project> can reconcile a no-artifact project without touching the real user environment
  • basectl check <project> and basectl doctor <project> produce expected text and JSON output for the fake project
  • basectl test <project> runs a manifest-declared command from the project root and passes extra args after --
  • basectl update-profile --dry-run reports planned profile changes without writing real shell startup files
  • Homebrew-layout simulation verifies that project discovery does not assume project repos live beside Brew-installed BASE_HOME

Suggested implementation

  • Add integration tests under tests/integration/ or integration/.
  • Prefer BATS for shell-level workflow tests because the public surface is command-oriented.
  • Factor common setup helpers for temporary workspaces, fake PATH tools, manifests, and assertions.
  • Add an integration job to .github/workflows/tests.yml after the first suite is stable.
  • Keep full machine bootstrap, real Homebrew installs, and networked package installation out of the default PR path.

Relationship to future CI work

This complements the planned basectl ci work. The integration suite should validate today�s public commands first, while basectl ci can later add a purpose-built non-interactive CI command surface.

Acceptance criteria

  • A new integration suite runs locally with one command and does not mutate the developer�s real home directory, shell profiles, Homebrew state, or project repos.
  • The suite is network-free by default.
  • GitHub Actions runs the integration suite on Ubuntu for PRs.
  • The suite covers at least project discovery, project setup/check/doctor, project test delegation, and profile dry-run behavior.
  • Documentation explains when to add unit tests versus integration tests.
  • Follow-up issues are opened for macOS/Homebrew packaging smoke tests if those require slower or privileged runners.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions