Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/harness.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,9 @@ jobs:
- name: Install deadcode
run: go install golang.org/x/tools/cmd/deadcode@latest
- name: Run deadcode
# -test flag also considers test-only entry points.
run: deadcode -test ./...
# -test includes test-only entry points; e2e,vm tags expose callers
# in the destructive e2e suite (testutil.BuildTestBinary etc.).
run: deadcode -test -tags="e2e,vm" ./...

mod-tidy:
name: go mod tidy diff (drift)
Expand Down
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,8 @@ These are loaded automatically when Claude runs in this repo.
- `git push --force` against `main` or release tags.
- `git commit --amend` on commits already pushed.
- `git reset --hard` discarding uncommitted work.
- Running `make test-vm-inner` (or `test-vm-inner-run`) outside a throwaway
machine — these install real packages onto the current host.
- Triggering L4 e2e tests outside CI — they install real packages onto the
current host. L4 runs only in GitHub Actions (`vm-e2e-spike.yml`).
- Anything that modifies the user's `~/.zshrc`, Homebrew install, or
macOS `defaults`.

Expand Down
3 changes: 1 addition & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,7 @@ make build-release VERSION=0.25.0 # optimized + UPX
# Test — full tier table in CONTRIBUTING.md
make test-unit # L1 (~75s) — unit + integration + contract; pre-push hook
make test-e2e # L3 compiled binary
make test-vm-inner # L4 — full destructive e2e suite (runs in CI on macos-14; locally on a spare machine only)
make test-vm-inner-run TEST=Foo # L4 — single test
# L4 — destructive e2e runs in CI only (vm-e2e-spike.yml on macos-14)
make test-coverage # coverage.out + coverage.html

# Single test
Expand Down
11 changes: 3 additions & 8 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Tests are split across four tiers. Which one runs where:
| **L1 Unit + Integration + Contract** | Pure-Go logic with faked `Runner` *plus* real `brew` / `git` / `npm` against temp dirs and real `httptest` servers | `make test-unit` (~75s) | Every push (pre-push hook); CI on push/PR |
| **L2 Contract schema** | JSON schema validation against [openboot-contract](https://github.com/openbootdotdev/openboot-contract) | (runs in CI only) | CI on push/PR |
| **L3 E2E binary** | Compiled binary driven by scripts; `-tags=e2e` | `make test-e2e` | CI on release |
| **L4 VM e2e** | Full destructive suite (`-tags="e2e,vm"`). Installs real packages, modifies `~/.zshrc`, writes `defaults`. Each run requires a clean macOS host (Apple Silicon). | `make test-vm-inner` (or single test: `make test-vm-inner-run TEST=Foo`) | **CI** — runs on GitHub Actions `macos-14` runner (every PR via `vm-e2e-spike.yml`). Locally only on a throwaway machine. |
| **L4 VM e2e** | Full destructive suite (`-tags="e2e,vm"`). Installs real packages, modifies `~/.zshrc`, writes `defaults`. Each run requires a clean macOS host (Apple Silicon). | CI only — `vm-e2e-spike.yml` on `macos-14` | **CI** — GitHub Actions `macos-14` runner, two parallel jobs. No local target. |

Rules of thumb:

Expand All @@ -47,13 +47,8 @@ Rules of thumb:

## VM E2E

L4 tests run on GitHub Actions (`macos-14` runner, Apple Silicon). Each job
gets a fresh macOS VM — no local setup required.

```bash
make test-vm-inner # full suite (use on a throwaway machine only)
make test-vm-inner-run TEST=TestVM_Journey_FirstTimeUser # single test
```
L4 tests run on GitHub Actions (`macos-14` runner, Apple Silicon, `vm-e2e-spike.yml`).
Each job gets a fresh macOS VM — no local setup or Makefile target required.

## Git Hooks

Expand Down
16 changes: 0 additions & 16 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
.PHONY: test-unit test-e2e test-coverage test-all \
test-vm-inner test-vm-inner-run \
install-hooks uninstall-hooks

BINARY_NAME=openboot
Expand Down Expand Up @@ -28,21 +27,6 @@ test-all:
$(MAKE) test-unit
$(MAKE) test-coverage

# =============================================================================
# L4 VM e2e — destructive tests tagged `e2e,vm`. Run directly on any clean
# macOS host (Apple Silicon). In CI this is a GitHub Actions macos-14 runner
# (see .github/workflows/vm-e2e-spike.yml). Locally, run on a throwaway
# machine or a Tart VM — do NOT run on your primary dev machine.
# =============================================================================

# Run the full L4 suite (same command CI uses).
test-vm-inner:
go test -v -timeout 60m -tags="e2e,vm" ./test/e2e/...

# Run a single L4 test by name: make test-vm-inner-run TEST=TestVM_Journey_FirstTimeUser
test-vm-inner-run:
go test -v -timeout 45m -tags="e2e,vm" -run '$(TEST)' ./test/e2e/...

build:
go build -ldflags="$(LDFLAGS)" -o $(BINARY_PATH) ./cmd/openboot

Expand Down
2 changes: 1 addition & 1 deletion docs/HARNESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ Three regulation categories:
| Behav. | L1 unit + integration + contract (faked runners *and* real brew/git/npm in temp dirs) | pre-push, CI | `make test-unit` |
| Behav. | L2 contract schema (against openboot-contract repo) | CI | `.github/workflows/test.yml` `contract` job |
| Behav. | L3 e2e binary | release | `make test-e2e` |
| Behav. | L4 VM e2e (`vm`) — full destructive suite on a clean macOS host | every PR | `.github/workflows/vm-e2e-spike.yml` (macos-14 runner, two parallel jobs); `make test-vm-inner` for local runs |
| Behav. | L4 VM e2e (`vm`) — full destructive suite on a clean macOS host | every PR | `.github/workflows/vm-e2e-spike.yml` (macos-14 runner, two parallel jobs) |
| Behav. | curl\|bash smoke (install.sh + mock server) | every PR | `.github/workflows/test.yml` `curl-bash-smoke` job |
| Behav. | Auto-release sensor — patch fast lane (`fix:`-only) auto-tags + dispatches `release.yml`; feat threshold opens a `release-ready` issue (check L4 CI green, then tag manually) | push to `main` | `.github/workflows/auto-release.yml` |
| Behav. | Release notes — Conventional Commits since previous tag, grouped by type (Features / Bug Fixes / etc) + Full Changelog link, appended to the install-instructions template | tag push or `workflow_dispatch` | `.github/workflows/release.yml` (`Write release notes` step) |
Expand Down
56 changes: 0 additions & 56 deletions test/e2e/vm_helpers_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,28 +15,6 @@ import (

const brewPath = "/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"

// vmInstallViaBrewTap installs openboot via the install.sh script (curl | bash).
// Requires a TTY — prefer vmInstallViaBrew for non-interactive contexts.
// Returns the installed openboot version string.
func vmInstallViaBrewTap(t *testing.T, vm *testutil.MacHost) string {
t.Helper()

script := strings.Join([]string{
"export NONINTERACTIVE=1",
fmt.Sprintf("export PATH=%q", brewPath),
`/bin/bash -c "$(curl -fsSL https://openboot.dev/install.sh)" -- --help`,
}, " && ")

output, err := vm.Run(script)
t.Logf("install output:\n%s", output)
if err != nil {
t.Fatalf("failed to install openboot via brew: %v", err)
}

version, _ := vm.Run(fmt.Sprintf("export PATH=%q && openboot version", brewPath))
return strings.TrimSpace(version)
}

// vmInstallViaBrew installs openboot via `brew tap && brew install` — no TTY required.
// Use this instead of vmInstallViaBrewTap when running over SSH without -t.
// Returns the installed openboot version string.
Expand Down Expand Up @@ -107,17 +85,6 @@ func vmRunDevBinary(t *testing.T, vm *testutil.MacHost, binaryPath, args string)
return vm.Run(cmd)
}

// vmRunOpenbootWithGit runs openboot with git identity env vars set.
func vmRunOpenbootWithGit(t *testing.T, vm *testutil.MacHost, args string) (string, error) {
t.Helper()
env := map[string]string{
"PATH": brewPath,
"OPENBOOT_GIT_NAME": "E2E Test User",
"OPENBOOT_GIT_EMAIL": "e2e@openboot.test",
}
return vm.RunWithEnv(env, "openboot "+args)
}

// vmRunDevBinaryWithGit runs the dev binary with git identity env vars set.
func vmRunDevBinaryWithGit(t *testing.T, vm *testutil.MacHost, binaryPath, args string) (string, error) {
t.Helper()
Expand Down Expand Up @@ -183,26 +150,3 @@ func writeFile(path, content string) error {
return os.WriteFile(path, []byte(content), 0644)
}

// vmWriteTestSnapshot writes a minimal valid snapshot JSON to a path on the VM.
func vmWriteTestSnapshot(t *testing.T, vm *testutil.MacHost, remotePath string, formulae, casks, npm []string) {
t.Helper()

toJSON := func(ss []string) string {
if len(ss) == 0 {
return "[]"
}
quoted := make([]string, len(ss))
for i, s := range ss {
quoted[i] = fmt.Sprintf("%q", s)
}
return "[" + strings.Join(quoted, ",") + "]"
}

json := fmt.Sprintf(
`{"version":1,"captured_at":"2026-01-01T00:00:00Z","hostname":"test-vm","packages":{"formulae":%s,"casks":%s,"taps":[],"npm":%s},"macos_prefs":[],"shell":{},"git":{},"dotfiles":{},"dev_tools":[],"matched_preset":"","catalog_match":{},"health":{"failed_steps":[],"partial":false}}`,
toJSON(formulae), toJSON(casks), toJSON(npm),
)

_, err := vm.Run(fmt.Sprintf("cat > %s << 'SNAPEOF'\n%s\nSNAPEOF", remotePath, json))
require.NoError(t, err, "failed to write test snapshot to VM")
}
22 changes: 0 additions & 22 deletions testutil/helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,25 +48,3 @@ func findProjectRoot(t *testing.T) string {
}
}

func IsPackageInstalled(packageName string) bool {
cmd := exec.Command("which", packageName)
err := cmd.Run()
return err == nil
}

func UninstallPackage(t *testing.T, packageName string) {
if !IsPackageInstalled(packageName) {
return
}
cmd := exec.Command("brew", "uninstall", "--force", packageName)
if err := cmd.Run(); err != nil {
t.Logf("warning: failed to uninstall %s: %v", packageName, err)
}
}

func EnsurePackageNotInstalled(t *testing.T, packageName string) {
UninstallPackage(t, packageName)
if IsPackageInstalled(packageName) {
t.Fatalf("failed to ensure %s is not installed", packageName)
}
}
3 changes: 0 additions & 3 deletions testutil/machost.go
Original file line number Diff line number Diff line change
Expand Up @@ -102,9 +102,6 @@ func (h *MacHost) CopyFile(src, dst string) error {
return nil
}

// Destroy is a no-op — the CI runner is the sandbox.
func (h *MacHost) Destroy() {}

func shellescape(s string) string {
return "'" + strings.ReplaceAll(s, "'", "'\\''") + "'"
}
Expand Down
Loading