Use per-test-file scoped repositories and self-contained idempotent test setup

[MariusStorhaug]

Integration tests that share repository infrastructure experience cross-test pollution and stale state failures when running in parallel or on workflow re-runs. Two distinct failure patterns were observed in run 25260449736:

Environment overlap — Environments.Tests.ps1 expects an initially empty repository but finds environments created by Secrets.Tests.ps1 and Variables.Tests.ps1. Both Secrets and Variables tests call Set-GitHubEnvironment on the shared Test-{OS}-{TokenType}-{RunID} repository (creating Secrets-{OS}-{TokenType}-{RunID} and Variables-{OS}-{TokenType}-{RunID} environments respectively) but neither test file ever calls Remove-GitHubEnvironment to clean them up. Since all three test files run in parallel across OSes and in sequence across auth contexts on the same shared repository, the Environments tests see foreign environments and fail assertions like "should return an empty list when no environments exist" and "should list one remaining environment."

Stale releases on re-runs — Releases.Tests.ps1 fails on workflow re-runs (same GITHUB_RUN_ID) because releases, tags, and assets from the prior attempt persist on the shared repository. Tags like v1.1 and v1.3 already exist, causing 422 Validation Failed (already_exists) errors. Subsequent assertions about "latest release" and asset counts also fail because the state is cumulative across attempts.

These are both symptoms of the same architectural gap: test files share a single repository but make incompatible assumptions about its state.

Request

Desired capability

Each test file operates on its own dedicated repository, scoped by test name, eliminating all cross-file resource collisions. Each test file is self-contained: it provisions the resources it needs, cleans up stale resources from prior runs, and tears down after itself. Any individual test file or auth context can be re-run independently without depending on clean state from other test files or prior workflow attempts.

Acceptance criteria

• Environments.Tests.ps1 passes regardless of whether Secrets.Tests.ps1 or Variables.Tests.ps1 have run or are running concurrently
• Releases.Tests.ps1 passes on workflow re-runs with the same GITHUB_RUN_ID
• No test file creates resources that affect another test file's assertions
• The pattern scales across all parallel dimensions: OS (Linux, macOS, Windows) × auth context (7 cases) × test file — with zero cross-file interference
• BeforeAll.ps1 and AfterAll.ps1 provision and tear down per-test-file repositories
• Test instructions (tests.instructions.md) are updated to document the per-test repo naming convention and self-containment principle

---

Technical decisions

Repository scoping — per-test-file repos: Instead of all test files sharing a single Test-{OS}-{TokenType}-{RunID} repository, each test file gets its own {TestName}-{OS}-{TokenType}-{RunID} repository. This eliminates cross-file resource collisions entirely. Examples:

• Environments-Linux-USER_FG_PAT-12345678
• Secrets-Linux-USER_FG_PAT-12345678
• Releases-macOS-APP_ORG-12345678

Extra repos for org-scoped tests: Secrets.Tests.ps1 and Variables.Tests.ps1 currently need companion -2 and -3 repositories for SelectedRepository tests. These become {TestName}-{OS}-{TokenType}-{RunID}-2 and -3.

Self-contained setup/teardown principle: Each test file is responsible for:

1. BeforeAll (per-context): Ensure its repository exists via Set-GitHubRepository (idempotent get-or-create). Clean up any stale test-specific resources (releases, environments, etc.) from prior runs of the same GITHUB_RUN_ID.
2. AfterAll (per-context): Remove all test-specific resources created during the run. Disconnect all GitHub contexts.

Idempotent cleanup in BeforeAll: For Releases.Tests.ps1, the per-context BeforeAll should remove all existing releases on its repository before creating new ones. For Secrets.Tests.ps1 and Variables.Tests.ps1, they already clean up org-scoped resources by prefix — they should also clean up the environments they create on the repository.

Global setup (BeforeAll.ps1): Updated to provision per-test-file repos instead of shared repos. The list of test names that need repos should be derived from the test files themselves or defined as a configuration array. Each test file's BeforeAll also calls Set-GitHubRepository as a safety net, so the global setup is an optimization rather than a hard dependency.

Global teardown (AfterAll.ps1): Updated to remove per-test-file repos by their deterministic names.

Test files that need their own repo:

Test file	Repo name pattern	Extra repos
Environments.Tests.ps1	Environments-{OS}-{TokenType}-{RunID}	None
Secrets.Tests.ps1	Secrets-{OS}-{TokenType}-{RunID}	-2, -3 (org only)
Variables.Tests.ps1	Variables-{OS}-{TokenType}-{RunID}	-2, -3 (org only)
Releases.Tests.ps1	Releases-{OS}-{TokenType}-{RunID}	None
Actions.Tests.ps1	Actions-{OS}-{TokenType}-{RunID}	None
Permissions.Tests.ps1	Permissions-{OS}-{TokenType}-{RunID}	None

Test files that do NOT need changes:

Test file	Reason
Repositories.Tests.ps1	Already creates its own repos (Repositories-{OS}-{TokenType}-{RunID})
Teams.Tests.ps1	Does not use repos — operates on orgs only
Organizations.Tests.ps1	Creates its own enterprise orgs
Apps.Tests.ps1	Uses auth context only, no test repo
Artifacts.Tests.ps1	Uses $env:GITHUB_REPOSITORY (current repo)
Emojis.Tests.ps1	Read-only, no repos
Users.Tests.ps1	Read-only, no repos
Enterprise.Tests.ps1	Enterprise-scoped, no repos
GitHub.Tests.ps1	Auth context tests only

Naming convention update: The naming convention table in tests.instructions.md should be updated to reflect the per-test pattern:

Resource	Pattern	Example
Repo	{TestName}-{OS}-{TokenType}-{RunID}	Releases-Linux-USER_FG_PAT-1234
Extra repo	{TestName}-{OS}-{TokenType}-{RunID}-{N}	Secrets-Linux-ORG_FG_PAT-1234-2

No breaking change to test isolation guarantees: The run-scoped naming guarantee from #541 is preserved — repos are still scoped by GITHUB_RUN_ID. This change adds a {TestName} dimension to prevent cross-file collisions within the same run.

---

Implementation plan

Core changes

• Update BeforeAll.ps1 to provision per-test-file repos (one per test name × OS × auth case) instead of shared Test-* repos
• Update AfterAll.ps1 to tear down per-test-file repos by their deterministic names
• Update Environments.Tests.ps1 to use repo name Environments-{OS}-{TokenType}-{RunID}
• Update Secrets.Tests.ps1 to use repo name Secrets-{OS}-{TokenType}-{RunID} (plus -2, -3 for org)
• Update Variables.Tests.ps1 to use repo name Variables-{OS}-{TokenType}-{RunID} (plus -2, -3 for org)
• Update Releases.Tests.ps1 to use repo name Releases-{OS}-{TokenType}-{RunID}
• Update Actions.Tests.ps1 to use repo name Actions-{OS}-{TokenType}-{RunID}
• Update Permissions.Tests.ps1 to use repo name Permissions-{OS}-{TokenType}-{RunID}

Self-contained cleanup

• Add release cleanup in Releases.Tests.ps1 per-context BeforeAll: remove all existing releases before creating test releases
• Add environment cleanup in Secrets.Tests.ps1 AfterAll: remove the Secrets-{OS}-{TokenType}-{RunID} environment it creates
• Add environment cleanup in Variables.Tests.ps1 AfterAll: remove the Variables-{OS}-{TokenType}-{RunID} environment it creates

Documentation

• Update tests.instructions.md naming convention table to use {TestName}-{OS}-{TokenType}-{RunID} pattern
• Update tests.instructions.md "Shared test repositories" section to describe per-test-file repo model
• Add self-containment principle to tests.instructions.md: each test file provisions its own resources, cleans up stale state in BeforeAll, and tears down in AfterAll
• Update tests.instructions.md "Test file structure" code example to reflect the new naming pattern