Skip to content

Update tests.instructions.md to document per-test-file repository model #597

@MariusStorhaug

Description

The test instructions file (.github/instructions/tests.instructions.md) documents the integration test architecture for contributors and automated agents. It currently describes a shared Test-{OS}-{TokenType}-{RunID} repository model, which does not reflect the correct per-test-file scoping needed to prevent cross-file resource collisions during parallel execution.

Request

What is confusing or missing

The instructions file tells contributors that all test files share a single repository per OS/auth/run combination. This is incorrect — each test file should own its own repository to avoid cross-file interference. A developer following the current instructions would write tests that conflict with other test files when running in parallel.

Acceptance criteria

  • The "Shared test repositories" section is renamed to "Per-test-file repositories" and describes the correct architecture
  • The repository naming pattern is updated from Test-{OS}-{TokenType}-{RunID} to {TestName}-{OS}-{TokenType}-{RunID} in all code examples and the naming conventions table
  • The "Setup and teardown" section describes per-test-file provisioning: each test file ensures its own repository exists, cleans up stale resources from prior runs, and tears down after itself
  • The "Test file structure" code example includes stale resource cleanup in BeforeAll and resource teardown in AfterAll
  • The "Key rules" section documents the self-containment principle: each test file owns its repo, handles idempotent setup, cleans up after itself, and must work independently across parallel OSes and sequential auth contexts
  • The frontmatter description reflects the updated scope

Technical decisions

Scope: Documentation-only change to .github/instructions/tests.instructions.md. No test code changes — those are tracked in #595.

Branch: The changes have already been authored on branch docs/test-instructions-per-test-repos.

Naming pattern update: All occurrences of Test-{OS}-{TokenType}-{RunID} become {TestName}-{OS}-{TokenType}-{RunID}. The naming conventions table gains:

Resource Pattern Example
Repo {TestName}-{OS}-{TokenType}-{RunID} Releases-Linux-USER_FG_PAT-1234
Extra repo {TestName}-{OS}-{TokenType}-{RunID}-{N} Secrets-Linux-ORG_FG_PAT-1234-2

Self-containment principle: Each test file is responsible for:

  1. Ensuring its repository exists (idempotent Set-GitHubRepository in BeforeAll)
  2. Cleaning up stale resources from prior runs of the same GITHUB_RUN_ID
  3. Tearing down test-specific resources in AfterAll

Relationship to other issues: This is a documentation sub-task of #595. The actual test code changes that implement the per-test-file model are in #595. The Organizations Install-GitHubApp 404 fix is in #596.


Implementation plan

  • Rename "Shared test repositories" section to "Per-test-file repositories"
  • Update repo naming pattern from Test-{OS}-{TokenType}-{RunID} to {TestName}-{OS}-{TokenType}-{RunID} in all code examples
  • Update naming conventions table with per-test-file pattern and examples
  • Rewrite "Setup and teardown" section to describe per-test-file provisioning instead of shared provisioning
  • Update "Test file structure" code example to include stale resource cleanup in BeforeAll and resource teardown in AfterAll
  • Update "Key rules" section with self-containment principle
  • Update frontmatter description

Metadata

Metadata

Labels

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions