🩹 [Patch]: Integration tests no longer collide and Organizations enterprise flow is reliable

[Marius Storhaug (MariusStorhaug)]

Integration tests no longer collide when run in parallel across test files. Releases tests survive workflow reruns. The Organizations enterprise auth case can reliably create an org, install the app, and delete the org — across reruns as well as first-time runs.

• Fixes Use per-test-file scoped repositories and self-contained idempotent test setup #595
• Fixes Fix Install-GitHubApp 404 in Organizations tests — missing enterprise_organization_installations permission #596

Changed: Each test file uses its own repository

Test files no longer share a single Test-{OS}-{TokenType}-{RunID} repository. Each test file now operates on a per-test-file repository scoped by test name: {TestName}-{OS}-{TokenType}-{RunID}. This eliminates cross-file resource collisions when test files run in parallel across OSes and in sequence across auth contexts.

Resource	Pattern	Example
Repo	{TestName}-{OS}-{TokenType}-{RunID}	Releases-Linux-USER_FG_PAT-1234
Extra repo	{TestName}-{OS}-{TokenType}-{RunID}-{N}	Secrets-Linux-ORG_FG_PAT-1234-2

Affected test files: Environments, Secrets, Variables, Releases, Actions, plus TEMPLATE.ps1. Repositories.Tests.ps1 already managed its own repos and is unchanged. Permissions.Tests.ps1 does not use a repository and is unchanged.

Fixed: Environments tests no longer fail because of foreign environments

Environments.Tests.ps1 previously asserted that the shared repository contained no environments, but Secrets.Tests.ps1 and Variables.Tests.ps1 were creating their own environments on that same repository and never cleaning them up. With per-test-file repositories, each test file now sees only the environments it owns. As a defense-in-depth measure, Secrets and Variables AfterAll now also remove the environment they create.

Fixed: Releases tests now pass on workflow reruns

Releases.Tests.ps1 previously failed on workflow reruns (same GITHUB_RUN_ID) with 422 Validation Failed (already_exists) because release tags from the prior attempt persisted on the shared repository. The per-context BeforeAll now removes any pre-existing releases on the test repository before creating new ones, making the test idempotent across reruns.

Fixed: Organizations enterprise auth case is now reliable across reruns

Three separate problems caused the enterprise auth case to fail, and all three are fixed:

1 — Test ordering: Remove-GitHubOrganization (enterprise IAT, Should -Throw) was running before Install-GitHubApp. If the enterprise app unexpectedly had organization_administration: write, it would delete the org and cause Install-GitHubApp to return 404. The Should -Throw test is now placed after Install-GitHubApp and the org-IAT tests, so the org is guaranteed to exist during the install step. The throw is still expected — DELETE /orgs/{org} requires org-level administration: write, which the enterprise IAT does not carry by design.

2 — Propagation delay: After creating an enterprise organization, the enterprise apps endpoint can return 404 before the new org has propagated. Install-GitHubApp now retries up to 5 times with a 3-second delay between attempts.

3 — 90-day org name hold: Deleted GitHub organizations are unavailable for 90 days. The old code used Get-Random which avoided this; the switch to GITHUB_RUN_ID caused reruns of the same workflow run to collide with the held name. The org name now appends GITHUB_RUN_ATTEMPT on reruns (attempt 2 → {id}-2, attempt 3 → {id}-3, etc.), guaranteeing a fresh name per attempt. Attempt 1 is unchanged.

As a safety net, BeforeAll also detects a stale org from a failed prior attempt (same run ID, same attempt number) and removes it before proceeding.

New-GitHubOrganization now has explicit assertions so a silent partial-error response (GraphQL UNPROCESSABLE) fails the test immediately rather than cascading into four downstream failures.

Technical Details

• tests/BeforeAll.ps1: Provisions one repository per (TestName, OS, AuthCase) combination instead of one shared repository per (OS, AuthCase). The $testNames array is the single source of truth for which test files require a repository. Companion -2/-3 repositories are now provisioned only for $testNamesWithExtraRepos (Secrets, Variables).
• tests/AfterAll.ps1: Mirrors BeforeAll.ps1 — tears down the same set of per-test-file repositories. The hard-coded $prefix = 'Test' is removed.
• tests/Environments.Tests.ps1, tests/Secrets.Tests.ps1, tests/Variables.Tests.ps1, tests/Releases.Tests.ps1, tests/Actions.Tests.ps1, tests/TEMPLATE.ps1: Repo prefix changed from "Test-$os-$TokenType" to "$testName-$os-$TokenType".
• tests/Releases.Tests.ps1: Per-context BeforeAll calls Get-GitHubRelease -AllVersions | Remove-GitHubRelease before creating new releases.
• tests/Secrets.Tests.ps1, tests/Variables.Tests.ps1: AfterAll now removes the test environment created during the run.
• tests/Organizations.Tests.ps1: Install-GitHubApp wrapped in a 5-attempt retry loop; Remove-GitHubOrganization (enterprise IAT) moved to after the org-IAT tests; New-GitHubOrganization now asserts $org.Name; GITHUB_RUN_ATTEMPT appended to $id on reruns; BeforeAll detects and removes stale orgs from failed prior attempts.
• .github/instructions/tests.instructions.md: APP_ENT section documents enterprise_organization_installations: write as a required permission and explains why GitHub returns 404 (not 403) when it is missing.
• Implementation plan progress (Use per-test-file scoped repositories and self-contained idempotent test setup #595): All tasks completed.
• Implementation plan progress (Fix Install-GitHubApp 404 in Organizations tests — missing enterprise_organization_installations permission #596): All code and documentation tasks completed. Enabling enterprise_organization_installations: write on the psmodule-enterprise-app is a GitHub app settings step, not a code change.

[github-actions]

No Significant Changes Detected

This PR does not contain changes to files that would trigger a new release:

Pattern	Description
^src/	Matches files where path matches this pattern
^README\.md$	Matches files where path matches this pattern

Build, test, and publish stages will be skipped for this PR.

If you believe this is incorrect, please verify that your changes are in the correct locations.

[github-actions]

No Significant Changes Detected

This PR does not contain changes to files that would trigger a new release:

Pattern	Description
^src/	Matches files where path matches this pattern
^README\.md$	Matches files where path matches this pattern

Build, test, and publish stages will be skipped for this PR.

If you believe this is incorrect, please verify that your changes are in the correct locations.

[github-actions]

Super-linter summary

Language	Validation result
CHECKOV	Pass ✅
GITHUB_ACTIONS	Pass ✅
GITLEAKS	Pass ✅
GIT_MERGE_CONFLICT_MARKERS	Pass ✅
MARKDOWN	Pass ✅
NATURAL_LANGUAGE	Pass ✅
PRE_COMMIT	Pass ✅
SPELL_CODESPELL	Pass ✅
TRIVY	Pass ✅
YAML	Pass ✅

All files and directories linted successfully

For more information, see the GitHub Actions workflow run

Powered by Super-linter

[github-actions]

No Significant Changes Detected

This PR does not contain changes to files that would trigger a new release:

Pattern	Description
^src/	Matches files where path matches this pattern
^README\.md$	Matches files where path matches this pattern

Build, test, and publish stages will be skipped for this PR.

If you believe this is incorrect, please verify that your changes are in the correct locations.

[github-actions]

Super-linter summary

Language	Validation result
CHECKOV	Pass ✅
GITHUB_ACTIONS	Pass ✅
GITLEAKS	Pass ✅
GIT_MERGE_CONFLICT_MARKERS	Pass ✅
MARKDOWN	Pass ✅
NATURAL_LANGUAGE	Pass ✅
PRE_COMMIT	Pass ✅
SPELL_CODESPELL	Pass ✅
TRIVY	Pass ✅
YAML	Pass ✅

All files and directories linted successfully

For more information, see the GitHub Actions workflow run

Powered by Super-linter

[github-actions]

No Significant Changes Detected

This PR does not contain changes to files that would trigger a new release:

Pattern	Description
^src/	Matches files where path matches this pattern
^README\.md$	Matches files where path matches this pattern

Build, test, and publish stages will be skipped for this PR.

If you believe this is incorrect, please verify that your changes are in the correct locations.

[github-actions]

Super-linter summary

Language	Validation result
CHECKOV	Pass ✅
GITHUB_ACTIONS	Pass ✅
GITLEAKS	Pass ✅
GIT_MERGE_CONFLICT_MARKERS	Pass ✅
MARKDOWN	Pass ✅
NATURAL_LANGUAGE	Pass ✅
PRE_COMMIT	Pass ✅
SPELL_CODESPELL	Pass ✅
TRIVY	Pass ✅
YAML	Pass ✅

All files and directories linted successfully

For more information, see the GitHub Actions workflow run

Powered by Super-linter

[github-actions]

No Significant Changes Detected

This PR does not contain changes to files that would trigger a new release:

Pattern	Description
^src/	Matches files where path matches this pattern
^README\.md$	Matches files where path matches this pattern

Build, test, and publish stages will be skipped for this PR.

If you believe this is incorrect, please verify that your changes are in the correct locations.

[github-actions]

Super-linter summary

Language	Validation result
CHECKOV	Pass ✅
GITHUB_ACTIONS	Pass ✅
GITLEAKS	Pass ✅
GIT_MERGE_CONFLICT_MARKERS	Pass ✅
MARKDOWN	Pass ✅
NATURAL_LANGUAGE	Pass ✅
PRE_COMMIT	Pass ✅
SPELL_CODESPELL	Pass ✅
TRIVY	Pass ✅
YAML	Pass ✅

All files and directories linted successfully

For more information, see the GitHub Actions workflow run

Powered by Super-linter

[github-actions]

No Significant Changes Detected

This PR does not contain changes to files that would trigger a new release:

Pattern	Description
^src/	Matches files where path matches this pattern
^README\.md$	Matches files where path matches this pattern

Build, test, and publish stages will be skipped for this PR.

If you believe this is incorrect, please verify that your changes are in the correct locations.

[github-actions]

Super-linter summary

Language	Validation result
CHECKOV	Pass ✅
GITHUB_ACTIONS	Pass ✅
GITLEAKS	Pass ✅
GIT_MERGE_CONFLICT_MARKERS	Pass ✅
MARKDOWN	Pass ✅
NATURAL_LANGUAGE	Pass ✅
PRE_COMMIT	Pass ✅
SPELL_CODESPELL	Pass ✅
TRIVY	Pass ✅
YAML	Pass ✅

All files and directories linted successfully

For more information, see the GitHub Actions workflow run

Powered by Super-linter

[github-actions]

No Significant Changes Detected

This PR does not contain changes to files that would trigger a new release:

Pattern	Description
^src/	Matches files where path matches this pattern
^README\.md$	Matches files where path matches this pattern

Build, test, and publish stages will be skipped for this PR.

If you believe this is incorrect, please verify that your changes are in the correct locations.

[github-actions]

Super-linter summary

Language	Validation result
CHECKOV	Pass ✅
GITHUB_ACTIONS	Pass ✅
GITLEAKS	Pass ✅
GIT_MERGE_CONFLICT_MARKERS	Pass ✅
MARKDOWN	Pass ✅
NATURAL_LANGUAGE	Pass ✅
PRE_COMMIT	Pass ✅
SPELL_CODESPELL	Pass ✅
TRIVY	Pass ✅
YAML	Pass ✅

All files and directories linted successfully

For more information, see the GitHub Actions workflow run

Powered by Super-linter

[Copilot]

Pull request overview

This PR updates the integration test infrastructure to reduce cross-test interference during parallel runs and improve idempotency on workflow reruns, plus documents required enterprise app permissions for the Organizations enterprise flow.

Changes:

• Switch repository naming from a shared run-scoped repo to per-test-file run-scoped repos ({TestName}-{OS}-{TokenType}-{RunID}) across affected test files and global setup/teardown.
• Add targeted cleanup to improve rerun-idempotency (remove stale releases; remove environments created by Secrets/Variables tests).
• Strengthen Organizations enterprise-org creation assertions and document the enterprise permission required for app installation on enterprise-owned orgs.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/BeforeAll.ps1	Provisions per-test-file repos (and extra repos for Secrets/Variables org cases) instead of a single shared repo.
tests/AfterAll.ps1	Mirrors global teardown to remove the per-test-file repos created during setup.
tests/Environments.Tests.ps1	Uses per-test-file repo prefix to avoid foreign environment collisions.
tests/Secrets.Tests.ps1	Uses per-test-file repo prefix and removes the created environment in AfterAll to prevent rerun leakage.
tests/Variables.Tests.ps1	Uses per-test-file repo prefix and removes the created environment in AfterAll to prevent rerun leakage.
tests/Releases.Tests.ps1	Uses per-test-file repo prefix and removes pre-existing releases on the test repo to make reruns idempotent.
tests/Actions.Tests.ps1	Uses per-test-file repo prefix to isolate repo-scoped OIDC tests.
tests/TEMPLATE.ps1	Updates template guidance to use per-test-file repositories.
tests/Organizations.Tests.ps1	Adds run-attempt suffixing for enterprise org names, adds stale-org cleanup, and asserts org creation output more strictly.
.github/instructions/tests.instructions.md	Documents the required enterprise_organization_installations: write permission and why missing perms can appear as 404s.

[github-actions]

No Significant Changes Detected

This PR does not contain changes to files that would trigger a new release:

Pattern	Description
^src/	Matches files where path matches this pattern
^README\.md$	Matches files where path matches this pattern

Build, test, and publish stages will be skipped for this PR.

If you believe this is incorrect, please verify that your changes are in the correct locations.

[github-actions]

Super-linter summary

Language	Validation result
CHECKOV	Pass ✅
GITHUB_ACTIONS	Pass ✅
GITLEAKS	Pass ✅
GIT_MERGE_CONFLICT_MARKERS	Pass ✅
MARKDOWN	Pass ✅
NATURAL_LANGUAGE	Pass ✅
PRE_COMMIT	Pass ✅
SPELL_CODESPELL	Pass ✅
TRIVY	Pass ✅
YAML	Pass ✅

All files and directories linted successfully

For more information, see the GitHub Actions workflow run

Powered by Super-linter

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

[github-actions]

No Significant Changes Detected

This PR does not contain changes to files that would trigger a new release:

Pattern	Description
^src/	Matches files where path matches this pattern
^README\.md$	Matches files where path matches this pattern

Build, test, and publish stages will be skipped for this PR.

If you believe this is incorrect, please verify that your changes are in the correct locations.

[github-actions]

Super-linter summary

Language	Validation result
CHECKOV	Pass ✅
GITHUB_ACTIONS	Pass ✅
GITLEAKS	Pass ✅
GIT_MERGE_CONFLICT_MARKERS	Pass ✅
MARKDOWN	Pass ✅
NATURAL_LANGUAGE	Pass ✅
PRE_COMMIT	Pass ✅
SPELL_CODESPELL	Pass ✅
TRIVY	Pass ✅
YAML	Pass ✅

All files and directories linted successfully

For more information, see the GitHub Actions workflow run

Powered by Super-linter