Batch Mode: Failure Scenarios, Recovery, and Operational Gotchas

[AlexDeMichieli]

Context

During E2E deployment testing of batch mode (submit-repos.yml) on an EMU enterprise (volcano-coffee), we encountered multiple failure scenarios that required manual intervention. This discussion documents each failure mode, its root cause, recovery steps, and whether the proposed architecture in #14 would address it.

Test environment: EMU enterprise, SAML SSO enforced, single org (alexdemichieli-migrations), Copilot Enterprise license.

---

Failure Scenarios

1. PAT-Based Agent Assignment Fails on SSO-Enforced Orgs

Symptom: Every batch-created issue gets: "The agent encountered an error and was unable to start working on this issue: Please try again later, or contact support if the problem persists."

Root cause: Platform bug in github/github#424974 (merged March 27, 2026). When submit-repos.yml assigns copilot-swe-agent via GraphQL using a PAT, the monolith mints an OAuth token but computes the wrong SSO org IDs — the minted token has no access to the target repo.

Who's affected: Any org/enterprise with SAML SSO enabled. Both classic and fine-grained PATs. Affects programmatic assignment only — manual assignment via Copilot Chat UI uses session credentials and works fine.

Fix: github/github#425401 (merged March 31). Rolling out behind feature flag copilot_mint_token_from_pat_with_all_orgs.

Recovery:

• Wait for the fix to ship globally
• Workaround: assign Copilot manually via Copilot Chat UI (manual mode still works)
• If the user has recently used the UI, a cached token with valid SSO grants may allow API assignment to succeed temporarily (30-day cache TTL)

Would #14 fix this? Partially. The agentic workflows proposal eliminates per-repo issue creation via PAT, but the underlying token minting bug would still affect any PAT-based orchestration.

---

2. Ghost Sessions from Failed Assignments

Symptom: github.com/copilot/agents shows N "Active" sessions, but clicking the Active tab shows no sessions. New Copilot assignments fail with the same generic error.

Root cause: Failed agent assignments (from Scenario 1 or transient errors) create phantom sessions that count against concurrency limits but aren't visible or stoppable in the UI.

Recovery:

1. Go to each open issue where Copilot is assigned
2. Unassign Copilot from each stale issue
3. Refresh github.com/copilot/agents — Active count should drop
4. Retry the assignment

Detection: If the Active count > 0 but no sessions are listed, you have ghost sessions.

Would #14 fix this? Yes — agentic workflows wouldn't create per-repo issues, eliminating the ghost session scenario.

---

3. Tag Clear Fails — Repos Re-Processed on Next Batch Run

Symptom: Batch workflow logs show Failed to update custom property for <repo>: Resource not accessible by integration. The GH_MIGRATION_TYPE tag stays set, so the repo is picked up again on the next batch run, creating duplicate issues.

Root cause: The GitHub App has organization_custom_properties: write (for managing the org-level schema) but not custom_properties at the repository level. The endpoint PATCH /repos/{owner}/{repo}/properties/values requires Repository: Custom properties (Read & Write).

Recovery:

1. Add Repository: Custom properties → Read & Write to the GitHub App permissions
2. Re-approve the updated installation
3. Manually clear tags on affected repos:

   gh api --method PATCH /repos/ORG/REPO/properties/values \
     -f 'properties[][property_name]=GH_MIGRATION_TYPE' \
     -f 'properties[][value]=None'

Prevention: Add this permission to the Phase 1 deployment checklist. The upstream deployment.md doesn't mention it.

Would #14 fix this? Yes — the proposal uses a tracking file in .github-private instead of custom properties.

---

4. Enterprise Custom Agents Selector Conflict (Shared Enterprise)

Symptom: Custom agents disappear from github.com/copilot/agents. The agent dropdown is empty when assigning Copilot to issues.

Root cause: Enterprise AI Controls → Custom agents only allows one org to be selected. On a shared enterprise (e.g., volcano-coffee with 133 orgs), another admin selected their org, removing yours.

Recovery:

1. Go to Enterprise → AI Controls → Custom agents
2. Re-select your org from the dropdown
3. Wait 5–10 minutes for propagation

Prevention: Coordinate with enterprise admins. For production use, request a dedicated enterprise or document the shared-enterprise limitation.

Would #14 fix this? No — this is an enterprise configuration issue independent of architecture.

---

5. Duplicate Issues from Partial Batch Failures

Symptom: Multiple identical [Actions Migration] issues on the same repo, each with the full agent prompt as the body.

Root cause: The batch workflow has no idempotency check. If tag clear fails (Scenario 3) and the workflow is re-run, it finds the same tagged repos and creates new issues without checking for existing open migration issues.

Recovery:

1. Close duplicate issues
2. Unassign Copilot from all but the latest
3. Manually clear the tag after the surviving issue completes

Prevention: Add a pre-check to submit-repositories.js: before creating an issue, search for open issues titled [Actions Migration] on the target repo. Skip if one exists.

Would #14 fix this? Yes — eliminates the IssueOps pattern entirely.

---

6. Firewall Allowlist Required on EMU (Per-Repo)

Symptom: Copilot agent session starts but git push returns 403. The agent can read files but can't create branches or push commits.

Root cause: On EMU enterprises, the Copilot coding agent firewall blocks api.github.com and github.com by default. Each target repo needs a custom allowlist.

Recovery:

1. Go to repo Settings → Copilot → Coding agent
2. Add to Custom allowlist: github.com, api.github.com, uploads.github.com

Prevention: The create-environment-secrets.js script in settings.yml already iterates every repo — it could also configure the firewall allowlist programmatically if an API exists.

Would #14 fix this? No — per-repo firewall config is required regardless of architecture.

---

Summary: Architecture Impact

Failure	Current IssueOps	#14 (Agentic Workflows)
SSO token minting bug	Blocked	Still affected if PAT used
Ghost sessions	Common with failures	Eliminated
Tag clear permissions	Manual fix needed	Eliminated (no custom properties)
Enterprise selector conflict	Coordination required	Same
Duplicate issues	No idempotency	Eliminated (no issues)
Firewall allowlist	Per-repo manual	Same

---

Related

• Proposal: Replace IssueOps Orchestration with Agentic Workflows from .github-private #14 — Proposal: Replace IssueOps Orchestration with Agentic Workflows
• #coding-agent-team — Platform bug tracking for SSO token minting
• github/github#425401 — Fix PR for SSO token minting