Batch Mode: Failure Scenarios, Recovery, and Operational Gotchas

[AlexDeMichieli]

Context

During E2E deployment testing of batch mode (submit-repos.yml) on an EMU enterprise (volcano-coffee), I encountered multiple failure scenarios that required manual intervention. This discussion documents each failure mode, its root cause, recovery steps, and whether the proposed architecture in #14 would address it.

Test environment: EMU enterprise, SAML SSO enforced, single org (alexdemichieli-migrations), Copilot Enterprise license.

---

Failure Scenarios

1.Sessions from Failed Assignments

Symptom: github.com/copilot/agents shows N "Active" sessions. New Copilot assignments fail with the same generic error "The agent encountered an error and was unable to start working on this issue: Please try again later, or contact support if the problem persists.".

Recovery:

1. Go to each open issue where Copilot is assigned
2. Unassign Copilot from each stale issue
3. Refresh github.com/copilot/agents. The active count should drop
4. Retry the assignment

Would #14 fix this? Yes, the agentic workflows wouldn't create per-repo issues, eliminating the session scenario.

---

2. Tag Clear Fails. Repos Re-Processed on Next Batch Run

Symptom: Batch workflow logs show Failed to update custom property for <repo>: Resource not accessible by integration. The GH_MIGRATION_TYPE tag stays set, so the repo is picked up again on the next batch run, creating duplicate issues.

Root cause: This happened because I have incorrectly set the app's permissions. The GitHub App had organization_custom_properties: write (for managing the org-level schema) but not custom_properties at the repository level. The endpoint PATCH /repos/{owner}/{repo}/properties/values requires Repository: Custom properties (Read & Write).

Recovery:

1. Added Repository: Custom properties → Read & Write to the GitHub App permissions
2. Re-approved the updated installation
3. Manually cleared tags on affected repos:

   gh api --method PATCH /repos/ORG/REPO/properties/values \
     -f 'properties[][property_name]=GH_MIGRATION_TYPE' \
     -f 'properties[][value]=None'

Prevention: Add this permission to the Phase 1 deployment checklist. The upstream deployment.md doesn't mention it.

Would #14 fix this? Yes. The proposal uses a tracking file in .github-private instead of custom properties.

---

3. Duplicate Issues from Partial Batch Failures

Symptom: Multiple identical [Actions Migration] issues on the same repo, each with the full agent prompt as the body.

Root cause: The batch workflow has no idempotency check. If tag clear fails (Scenario 3) and the workflow is re-run, it finds the same tagged repos and creates new issues without checking for existing open migration issues.

Recovery:

1. Close duplicate issues
2. Unassign Copilot from all but the latest
3. Manually clear the tag after the surviving issue completes

Prevention: Add a pre-check to submit-repositories.js: before creating an issue, search for open issues titled [Actions Migration] on the target repo. Skip if one exists.

Would #14 fix this? Yes — eliminates the IssueOps pattern entirely as the Coding agent will start the process via API, without creating a new issue.

---

Related

• Proposal: Replace IssueOps Orchestration with Agentic Workflows from .github-private #14 — Proposal: Replace IssueOps Orchestration with Agentic Workflows
• #coding-agent-team — Platform bug tracking for SSO token minting
• github/github#425401 — Fix PR for SSO token minting

[AlexDeMichieli]

Closing in favor of #17