Skip to content

Rollback Procedure

Jump to bottom
Brinda Sasikumar edited this page Apr 27, 2026 · 3 revisions

Rollback Procedure

Use this runbook when a release causes a critical issue in UAT or Production and you need to return to the previous known good state.

Rule #1: Do not improvise under pressure. Follow this document in order.

Step 1 — Assess before acting

Answer these questions before touching anything:

Question Answer guides you to
Is it frontend-only? (UI broken, API responses are fine) Frontend-only rollback
Is the API returning errors but the database looks intact? Full application rollback
Is data corrupted or missing? Database restore from backup — do NOT use migrate:rollback
Did a migration fail mid-run? Partial migration failure
Is the Reverb / WebSocket service down only? Reverb service restart

If data is corrupted or a destructive migration ran — go straight to Database restore from backup. migrate:rollback cannot recover dropped columns or deleted rows.

Step 2 — Identify the previous good tag

git tag --sort=-version:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -5

The tag immediately before the broken release is your rollback target.

Broken release Rollback target
v2.71.0 v2.70.0
v2.70.0 v2.69.3

Frontend-only rollback

Use when: UI is broken, GraphQL API responds correctly, database is intact.

  1. Go to Azure DevOps → Pipelines → GCTalent
  2. Click Run pipeline
  3. Set the ref to the previous release tag (e.g. refs/tags/v2.70.0)
  4. Run — this produces a frontend artifact from the previous tag
  5. Create a release in Azure DevOps from that artifact and deploy to the affected environment
  6. Verify the UI is restored
  7. No database changes needed

Full application rollback

Use when: API errors, database is intact, no data corruption.

1. Check which migrations the rollback involves

# From your local repo — compare migration files between tags
git diff v2.70.0..v2.71.0 --name-only -- 'api/database/migrations/*.php'

Cross-reference with the per-release rollback safety table below before proceeding.

2. Roll back migrations

# Connect to the affected environment's database and check current state first
php artisan migrate:status

# Roll back one step at a time — never use --step=N blindly
php artisan migrate:rollback --step=1
# verify, then continue
php artisan migrate:rollback --step=1
# repeat until back to the target release's migration state

Check migrate:status after each step. Stop if you see a migration marked risky rollback in the table below — use a database restore instead.

3. Re-deploy the previous artifact

  1. Go to Azure DevOps → Pipelines → GCTalent
  2. Run pipeline against the previous release tag
  3. Deploy the artifact to the affected environment

4. Verify

  • API health check responds
  • Auth flow works (log in as a test user)
  • Core user journey works (job search, application)
  • Check application logs for errors

Database restore from backup

Use when: data is corrupted, a destructive migration ran and data was lost, or migrate:rollback is not safe.

This is the nuclear option. It restores the entire database to a point in time before the failed deployment. All data written after the backup timestamp will be lost.

1. Stop the application

Prevent new writes while restoring.

  1. Azure DevOps → take the environment offline or point traffic to a maintenance page

2. Identify the backup

Find the most recent backup taken before the deployment started.

  • Backups are managed in Azure — confirm the backup timestamp predates the php artisan migrate run
  • Coordinate with your infrastructure team / IAC repo to locate the correct snapshot

3. Restore

Follow Azure Database for PostgreSQL point-in-time restore procedure.

Document the exact restore timestamp and who authorized it here when you run this.

4. Re-deploy the previous artifact

After restore completes:

  1. Run the Azure DevOps pipeline against the previous release tag
  2. Deploy — do not run php artisan migrate (the database is already at the correct schema state)
  3. Run php artisan migrate:status to confirm migration state matches the restored tag

5. Verify and communicate

  • migrate:status shows correct state for the rollback tag
  • Auth flow works
  • Spot-check data integrity (recent records present, no corruption)
  • Notify stakeholders of the data loss window (time of backup → time of incident)

Partial migration failure

Use when: php artisan migrate failed partway through, leaving the database in an intermediate state.

1. Check what ran

php artisan migrate:status

Migrations marked Ran have been applied. Find the last successfully applied migration and the one that failed.

2. If the failed migration is safe to retry

Fix the underlying issue (e.g. null data failing a non-nullable constraint) and re-run:

php artisan migrate

3. If the failed migration left partial state

Some migrations create a table then fail on a subsequent step. Check for orphaned tables or columns:

-- Check if a table was partially created
SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'
ORDER BY table_name;

Manually clean up the partial state, then re-run or roll back as appropriate.

Reverb service restart

Use when: WebSocket subscriptions are not working but the rest of the application is fine.

# Restart the Reverb service on the affected environment
php artisan reverb:restart

# Or if managed as a system service
sudo systemctl restart reverb

Check the Reverb logs for the root cause before restarting in production.

Per-release migration rollback safety

Use this table when deciding whether migrate:rollback is safe or a database restore is needed.

v2.71.0 → v2.70.0

Migration Safe to rollback? Notes
2026_04_16_165303_create_development_program_user ✅ Yes Drops development_program_user table — no user-facing data loss
2026_04_10_162756_update_development_program_nullable ✅ Yes Makes columns nullable again — safe
2026_04_10_141222_add_development_program_abbreviation ✅ Yes Drops column — data loss only if abbreviations were entered
2026_04_10_141210_add_development_program_information_url ✅ Yes Drops column — data loss only if URLs were entered
2026_04_10_171836_add_last_sign_in_iss ✅ Yes Drops column — non-critical data
2026_04_08_180346_update_development_program_interests ⚠️ Partial Drops community_development_program_id — any data written to it after deploy is lost

v2.70.0 → v2.69.3

Migration Safe to rollback? Notes
2026_04_07_174344_create_additional_community_development_program_pivots ✅ Yes Drops two pivot tables
2026_04_02_184012_preset_dimissible_announcement ✅ Yes Removes updatedAt and isDismissible from settings JSON
2026_03_30_200030_create_community_development_program_pivot ⚠️ Caution Reverses community_id to non-nullable — will fail if any rows have null
2026_03_30_190143_add_community_contact_email ✅ Yes Drops column
2026_03_30_183810_add_community_information_url ✅ Yes Drops column
2026_03_24_133218_drop_legacy_candidate_status_columns ❌ Do not rollback down() repopulates from lossy mapping — use database restore instead
2026_03_01_024542_add_referral_pause_columns_to_pool_candidates ✅ Yes Drops three columns

After any rollback — mandatory actions

  • Write an incident note: what broke, what was rolled back, when, who approved
  • Add the rollback event to this wiki under the affected release page
  • Open a follow-up issue in GitHub for the root cause
  • If a database restore was used — document the data loss window and notify affected users if required
  • Update the deployment plan for the re-release to address the root cause

Contacts

Role Action
Database restore Coordinate with infrastructure team via IAC repo
Azure DevOps pipeline Release manager runs the pipeline
Stakeholder communication Release manager notifies

Clone this wiki locally