Skip to content

Rollback Procedure

Jump to bottom
Brinda Sasikumar edited this page Apr 27, 2026 · 3 revisions

Rollback Procedure

Use this runbook when a release causes a critical issue in UAT or Production and you need to return to the previous known good state.

Rule #1: Do not improvise under pressure. Follow this document in order.

Step 1 — Assess before acting

Answer these questions before touching anything:

Question Answer guides you to
Is it frontend-only? (UI broken, API responses are fine) Frontend-only rollback
Is the API returning errors but the database looks intact? Full application rollback
Is data corrupted or missing? Database restore from backup — do NOT use migrate:rollback
Did a migration fail mid-run? Partial migration failure
Is the Reverb / WebSocket service down only? Reverb service restart

If data is corrupted or a destructive migration ran — go straight to Database restore from backup. migrate:rollback cannot recover dropped columns or deleted rows.

Step 2 — Identify the previous good tag

git tag --sort=-version:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -5

The tag immediately before the broken release is your rollback target.

Broken release Rollback target
v2.71.0 v2.70.0
v2.70.0 v2.69.3

Frontend-only rollback

Use when: UI is broken, GraphQL API responds correctly, database is intact.

  1. Go to Azure DevOps → Pipelines → GCTalent
  2. Click Run pipeline
  3. Set the ref to the previous release tag (e.g. refs/tags/v2.70.0)
  4. Run — this produces a frontend artifact from the previous tag
  5. Create a release in Azure DevOps from that artifact and deploy to the affected environment
  6. Verify the UI is restored
  7. No database changes needed

Full application rollback

Use when: API errors, database is intact, no data corruption.

1. Check which migrations the rollback involves

# From your local repo — compare migration files between tags
git diff v2.70.0..v2.71.0 --name-only -- 'api/database/migrations/*.php'

Cross-reference with the Migration rollback safety table on the affected release's wiki page (e.g. v2.71.0) before proceeding.

2. Roll back migrations

# Connect to the affected environment's database and check current state first
php artisan migrate:status

# Roll back one step at a time — never use --step=N blindly
php artisan migrate:rollback --step=1
# verify, then continue
php artisan migrate:rollback --step=1
# repeat until back to the target release's migration state

Check migrate:status after each step. Stop if the release wiki page marks a migration as ❌ or ⚠️ — use a database restore instead.

3. Re-deploy the previous artifact

  1. Go to Azure DevOps → Pipelines → GCTalent
  2. Run pipeline against the previous release tag
  3. Deploy the artifact to the affected environment

4. Verify

  • API health check responds
  • Auth flow works (log in as a test user)
  • Core user journey works (job search, application)
  • Check application logs for errors

Database restore from backup

Use when: data is corrupted, a destructive migration ran and data was lost, or migrate:rollback is not safe.

This is the nuclear option. It restores the entire database to a point in time before the failed deployment. All data written after the backup timestamp will be lost.

1. Stop the application

Prevent new writes while restoring.

  1. Azure DevOps → take the environment offline or point traffic to a maintenance page

2. Identify the backup

Find the most recent backup taken before the deployment started.

  • Backups are managed in Azure — confirm the backup timestamp predates the php artisan migrate run
  • Coordinate with your infrastructure team / IAC repo to locate the correct snapshot

3. Restore

Follow Azure Database for PostgreSQL point-in-time restore procedure.

Document the exact restore timestamp and who authorized it here when you run this.

4. Re-deploy the previous artifact

After restore completes:

  1. Run the Azure DevOps pipeline against the previous release tag
  2. Deploy — do not run php artisan migrate (the database is already at the correct schema state)
  3. Run php artisan migrate:status to confirm migration state matches the restored tag

5. Verify and communicate

  • migrate:status shows correct state for the rollback tag
  • Auth flow works
  • Spot-check data integrity (recent records present, no corruption)
  • Notify stakeholders of the data loss window (time of backup → time of incident)

Partial migration failure

Use when: php artisan migrate failed partway through, leaving the database in an intermediate state.

1. Check what ran

php artisan migrate:status

Migrations marked Ran have been applied. Find the last successfully applied migration and the one that failed.

2. If the failed migration is safe to retry

Fix the underlying issue (e.g. null data failing a non-nullable constraint) and re-run:

php artisan migrate

3. If the failed migration left partial state

Some migrations create a table then fail on a subsequent step. Check for orphaned tables or columns:

-- Check if a table was partially created
SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'
ORDER BY table_name;

Manually clean up the partial state, then re-run or roll back as appropriate.

Reverb service restart

Use when: WebSocket subscriptions are not working but the rest of the application is fine.

# Restart the Reverb service on the affected environment
php artisan reverb:restart

# Or if managed as a system service
sudo systemctl restart reverb

Check the Reverb logs for the root cause before restarting in production.

After any rollback — mandatory actions

  • Write an incident note: what broke, what was rolled back, when, who approved
  • Add the rollback event to this wiki under the affected release page
  • Open a follow-up issue in GitHub for the root cause
  • If a database restore was used — document the data loss window and notify affected users if required
  • Update the deployment plan for the re-release to address the root cause

Clone this wiki locally