-
Notifications
You must be signed in to change notification settings - Fork 0
Runbook Rollback
CYPT71 edited this page Aug 21, 2026
·
2 revisions
Mirrored from
docs/runbooks/rollback.mdin the repository, 2026-08-05. The repo copy is authoritative for anything CI-cited (exact commands, workflow files, versions) — this page exists so the wiki is self-contained, but re-sync it from the source if the two drift.
This document provides step-by-step procedures for rolling back deployments and reverting to previous versions of platform-factory components.
- General Rollback Principles
- Version Identification
- Component-Specific Rollback Procedures
- Rollback Testing
- Post-Rollback Procedures
Rollback should be performed when:
- A deployment causes critical failures
- A new version introduces security vulnerabilities
- Performance degrades below acceptable levels
- Data corruption is detected
- User impact is severe and immediate
┌─────────────────────────────────────┐
│ Is the issue critical? │
└───────────────────┬─────────────────┘
│
Yes │ No
│
▼ │ ▼
┌─────────────────────────────────────┐
│ Can it be fixed with a hot patch? │
└───────────────────┬─────────────────┘
│
No │ Yes
│
▼ │ ▼
┌─────────────────────────────────────┐
│ Rollback to previous version │ Apply hot patch
└─────────────────────────────────────┘
Before performing a rollback, verify:
- The previous version is known to be stable
- The previous version's artifacts are available
- Data migration (if any) can be reversed
- Rollback procedure has been tested
- Stakeholders have been notified
- Monitoring is in place to verify rollback success
# Check platform-factory version
./platform-factory version
# Check control plane version
./platform-factory-control-plane version
# Check worker version
./platform-factory-worker version# List available versions
git tag | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V
# Check what's installed
ls -la /usr/local/bin/platform-factory*
# Check release assets
ls -la dist/| Component | Current | Previous | Compatible |
|---|---|---|---|
| platform-factory | v4.0.0 | v3.2.1 | ✅ Yes |
| control-plane | v4.0.0 | v3.2.1 | ✅ Yes |
| worker | v4.0.0 | v3.2.1 | ✅ Yes |
Note: Always roll back all components to the same version to ensure compatibility.
# List installed versions
./scripts/list-installed-versions.sh
# Rollback to previous version
sudo ./scripts/rollback-cli.sh v3.2.1
# Verify rollback
./platform-factory version# 1. Download previous version
PREV_VERSION="v3.2.1"
wget https://github.com/CYPT71/platform-factory/releases/download/${PREV_VERSION}/platform-factory-${PREV_VERSION}-$(uname -s)-$(uname -m)
# 2. Install previous version
chmod +x platform-factory-${PREV_VERSION}-*
sudo mv platform-factory-${PREV_VERSION}-* /usr/local/bin/platform-factory
# 3. Verify installation
./platform-factory version
# 4. Clean up
rm -f platform-factory-${PREV_VERSION}-*# Backup current configuration
cp -r ~/.config/platform-factory ~/.config/platform-factory-backup-$(date +%Y%m%d-%H%M%S)
# Restore previous configuration (if available)
cp -r ~/.config/platform-factory-backups/v3.2.1 ~/.config/platform-factory
# Verify configuration
./platform-factory config validate# 1. Stop current service
sudo systemctl stop platform-factory-control-plane
# 2. Rollback binary
sudo cp /usr/local/bin/platform-factory-control-plane-backups/platform-factory-control-plane-v3.2.1 /usr/local/bin/platform-factory-control-plane
# 3. Reload systemd
sudo systemctl daemon-reload
# 4. Start service
sudo systemctl start platform-factory-control-plane
# 5. Verify
sudo systemctl status platform-factory-control-plane
./platform-factory-control-plane version# 1. Stop current container
docker stop platform-factory-control-plane
# 2. Remove current container
docker rm platform-factory-control-plane
# 3. Pull previous version
docker pull ghcr.io/cypt71/platform-factory-control-plane:v3.2.1
# 4. Start previous version
docker run -d \
--name platform-factory-control-plane \
-v /path/to/config:/config \
-v /path/to/state:/state \
--network host \
ghcr.io/cypt71/platform-factory-control-plane:v3.2.1
# 5. Verify
docker logs platform-factory-control-plane# 1. Check deployment history
kubectl rollout history deployment/platform-factory-control-plane
# 2. Rollback to previous revision
kubectl rollout undo deployment/platform-factory-control-plane
# 3. Or rollback to specific revision
kubectl rollout undo deployment/platform-factory-control-plane --to-revision=2
# 4. Monitor rollout status
kubectl rollout status deployment/platform-factory-control-plane
# 5. Verify
kubectl get pods -l app=platform-factory-control-plane# 1. Identify worker
./platform-factory-worker id
# 2. Stop worker
sudo systemctl stop platform-factory-worker@<worker-id>
# 3. Rollback binary
sudo cp /usr/local/bin/platform-factory-worker-backups/platform-factory-worker-v3.2.1 /usr/local/bin/platform-factory-worker
# 4. Start worker
sudo systemctl start platform-factory-worker@<worker-id>
# 5. Verify
sudo systemctl status platform-factory-worker@<worker-id>
./platform-factory-worker version# 1. Stop all workers
for worker in $(systemctl list-units --type=service | grep platform-factory-worker | awk '{print $1}'); do
sudo systemctl stop $worker
done
# 2. Rollback binary for all
sudo cp /usr/local/bin/platform-factory-worker-backups/platform-factory-worker-v3.2.1 /usr/local/bin/platform-factory-worker
# 3. Start all workers
for worker in $(systemctl list-units --type=service | grep platform-factory-worker | awk '{print $1}'); do
sudo systemctl start $worker
done
# 4. Verify all workers
./scripts/verify-workers.sh# 1. Check deployment history
kubectl rollout history deployment/platform-factory-worker
# 2. Rollback workers
kubectl rollout undo deployment/platform-factory-worker
# 3. Monitor rollout
kubectl rollout status deployment/platform-factory-worker
# 4. Verify workers are ready
kubectl get pods -l app=platform-factory-workerWARNING: Database rollback can cause data loss. Only perform if you have a backup and understand the implications.
# 1. Backup current state
./platform-factory cas backup /path/to/backup-$(date +%Y%m%d-%H%M%S)
# 2. Restore from previous backup
./platform-factory cas restore /path/to/backup-v3.2.1
# 3. Verify consistency
./platform-factory cas verify# 1. Backup current state
cp /var/lib/platform-factory/control-plane-state.json /var/lib/platform-factory/control-plane-state.json.backup-$(date +%Y%m%d-%H%M%S)
# 2. Restore previous state
cp /var/lib/platform-factory/backups/control-plane-state-v3.2.1.json /var/lib/platform-factory/control-plane-state.json
# 3. Set correct permissions
chmod 600 /var/lib/platform-factory/control-plane-state.json
chown platform-factory:platform-factory /var/lib/platform-factory/control-plane-state.json
# 4. Restart control plane
sudo systemctl restart platform-factory-control-plane# 1. Stop all services
sudo systemctl stop platform-factory-control-plane
for worker in $(systemctl list-units --type=service | grep platform-factory-worker | awk '{print $1}'); do
sudo systemctl stop $worker
done
# 2. Rollback all binaries
sudo ./scripts/rollback-all.sh v3.2.1
# 3. Restore all configurations
sudo ./scripts/restore-configs.sh v3.2.1
# 4. Restore state (if needed)
sudo ./scripts/restore-state.sh v3.2.1
# 5. Start all services
sudo systemctl start platform-factory-control-plane
for worker in $(systemctl list-units --type=service | grep platform-factory-worker | awk '{print $1}'); do
sudo systemctl start $worker
done
# 6. Verify all components
./scripts/verify-installation.sh# 1. Check system health before rollback
./scripts/health-check.sh
# 2. Backup all state
./scripts/backup-all.sh v4.0.0-rollback-$(date +%Y%m%d-%H%M%S)
# 3. Verify backups
./scripts/verify-backups.sh v4.0.0-rollback-$(date +%Y%m%d-%H%M%S)
# 4. Check current version functionality
./scripts/verify-version.sh v4.0.0# 1. Verify version
./platform-factory version
# 2. Run health checks
./scripts/health-check.sh
# 3. Run smoke tests
./scripts/smoke-test.sh
# 4. Verify data consistency
./scripts/verify-data.sh
# 5. Check specific functionality
./platform-factory build --test examples/hello-world
./platform-factory push --test ghcr.io/your-repo/test:rollback# Test rollback procedure in staging
export ENVIRONMENT=staging
./scripts/test-rollback.sh v3.2.1
# Verify rollback in staging
./scripts/verify-rollback.sh v3.2.1# Run comprehensive verification
./scripts/post-rollback-verification.sh
# Check metrics
# Open Grafana: https://grafana.platform-factory.dev
# Check logs
journalctl -u platform-factory-* --since "rollback time" -f# Notify stakeholders (template)
cat <<EOF | mail -s "Rollback Completed: platform-factory v4.0.0 -> v3.2.1" stakeholders@your-org.com
Rollback Summary:
- Previous Version: v4.0.0
- Rollback Version: v3.2.1
- Rollback Time: $(date -u)
- Components Rolled Back:
* platform-factory CLI
* platform-factory-control-plane
* platform-factory-worker
- Status: Success
- Impact: Minimal
- Next Steps: Investigate v4.0.0 issues
Detailed Report: https://wiki.platform-factory.dev/rollbacks/2026-08-02
EOF# Analyze what went wrong
./scripts/analyze-failure.sh v4.0.0
# Collect logs from failed version
./scripts/collect-failure-logs.sh v4.0.0
# Identify root cause
./scripts/identify-root-cause.sh v4.0.0# Create rollback report
./scripts/create-rollback-report.sh v4.0.0 v3.2.1
# Update runbook with lessons learned
vim docs/runbooks/rollback.md
# Commit changes
git add .
git commit -m "docs: update rollback procedures based on v4.0.0 rollback"
git push origin main# Options after rollback:
# 1. Fix issues in v4.0.0 and re-release as v4.0.1
# 2. Create hot patch for v3.2.1
# 3. Investigate and fix in development branch
# Decision factors:
# - Severity of the issue
# - Time required to fix
# - Impact on users
# - Availability of workarounds
# Create action plan
echo "Action Plan for v4.0.0 Issues" > action-plan.md
echo "=============================" >> action-plan.md
echo "" >> action-plan.md
echo "1. Issue Analysis:" >> action-plan.md
echo " - [ ] Identify root cause" >> action-plan.md
echo " - [ ] Reproduce in test environment" >> action-plan.md
echo "" >> action-plan.md
echo "2. Fix Options:" >> action-plan.md
echo " - [ ] Hot patch for v3.2.1" >> action-plan.md
echo " - [ ] Fix in v4.0.0 and re-release as v4.0.1" >> action-plan.md
echo " - [ ] Fix in development for next major version" >> action-plan.md
echo "" >> action-plan.md
echo "3. Testing:" >> action-plan.md
echo " - [ ] Test fix in isolation" >> action-plan.md
echo " - [ ] Test rollback and roll-forward" >> action-plan.md
echo " - [ ] Test in staging environment" >> action-plan.md
echo "" >> action-plan.md
echo "4. Timeline:" >> action-plan.md
echo " - Target fix date: " >> action-plan.md
echo " - Target release date: " >> action-plan.mdSymptoms: All builds failing, build queue stuck
Rollback Procedure:
- Rollback control plane to previous version
- Rollback all workers to previous version
- Verify build functionality
- Re-trigger failed builds
Estimated Time: 15-30 minutes
Symptoms: Push/pull operations failing, authentication errors
Rollback Procedure:
- Rollback control plane
- Rollback registry client components
- Clear local cache
- Retry operations
Estimated Time: 10-20 minutes
Symptoms: MicroVMs failing to start, execution errors
Rollback Procedure:
- Rollback MicroVM manager
- Rollback hypervisor components
- Restart affected MicroVMs
- Verify MicroVM functionality
Estimated Time: 20-40 minutes
Symptoms: Signing operations failing, verification errors
Rollback Procedure:
- Rollback signing service
- Rollback all components that use signing
- Verify signature validation
- Re-sign affected artifacts
Estimated Time: 15-30 minutes
Symptoms: Complete system outage, multiple components failing
Rollback Procedure:
- Rollback all components to previous version
- Restore state from backup
- Verify all services
- Monitor for stability
Estimated Time: 45-90 minutes
# Full system backup
sudo ./scripts/backup-all.sh v4.0.0-$(date +%Y%m%d-%H%M%S)
# Incremental backup
sudo ./scripts/backup-incremental.sh
# Verify backups
sudo ./scripts/verify-backups.sh
# List backups
ls -la /var/backups/platform-factory/# Full system restore
sudo ./scripts/restore-all.sh v3.2.1
# Partial restore (selective components)
sudo ./scripts/restore-select.sh v3.2.1 control-plane worker-1 worker-2
# Verify restore
sudo ./scripts/verify-restore.sh v3.2.1| Backup Type | Retention | Location |
|---|---|---|
| Full Backups | 30 days | /var/backups/platform-factory/ |
| Incremental Backups | 7 days | /var/backups/platform-factory/incremental/ |
| State Backups | 365 days | /var/backups/platform-factory/state/ |
| Offsite Backups | Indefinite | S3/Cloud Storage |
# Test backup integrity
./scripts/test-backup.sh /var/backups/platform-factory/v4.0.0-20260802.tar.gz
# Test restore procedure
./scripts/test-restore.sh v3.2.1| Metric | Expected Behavior | Alert Threshold |
|---|---|---|
| Build Success Rate | Should return to 99.9% | < 99% |
| Push/Pull Success Rate | Should return to 99.99% | < 99.9% |
| MicroVM Startup Time | Should be < 2s | > 5s |
| Signing Success Rate | Should be 100% | < 100% |
| Error Rate | Should decrease to 0 | > 0 |
| Latency | Should return to baseline | 2x baseline |
# Watch build success rate
watch -n 5 "./platform-factory metrics get build_success_rate"
# Watch error rate
watch -n 5 "./platform-factory metrics get error_rate"
# Check system health
watch -n 10 "./scripts/health-check.sh"## 🔙 Rollback in Progress: platform-factory v4.0.0 → v3.2.1
**Status**: In Progress
**Start Time**: 2026-08-02T14:30:00Z
**Expected Completion**: 2026-08-02T15:00:00Z
### Impact
- All platform-factory services will be unavailable during rollback
- Expected downtime: 30 minutes
- Data loss: None expected
### Components Affected
- [x] platform-factory CLI
- [x] platform-factory-control-plane
- [x] platform-factory-worker (all instances)
- [ ] Other components
### Rollback Steps
- [x] Backup current state
- [x] Notify stakeholders
- [ ] Stop services
- [ ] Deploy previous version
- [ ] Start services
- [ ] Verify functionality
- [ ] Announce completion
### Contact
For questions or issues, contact @CYPT71 or security@platform-factory.dev## ✅ Rollback Complete: platform-factory v4.0.0 → v3.2.1
**Status**: Completed
**Start Time**: 2026-08-02T14:30:00Z
**Completion Time**: 2026-08-02T14:45:00Z
**Duration**: 15 minutes
### Results
- ✅ All services restored to v3.2.1
- ✅ Health checks passing
- ✅ Smoke tests passing
- ✅ No data loss
- ✅ No downtime beyond expected window
### Next Steps
1. Investigate v4.0.0 issues
2. Fix identified problems
3. Plan re-deployment
4. Update documentation
### Lessons Learned
- Issue was caused by [root cause]
- Rollback procedure worked as expected
- [Any improvements needed]
### Contact
For questions, contact @CYPT71 or maintainers@platform-factory.dev# Service Update: platform-factory v4.0.0 Rollback
We have rolled back platform-factory from version 4.0.0 to version 3.2.1 due to [brief
description of issue].
## Impact
- **Services Affected**: Build, Registry, Execution
- **Duration**: 15 minutes
- **Current Status**: All services operational
- **Data Loss**: None
## What Happened
[Brief description of what went wrong]
## What We're Doing
[Description of investigation and fix in progress]
## Timeline
| Time (UTC) | Event |
|------------|-------|
| 14:30 | Issue detected |
| 14:35 | Investigation started |
| 14:40 | Rollback decision made |
| 14:45 | Rollback completed |
| 14:50 | Services verified |
## Next Steps
We are investigating the issue and will provide an update within 24 hours.
For questions or concerns, please contact support@platform-factory.dev.| Script | Description |
|---|---|
scripts/rollback-cli.sh |
Rollback CLI to previous version |
scripts/rollback-all.sh |
Rollback all components |
scripts/rollback-control-plane.sh |
Rollback control plane |
scripts/rollback-workers.sh |
Rollback all workers |
scripts/test-rollback.sh |
Test rollback procedure |
scripts/verify-rollback.sh |
Verify rollback success |
| Script | Description |
|---|---|
scripts/backup-all.sh |
Backup all components and state |
scripts/backup-incremental.sh |
Incremental backup |
scripts/backup-state.sh |
Backup state only |
scripts/restore-all.sh |
Restore all from backup |
scripts/restore-select.sh |
Restore selective components |
scripts/verify-backups.sh |
Verify backup integrity |
| Script | Description |
|---|---|
scripts/health-check.sh |
Check system health |
scripts/smoke-test.sh |
Run smoke tests |
scripts/verify-data.sh |
Verify data consistency |
scripts/verify-installation.sh |
Verify installation |
scripts/verify-version.sh |
Verify version |
This document should be reviewed and updated:
- After each rollback (add lessons learned)
- When new components are added
- When rollback procedures change
- Quarterly (comprehensive review)
Last Updated: 2026-08-02 Owner: @CYPT71 Review Date: 2026-11-02
© 2026 CYPT71
platform-factory
Core
- Architecture and OCI Layout
- Next-generation Architecture
- Architecture Decision Records
- Security Model
- Threat Model and Residual Risks
- Independent Security Review Process
- CLI Reference
- Project Configuration and Dependency Freezing
- mTLS Configuration
- Meine Graal
CI/CD
Running an image
- Production Adoption Guide
- Dockerfile Consumer
- Local Dev (Podman/macOS)
- MicroVM Support
- MicroVM Administration
- Large-image streaming
Operating