diff --git a/docs/guides/hosting-guardrails/disaster-recovery/database-upgrade/index.md b/docs/guides/hosting-guardrails/disaster-recovery/database-upgrade/index.md index 5b0b09fa..bf3772fe 100644 --- a/docs/guides/hosting-guardrails/disaster-recovery/database-upgrade/index.md +++ b/docs/guides/hosting-guardrails/disaster-recovery/database-upgrade/index.md @@ -260,7 +260,7 @@ In a new session, initiate the `pg_dump` process using the `snapshot ID` obtaine nohup time pg_dump -h $SOURCE -U master --snapshot="00000062-000182C4-1" -F c -b -v -f data.dump turbot > dump.log 2>&1 & ``` - > [!CAUTION] + > [!CAUTION] > Enabling logical replication before a long pg_dump/restore can lead to WAL buildup and storage exhaustion. It is recommended to enable RDS storage autoscaling to prevent out-of-space errors during migration. ### Monitor @@ -523,7 +523,7 @@ Run smoke tests to Test both the restored and new database instances to confirm -Delete the new TED stack i.e. `turbot-einstein-green` along with its associated resources, including the S3 bucket, log groups, and AWS Backup. Clean up replication slots and subscriptions. +After verifying a successful upgrade and switchover, delete the new TED stack i.e `turbot-einstein-green` and remove its associated resources such as the S3 bucket, CloudWatch log groups, and AWS Backup configurations. Clean up replication slots and subscriptions. +### In Case of Upgrade Failure or Aborted Migration + +If the database upgrade fails or is manually aborted before completion, it’s important to clean up replication artifacts from the **source database** to avoid lingering replication objects and ensure a clean state for future upgrade attempts. + +Run the following SQL commands on the source (blue) database to drop replication artifacts: + +```sql +-- Check for existing publications +SELECT * FROM pg_publication; + +-- Drop the publication used for replication +DROP PUBLICATION pub_blue; + +-- Check for replication slots +SELECT * FROM pg_replication_slots; + +-- Drop the replication slot used for streaming +SELECT * FROM pg_drop_replication_slot('rs_blue'); + +-- Check for existing subscriptions +SELECT * FROM pg_subscription; + +-- Drop the subscription created during upgrade +DROP SUBSCRIPTION sub_blue; +``` + +**Example:** + +Here's an example showing how to check for and clean up a leftover publication: + +`select * from pg_publication` indicates the presence of left over `PUBLICATION` `pub_blue`, this needs to be dropped. Recheck after dropping the `PUBLICATION` + +``` +turbot=> select * from pg_publication +turbot-> ; + oid | pubname | pubowner | puballtables | pubinsert | pubupdate | pubdelete | pubtruncate | pubviaroot +------------+----------+----------+--------------+-----------+-----------+-----------+-------------+------------ + 2142597123 | pub_blue | 16397 | t | t | t | t | t | f +(1 row) + +turbot=> DROP PUBLICATION pub_blue; +DROP PUBLICATION + +turbot=> select * from pg_publication +; + oid | pubname | pubowner | puballtables | pubinsert | pubupdate | pubdelete | pubtruncate | pubviaroot +-----+---------+----------+--------------+-----------+-----------+-----------+-------------+------------ +(0 rows) +``` + +Also, make sure to delete associated resources created as part of the upgrade attempt: +- S3 bucket +- CloudWatch Log Groups +- AWS Backup plans and vaults +- Any temporary RDS instances or TED-related infrastructure + ## Step 18: Disable and Delete Subscriptions Disable and delete subscription and replication slots. - ```sql select * from pg_subscription; alter subscription sub_blue disable;