Skip to content
This repository has been archived by the owner on Feb 1, 2024. It is now read-only.

Add script to copy/anonymize the production database onto staging #1948

Merged
merged 2 commits into from Jul 6, 2022

Conversation

maurizi
Copy link
Contributor

@maurizi maurizi commented Jul 1, 2022

Overview

This adds an interactive script meant to be run on the staging bastion, which automates the process of restoring a production database snapshot, anonymizing it, creating a dump, and restoring that dump on top of staging.

Connects #1813

Demo

Screenshot 2022-07-01 111901

Screenshot 2022-07-01 105902

Notes

I ran this from the bastion.

To run it, I first had to manually install git and python3: sudo yum install git python37. Then I clone open-apparel-registry and checked out this branch and ran ./tools/copy_production_db_to_staging.

For testing purposes, I used a copy of the staging database I spun up from a recent snapshot.

Testing Instructions

  • SSH to the staging bastion
  • cd into open-apparel-registry/ and ensure the latest version of this branch is checked out.
  • Run ./tools/copy_production_db_to_staging. You will be asked to supply staging/production database passwords, as well as confirm data determined using the AWS CLI (I substituted the staging database copy for the staging database endpoint determined by the script).
  • At each step, confirm results expected for that step of the script in the AWS console or using psql on either the temporary RDS database or the database you will be copying onto (staging by default)

Checklist

  • fixup! commits have been squashed
  • CI passes after rebase
  • CHANGELOG.md updated with summary of features or fixes, following Keep a Changelog guidelines
  • This PR is targeted at the correct branch (develop vs. ogr/develop)
  • If this PR applies to both OAR and OGR a companion PR has been created

# Constants
STAGING_ID="openapparelregistry-enc-stg"
PRODUCTION_ID="openapparelregistry-enc-prd"
ALLOWED_EMAILS="person@example.com other@example.com"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jwalgran do we have the actual list we want here? Or do we not want to list the default like with the password fields?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are 3 API partners who's accounts we want to preserve but we do not want to name them here in the open source repo.

@jwalgran jwalgran self-requested a review July 5, 2022 20:23
@jwalgran jwalgran self-assigned this Jul 5, 2022
Copy link
Contributor

@jwalgran jwalgran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This worked delightfully well. Nice job.

I ran a successful test of this after one typo fixup. I made some other small recommendations that don't need another full round of review.

# Constants
STAGING_ID="openapparelregistry-enc-stg"
PRODUCTION_ID="openapparelregistry-enc-prd"
ALLOWED_EMAILS="person@example.com other@example.com"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are 3 API partners who's accounts we want to preserve but we do not want to name them here in the open source repo.

tools/copy_production_db_to_staging Outdated Show resolved Hide resolved
tools/copy_production_db_to_staging Outdated Show resolved Hide resolved
tools/copy_production_db_to_staging Outdated Show resolved Hide resolved
@jwalgran jwalgran assigned maurizi and unassigned jwalgran Jul 6, 2022
@maurizi maurizi force-pushed the feature/mvm/automate-db-restore branch from 72a4511 to b7fecdc Compare July 6, 2022 12:51
@maurizi maurizi merged commit 7833460 into develop Jul 6, 2022
@maurizi maurizi deleted the feature/mvm/automate-db-restore branch July 6, 2022 13:16
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants