MB-2799 Import fake pricing data to staging/experimental

[noelledusahel]

Description

Follows steps outlined in https://dp3.atlassian.net/browse/MB-2262 to import and upload fake pricing data via secure migration

• WIll not be deployed to production

• Deployed to Staging & Experimental only

Steps for deploying to experimental:

• https://github.com/transcom/mymove/wiki/deploy-to-experimental

• https://github.com/transcom/mymove/wiki/migrate-the-database#secure-migrations-for-one-environment

Reviewer Notes

• follow instructions in MB-2262 create new migration for pricing imports #4114
• aws-vault login $AWS_PROFILE
• open S3

Look for 20200615192021_import_pricing_data_june_2020.up.sql in the following folders

• transcom-ppp-app-experimental-us-west-2/secure-migrations
• transcom-ppp-app-staging-us-west-2/secure-migrations
• transcom-ppp-app-prod-us-west-2/secure-migrations (this file should only contain a comment, no sql)

Code Review Verification Steps

• Code follows the guidelines for Logging
• The requirements listed in
  Querying the Database Safely
  have been satisfied.
• Any new migrations/schema changes:
  • Follow our guidelines for zero-downtime deploys (see Zero-Downtime Deploys)
  • Have been communicated to #g-database
  • Secure migrations have been tested following the instructions in our docs
• There are no aXe warnings for UI.
• This works in Supported Browsers and their phone views (Chrome, Firefox, IE, Edge).
• Tested in the Experimental environment (for changes to containers, app startup, or connection to data stores)
• User facing changes have been reviewed by design.
• Request review from a member of a different team.
• Have the Jira acceptance criteria been met for this change?

References

• Jira story
• this article explains more about the approach used.

Screenshots

If this PR makes visible UI changes, an image of the finished UI can help reviewers and casual
observers understand the context of the changes. A before image is optional and
can be included at the submitter's discretion.

Consider using an animated image to show an entire workflow instead of using multiple images. You may want to use GIPHY CAPTURE for this! 📸

Please frame screenshots to show enough useful context but also highlight the affected regions.

[robot-mymove]

	Messages
📖	🔗 MB-2799

Generated by 🚫 dangerJS against 9cb6b11

[blvrd]

😮

Looks good to me!

[reggieriser]

It looks like the contract code is set to Pricing. Is that what we want for the test data? I'm thinking we may need to make it more obvious this is test data rather than real pricing. Also, my PR #4250 assumed it was named TRUSS (which I just made up), so I'll need to change that accordingly to whatever the name is here.

Note that if we do decide to change the code, we'll need to restore the experimental DB to make the migration run again there (infra can help us with that -- we've done it several times before).

[noelledusahel]

It looks like the contract code is set to Pricing. Is that what we want for the test data? I'm thinking we may need to make it more obvious this is test data rather than real pricing. Also, my PR #4250 assumed it was named TRUSS (which I just made up), so I'll need to change that accordingly to whatever the name is here.

Note that if we do decide to change the code, we'll need to restore the experimental DB to make the migration run again there (infra can help us with that -- we've done it several times before).

OK i definitely missed that the code 'Pricing' stuck, i meant for it to be 'june_2020' @jacquelineIO do you have any guidance on what Contract Code we want attached to this fake data?

[reggieriser]

I went ahead and had infra roll back the experimental database so we can run this migration against experimental again when we're ready (let's make it the last thing we do after reviewer approval but before merging).

See #4279 for a fix that will need to be merged into this PR after #4279 is approved.

Also, I think we should consider changing the name of the migration since import_pricing_data_june_2020 could imply this is real data. I'm thinking more like import_fake_pricing_data. We still need to get confirmation of the contract code we want to use.

With the fix from #4279, I checked all the re_* tables imported from this PR against a psql-loaded migration and they seemed to match. 🎉

[noelledusahel]

I went ahead and had infra roll back the experimental database so we can run this migration against experimental again when we're ready (let's make it the last thing we do after reviewer approval but before merging).

See #4279 for a fix that will need to be merged into this PR after #4279 is approved.

Also, I think we should consider changing the name of the migration since import_pricing_data_june_2020 could imply this is real data. I'm thinking more like import_fake_pricing_data. We still need to get confirmation of the contract code we want to use.

With the fix from #4279, I checked all the re_* tables imported from this PR against a psql-loaded migration and they seemed to match. 🎉

to change the name of the migraiton to import_fake_pricing_data_ i would need to write code in the script that differentiates between an upload of fake data vs real data. i guess it would look like an extra step in the script that asks the user whether this is real or fake data. would it be better to just give the fake data a contract code that indicates that its fake? like (Fake Truss Data)

[reggieriser]

to change the name of the migraiton to import_fake_pricing_data_ i would need to write code in the script that differentiates between an upload of fake data vs real data. i guess it would look like an extra step in the script that asks the user whether this is real or fake data. would it be better to just give the fake data a contract code that indicates that its fake? like (Fake Truss Data)

I don't think the script has to change. You should be able to change the migration filename after the pricing-import script creates it -- just remember to also change the migrations_manifest.txt file accordingly. I'm also fine with import_pricing_data_fake (if you want to keep the same prefix) or whatever you'd like that makes it clear that it's not real data. That reminds me -- we will also need to clean up S3 to remove the old migration in each environment and add the new one when it's ready.

[noelledusahel]

to change the name of the migraiton to import_fake_pricing_data_ i would need to write code in the script that differentiates between an upload of fake data vs real data. i guess it would look like an extra step in the script that asks the user whether this is real or fake data. would it be better to just give the fake data a contract code that indicates that its fake? like (Fake Truss Data)

I don't think the script has to change. You should be able to change the migration filename after the pricing-import script creates it -- just remember to also change the migrations_manifest.txt file accordingly. I'm also fine with import_pricing_data_fake (if you want to keep the same prefix) or whatever you'd like that makes it clear that it's not real data. That reminds me -- we will also need to clean up S3 to remove the old migration in each environment and add the new one when it's ready.

ok so you're talking about manually changing it right?

[reggieriser]

ok so you're talking about manually changing it right?

Right. But if you wanted to change the pricing-import script, I'd probably just give the user the option/flexibility to enter a migration name rather than asking them if it's fake data or not. But I think a manual change is fine for this story.

[reggieriser]

LGTM! I tried a make run_staging_migrations and compared the re_* tables in my deployed_migrations database against my local DB that had an imported version of the fake data via the pricing parser. Everything aligned (and the problem with the lack of leading zeros was gone). I looked at S3 for each environment and saw the same migration in experimental/staging and a no-op in prod (as expected).

In anticipation of this getting merged, I'll go ahead and update my PR #4250 to default to using the TRUSS_TEST contract code.