Script to update USC/LAUSD partner ZIP mappings

[molly-moen]

We have new zip mappings for LAUSD and need to remove some duplicate zips between USC and LAUSD. This script:

• replaces all current LAUSD ZIPs with ZIPs specified in the CSV
• determines overlap between CSV-specified set of ZIPs for LAUSD and current USC ZIPs
• unassigns all ZIPs in that overlapping set from USC

Background

I followed patterns in previous zip mapping scripts here and here

Deployment strategy

Will run this script in prod once it's reviewed, checked in, and DTP happens.

Links

• Jira

Testing story

I made a local copy of the zip mappings for USC and LAUSD and validated the script correctly removed the old LAUSD mappings, added the new ones, and removed the duplicates from USC. Also validated:

• The script is idempotent (will give the same result if run multiple times in a row)
• Other zip mappings for different partner ids are unaffected
• Zip mappings show up correctly in UI

Reviewer Checklist:

• Tests provide adequate coverage
• Code is well-commented
• New features are translatable or updates will not break translations
• Relevant documentation has been added or updated
• User impact is well-understood and desirable
• Pull Request is labeled appropriately
• Follow-up work items (including potential tech debt) are tracked and linked

[breville]

In general terms this is another place I wish we had an immutable mindset, where we never had to delete any data from production, but instead could add additional data that would override prior data. It would be a lot less risky and nerve-wracking to make these changes.

That aside, are you able to run this on localhost as well as on staging and test environments to rehearse before we actually run it on production.

And if something goes wrong on production, do you have a sense of how easy it is to fix it up, short of restoring from a database backup? (And if nothing else, is there a way you can snapshot the current data on production to make a restoration easier if it's necessary?)

[molly-moen]

@breville I have run on localhost multiple times, and can run on staging and test before it goes to production. For the revert, I have a local copy of the current zip mappings for the two partners and could fairly easily re-upload those to prod if there were issues. Do we have best practices for creating snapshots/backups before doing something like this?

In terms of the immutable mindset, I agree. It seems that isn't possible with the way we currently have things set up, but would be interested in finding ways to make things more immutable in the future.

In general terms this is another place I wish we had an immutable mindset, where we never had to delete any data from production, but instead could add additional data that would override prior data. It would be a lot less risky and nerve-wracking to make these changes.

That aside, are you able to run this on localhost as well as on staging and test environments to rehearse before we actually run it on production.

And if something goes wrong on production, do you have a sense of how easy it is to fix it up, short of restoring from a database backup? (And if nothing else, is there a way you can snapshot the current data on production to make a restoration easier if it's necessary?)

[breville]

@molly-moen It sounds like you've made a good set of precautions. One final thing we've often done is run the operation inside a transaction and rolled it back (e.g. https://github.com/code-dot-org/code-dot-org/blob/staging/bin/oneoff/unsent_pl_emails_send.rb#L61) as a dry-run on production, before manually removing that rollback temporarily when running it for real. This gives a final way to see what will happen on production before actually doing it.

[molly-moen]

@breville makes sense, added the rollback line

[bencodeorg]

You can't really test this on staging/test since the mappings don't exist there, right?

As long as you have a backup of the mappings, I feel like this is pretty low risk. Might be worth just dumping the entire mappings table before you run it, something like (from production directory, this is what @islemaster and I did when deleting some mappings a couple weeks ago):

bin/dashboard-sql "select * from pd_regional_partner_mappings;" > ../pd_regional_partner_mappings.txt

[bencodeorg]

LGTM! Cool to learn about ActiveRecord import.