Replicate foorm tables to Redshift

[bencodeorg]

Replicates foorm tables to Redshift. There might be foorm_ tables that don't contain PII, but following precedent with pd_ tables where we put everything in dashboard_production_pii schema. I think this is a good pattern so that everything in a given namespace appears in the same schema, instead of being split across the two (dashboard_production and dashboard_production_pii).

Testing story

So far, just checked whether this looks like it would appropriately update the cron-pii DMS task:

Benjamins-MacBook-Pro:code-dot-org benjaminbrooks$ RAILS_ENV=production bundle exec rake stack:data:validate
GetSecretValue: production/cdo/db_writer
Data layer including RedShift cluster configuration and synchronization with RDS instance.
Listing changes to existing stack `DATA-production`:
Modify DMSCronPii [AWS::DMS::ReplicationTask] Properties Replacement: Conditional (TableMappings)

Once approved, can run RAILS_ENV=production bundle exec rake stack:data:start to actually modify the CloudFormation template.

[sureshc]

I’m still investigating the failures exporting to Redshift that occurred the last 2 days, so check with my before applying this stack update.

[bencodeorg]

I'm ready to make it @sureshc, can you just comment here when it's ok to do so?

[sureshc]

Will do

[sureshc]

@bencodeorg Looks like this recent change to the size of the levels.properties column is causing the export to Resdhift to fail. Unlikely we’ll have a resolution today.

https://github.com/code-dot-org/code-dot-org/blob/staging/dashboard/db/migrate/20200609183124_change_level_properties_size.rb

[sureshc]

@bencodeorg I’m still investigating the export error that occurred Mon & Tue. The Wednesday export mysteriously succeeded, so it’s fine for you to apply this change while I continue to investigate the issue with the levels table.

[bencodeorg]

Success -- checked in AWS web console that cron-pii task now includes foorm_ tables.

Benjamins-MacBook-Pro:code-dot-org benjaminbrooks$ RAILS_ENV=production bundle exec rake stack:data:start
GetSecretValue: production/cdo/db_writer
update stack: DATA-production...
Stack update requested, waiting for provisioning to complete...
2020-06-18 17:37:09 UTC- DATA-production [UPDATE_IN_PROGRESS]: User Initiated
..2020-06-18 17:37:23 UTC- DMSCronPii [UPDATE_IN_PROGRESS]
.......2020-06-18 17:38:12 UTC- DMSCronPii [UPDATE_COMPLETE]
2020-06-18 17:38:15 UTC- DATA-production [UPDATE_COMPLETE_CLEANUP_IN_PROGRESS]

Stack update complete.
Outputs: