Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export census_summaries to Redshift #54524

Merged
merged 1 commit into from
Nov 15, 2023
Merged

Conversation

bakerfranke
Copy link
Contributor

@bakerfranke bakerfranke commented Oct 26, 2023

Added dashboard.census_summaries to redshift export cron.

Testing story

code-dot-org % export AWS_PROFILE=codeorg-admin
code-dot-org % bundle exec rake stack:data:validate RAILS_ENV=production
Finished stack:data:environment (less than a minute)
Pending update for stack `DATA-production`:
Modify DMSCron [AWS::DMS::ReplicationTask] Properties Replacement: Conditional (TableMappings)
Finished stack:data:validate (1 minute)

Follow-up work

This table gets updated once a year, and needs to be pulled into redshift at that time. I'm not sure that adding it to the daily replication process is the most efficient (unless there is some mechanism for DMS to know that nothing has changed and it ignores it. In which case, fine.

Privacy

This table has no PII.

PR Checklist:

  • Tests provide adequate coverage
  • [X ] Privacy and Security impacts have been assessed
  • Code is well-commented
  • New features are translatable or updates will not break translations
  • Relevant documentation has been added or updated
  • [X ] User impact is well-understood and desirable
  • Pull Request is labeled appropriately
  • Follow-up work items (including potential tech debt) are tracked and linked

added dashboard.censuse_summaries to redshift export cron.
@bakerfranke bakerfranke requested a review from a team as a code owner October 26, 2023 20:22
@sureshc sureshc changed the title Update tasks.yml Export census_summaries to Redshift Oct 26, 2023
@sureshc
Copy link
Contributor

sureshc commented Oct 26, 2023

Is there any sensitive data in the audit_data column? Do we need to export that column to Redshift?

@bakerfranke
Copy link
Contributor Author

@sureshc there is no sensitive data in the audit_data column. That column exists in redshift now. We also plan to use it for analysis.

@sureshc
Copy link
Contributor

sureshc commented Oct 26, 2023

Deployed successfully:

% bundle exec rake stack:data:start RAILS_ENV=production
Finished stack:data:environment (less than a minute)
Pending update for stack `DATA-production`:
Modify DMSCron [AWS::DMS::ReplicationTask] Properties Replacement: Conditional (TableMappings)
Proceed? [y/n]
y
 Stack update requested, waiting for provisioning to complete...
..2023-10-26 20:50:06 UTC- DMSCron [UPDATE_IN_PROGRESS]
........2023-10-26 20:50:55 UTC- DMSCron [UPDATE_COMPLETE]
.2023-10-26 20:51:00 UTC- DATA-production [UPDATE_COMPLETE_CLEANUP_IN_PROGRESS]

Stack update complete.
Outputs:
Finished stack:data:start (3 minutes)

@sureshc
Copy link
Contributor

sureshc commented Oct 31, 2023

@bakerfranke You can merge this!

@sureshc sureshc merged commit d385c4e into staging Nov 15, 2023
2 checks passed
@sureshc sureshc deleted the add-census-summaries-to-export branch November 15, 2023 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants