Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration of fac-backup-utility #3916

Merged
merged 56 commits into from
Jun 24, 2024
Merged

Integration of fac-backup-utility #3916

merged 56 commits into from
Jun 24, 2024

Conversation

asteel-gsa
Copy link
Contributor

@asteel-gsa asteel-gsa commented May 30, 2024 •
edited
Loading

Description:

Integration of the fac-backup-utility within the application. This, at a high level, removes the previously used django-dbbackup and replaces it with a table by table dump of the rds to s3 and secondary rds.

Note:

It appears my IDE stripped some line endings, hence why some of the files look like they have been substantially changed, but I will list everything below.

Overview:

  • dev, staging, and preview have been given their own, dedicated s3 bucket backups, managed by their respective module.
  • prod still has its backups bucket managed by the meta module.
  • all environments receive a new rds fac-snapshot-db.
  • meta managed production backups get tagged with s3 and are now up to date on version.
  • manifest.yml has been modified to ensure that backups and fac-snapshot-db are now bound to the running app, and will not collide due to sharing across spaces.
  • terraform-apply-env.yml no longer shares the production backups to staging.
  • docker-compose.yml and docker-compose-web.yml have been updated to include a new database so this can be run locally.
  • two bash scripts, fac-backup-util.sh and fac-restore-util.sh have been added and will replace the existing bash scripts, i.e backup-database.yml.
  • workflows have been modified to use the new backup/restore system.
  • only fac-admins can run the restore process at this time via workflow.
  • a new, scheduled job, which uses a matrix, has been added, which will do a scheduled_backup operation on all 3 environments via matrix.

Common Commands:

### Common Commands:
./fac-backup-util.sh v0.1.2 initial_backup
# Curl the utility
# Install AWS
# DB to S3 table dump (backups)
# DB to DB table dump (fac-db -> fac-snapshot-db) [No Truncate, as tables dont exist]
# AWS S3 sync (fac-private-s3 -> backups)

./fac-backup-util.sh v0.1.2 deploy_backup
# Curl the utility
# Install AWS
# DB to S3 table dump (backups)
# AWS S3 sync (fac-private-s3 -> backups)

./fac-backup-util.sh v0.1.2 scheduled_backup
# Curl the utility
# Install AWS
# DB to S3 table dump (fac-db -> backups)
# DB to DB table dump (fac-db -> fac-snapshot-db) [Truncate target table before dump]
# AWS S3 sync (fac-private-s3 -> backups)

./fac-restore-util.sh v0.1.2 s3_restore 202405301232
# Curl the utility
# Install AWS
# DB to S3 table dump (backups -> fac-db) [Truncate target table before --data-only pg_restore]
# AWS S3 sync (backups -> fac-private-s3)

./fac-restore-util.sh v0.1.2 db_restore
# Curl the utility
# Install AWS
# DB to DB table dump (fac-snapshot-db -> fac-db) [Truncate target table before dump]
# AWS S3 sync (fac-private-s3 -> backups)

Matrix Test/Validation:
Run: here

Todo:

  • review fac-backup-util main repo

Prep work before merge:

  • Unshare backups from prod manually: cf unshare-service backups -s staging -f
  • clean and delete backups in dev so terraform can rebuild it

@asteel-gsa asteel-gsa linked an issue May 30, 2024 that may be closed by this pull request
[MVP] FAC Backup Utility #3816
Closed
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 30, 2024 •
edited
Loading

File Coverage Missing
All files 87% ✅
api/serializers.py 88% ✅ 177-178 183 188
api/test_views.py 95% ✅ 103
api/uei.py 88% ✅ 87 118-119 163 167-168
api/views.py 98% ✅ 198-199 337-338
audit/forms.py 60% ❌ 31-38 109-116
audit/intake_to_dissemination.py 88% ✅ 57-62 264 308-316
audit/mixins.py 96% ✅ 28
audit/test_commands.py 91% ✅
audit/test_intakelib.py 88% ✅ 154-158
audit/test_manage_submission_access_view.py 98% ✅ 15 19
audit/test_mixins.py 90% ✅ 159-160 164-166 254-255 259-261
audit/test_validators.py 95% ✅ 439 443 611-612 851 858 865 872 1117-1118 1149-1150 1175-1180
audit/test_views.py 98% ✅ 132
audit/test_workbooks_should_fail.py 88% ✅ 58 87-88 92
audit/test_workbooks_should_pass.py 87% ✅ 59 74-76
audit/utils.py 86% ✅ 9 19 60-62 65
audit/validators.py 93% ✅ 138 190 279 419-420 435-436 519-520 622-626 631-635 651-660
audit/cross_validation/additional_ueis.py 93% ✅ 33
audit/cross_validation/check_award_ref_declaration.py 90% ✅
audit/cross_validation/check_award_reference_uniqueness.py 93% ✅
audit/cross_validation/check_certifying_contacts.py 87% ✅
audit/cross_validation/check_findings_count_consistency.py 87% ✅ 35
audit/cross_validation/check_ref_number_in_cap.py 91% ✅
audit/cross_validation/check_ref_number_in_findings_text.py 91% ✅
audit/cross_validation/errors.py 78% ❌ 30 77
audit/cross_validation/naming.py 93% ✅ 229
audit/cross_validation/submission_progress_check.py 91% ✅ 83 126 174 182-183
audit/cross_validation/tribal_data_sharing_consent.py 81% ❌ 33 36 40
audit/cross_validation/validate_general_information.py 65% ❌ 77 81-84 96 99
audit/fixtures/dissemination.py 71% ❌ 38
audit/fixtures/single_audit_checklist.py 55% ❌ 160-197 245-254
audit/intakelib/exceptions.py 71% ❌ 7-9 12
audit/intakelib/intermediate_representation.py 91% ✅ 27-28 73 91 129 200-203 212-213 283-284
audit/intakelib/mapping_audit_findings.py 97% ✅ 55
audit/intakelib/mapping_audit_findings_text.py 97% ✅ 54
audit/intakelib/mapping_federal_awards.py 93% ✅ 92
audit/intakelib/mapping_util.py 79% ❌ 21 25 29 63 99 104-105 114-120 130 145 150
audit/intakelib/checks/check_all_unique_award_numbers.py 79% ❌ 24
audit/intakelib/checks/check_cluster_names.py 75% ❌ 20-25
audit/intakelib/checks/check_cluster_total.py 95% ✅ 99
audit/intakelib/checks/check_finding_reference_pattern.py 74% ❌ 34 44-45
audit/intakelib/checks/check_findings_grid_validation.py 89% ✅ 59
audit/intakelib/checks/check_has_all_the_named_ranges.py 95% ✅ 66
audit/intakelib/checks/check_is_a_workbook.py 69% ❌ 20
audit/intakelib/checks/check_loan_balance_entries.py 83% ❌ 28
audit/intakelib/checks/check_look_for_empty_rows.py 91% ✅ 18
audit/intakelib/checks/check_no_major_program_no_type.py 76% ❌ 18 27
audit/intakelib/checks/check_no_repeat_findings.py 88% ✅ 21
audit/intakelib/checks/check_other_cluster_names.py 81% ❌ 23 33
audit/intakelib/checks/check_passthrough_name_when_no_direct.py 83% ❌ 11 49 58
audit/intakelib/checks/check_sequential_award_numbers.py 82% ❌ 25 35
audit/intakelib/checks/check_start_and_end_rows_of_all_columns_are_same.py 89% ✅ 14
audit/intakelib/checks/check_state_cluster_names.py 81% ❌ 23 33
audit/intakelib/checks/check_version_number.py 73% ❌ 30 40-41
audit/intakelib/checks/runners.py 95% ✅ 187 217
audit/intakelib/common/util.py 90% ✅ 22 39
audit/intakelib/transforms/xform_rename_additional_notes_sheet.py 81% ❌ 14
audit/management/commands/load_fixtures.py 47% ❌ 40-46
audit/models/models.py 85% ✅ 58 60 65 67 209 215 227 239-242 260 437 455-456 464 486 584-585 589 597 606 612
audit/views/audit_info_form_view.py 27% ❌ 25-74 77-117 120-137
audit/views/manage_submission.py 86% ✅ 73-80
audit/views/manage_submission_access.py 98% ✅ 113-114
audit/views/pre_dissemination_download_view.py 78% ❌ 15-16 21-22 29-39
audit/views/submission_progress_view.py 89% ✅ 117 182-183
audit/views/tribal_data_consent.py 34% ❌ 23-41 44-79
audit/views/unlock_after_certification.py 57% ❌ 28-51 73-87
audit/views/upload_report_view.py 26% ❌ 32-35 44 91-117 120-170 178-209
audit/views/views.py 53% ❌ 74 81-100 123-124 198-199 220-230 257 268-269 280-281 283-287 329-342 345-359 364-377 394-400 405-425 452-456 461-490 533-537 542-562 589-593 598-627 670-674 679-691 694-704 709-721 754-768
census_historical_migration/change_record.py 98% ✅ 30
census_historical_migration/end_to_end_core.py 26% ❌ 57-89 93-111 116-155 161-187 246-258 263 273-307
census_historical_migration/invalid_record.py 94% ✅ 50 54 58 62 66
census_historical_migration/migration_result.py 75% ❌ 17 21 25 29 33-42 46
census_historical_migration/report_type_flag.py 96% ✅ 19
census_historical_migration/test_federal_awards_xforms.py 99% ✅ 219-220
census_historical_migration/sac_general_lib/audit_information.py 91% ✅ 28 82-87 336
census_historical_migration/sac_general_lib/cognizant_oversight.py 68% ❌ 11
census_historical_migration/sac_general_lib/general_information.py 86% ✅ 166-167 177-178 186-187 195-200 233-255 354-355
census_historical_migration/sac_general_lib/sac_creator.py 90% ✅ 34
census_historical_migration/sac_general_lib/utils.py 84% ❌ 35 62-71
census_historical_migration/transforms/xform_remove_hyphen_and_pad_zip.py 92% ✅ 18
census_historical_migration/transforms/xform_retrieve_uei.py 67% ❌ 10
census_historical_migration/transforms/xform_string_to_bool.py 87% ✅ 17
census_historical_migration/workbooklib/additional_eins.py 84% ❌ 58-60 67-77
census_historical_migration/workbooklib/additional_ueis.py 77% ❌ 27-29 36-46
census_historical_migration/workbooklib/corrective_action_plan.py 46% ❌ 49-51 65 93-125 134-153
census_historical_migration/workbooklib/excel_creation_utils.py 69% ❌ 110 119-124 129-136 140-158 171-175 189-192
census_historical_migration/workbooklib/federal_awards.py 77% ❌ 181-184 262-301 487 554-562 572-597 621-622 918-1022
census_historical_migration/workbooklib/findings.py 69% ❌ 81-86 154-174 179-199 216-218 330-354
census_historical_migration/workbooklib/findings_text.py 46% ❌ 50-51 67 97-129 138-160
census_historical_migration/workbooklib/notes_to_sefa.py 66% ❌ 34-38 101-102 104-105 107-108 136-143 152-160 168-170 279-326
census_historical_migration/workbooklib/post_upload_utils.py 21% ❌ 22-35 66-83 89-111
census_historical_migration/workbooklib/secondary_auditors.py 88% ✅ 128-130 186-205
census_historical_migration/workbooklib/workbook_builder.py 38% ❌ 14-17 26-41
census_historical_migration/workbooklib/workbook_builder_loader.py 41% ❌ 18-30
config/error_handlers.py 94% ✅ 22
config/test_settings.py 92% ✅ 33-34 49-50
config/urls.py 72% ❌ 88
dissemination/file_downloads.py 81% ❌ 43-61 91-93
dissemination/forms.py 87% ✅ 135 144 255
dissemination/search.py 88% ✅ 113 115 119 127-128
dissemination/summary_reports.py 78% ❌ 274 300-302 306-310 421 438 459 511-575 603 638-640 664-672
dissemination/test_search.py 93% ✅ 51-66 473-474 579-596 608-632 644-669 677-693
dissemination/test_summary_reports.py 98% ✅
dissemination/views.py 75% ❌ 134 140-142 159-225 268 298 300 336 387 389 391 469-474
dissemination/migrations/0002_general_fac_accepted_date.py 47% ❌ 10-12
dissemination/searchlib/search_alns.py 37% ❌ 44-58 78-110 115-177 184-187
dissemination/searchlib/search_direct_funding.py 86% ✅ 21-22
dissemination/searchlib/search_findings.py 76% ❌ 18-24 34 36 38
dissemination/searchlib/search_general.py 96% ✅ 138
dissemination/searchlib/search_passthrough_name.py 35% ❌ 21-31
djangooidc/backends.py 78% ❌ 32 57-63
djangooidc/exceptions.py 66% ❌ 19 21 23 28
djangooidc/oidc.py 16% ❌ 32-35 45-51 64-70 92-149 153-199 203-226 230-275 280-281 286
djangooidc/views.py 81% ❌ 22 43 109-110 117
djangooidc/tests/common.py 97% ✅
report_submission/forms.py 92% ✅ 35
report_submission/test_views.py 98% ✅ 835
report_submission/views.py 79% ❌ 97 250 274-275 280-281 321-491 494-504 563 599-601 609-610 613-615
report_submission/templatetags/get_attr.py 76% ❌ 8 11-14 18
support/admin.py 88% ✅ 76 79 84 91-97 100-102
support/cog_over.py 91% ✅ 29-32 92 116-120 156
support/test_admin_api.py 81% ❌ 23 147-148 237-238 317-318
support/test_cog_over.py 98% ✅ 174-175 264
support/management/commands/seed_cog_baseline.py 98% ✅ 20-21
support/models/cog_over.py 89% ✅ 103-104
tools/update_program_data.py 89% ✅ 96
users/admin.py 99% ✅ 27
users/auth.py 96% ✅ 58-59
users/models.py 96% ✅ 18 74-75
users/fixtures/user_fixtures.py 91% ✅

Minimum allowed coverage is 85%

Generated by 🐒 cobertura-action against f69ad62

@asteel-gsa
Copy link
Contributor Author

asteel-gsa commented May 31, 2024 •
edited
Loading

Notes from review w/ Jadud:

  • change 18f -> gsa-tts in terraform
  • consider daily backups (just needs a new workflow)
  • file structure for s3 (hourly-00, hourly-02, etc etc and daily-0, daily-1)
  • deploy backup should go to rds instead of s3, and save the s3 for scheduled backups to maintain point in time recovery over more ranges, and utilize the rds backup methodology as a true cold store for deployment backups

Action items from data team discussion:

  • s3_to_db pipe rather than use disk. (This is probably a task @jadudm and I will have to take some time on and diagnose for the next iteration of development)
  • PG_DUMP_TABLE() format c -> plain for db_to_db dumps

@asteel-gsa asteel-gsa marked this pull request as ready for review June 4, 2024 14:00
@asteel-gsa asteel-gsa requested a review from a team as a code owner June 4, 2024 14:00
@asteel-gsa asteel-gsa changed the title [DRAFT] Integration of fac-backup-utility Integration of fac-backup-utility Jun 4, 2024
@asteel-gsa
Copy link
Contributor Author

Closing this temporarily, as I will be re-testing the workflows with push.

@asteel-gsa asteel-gsa closed this Jun 4, 2024
@asteel-gsa
Copy link
Contributor Author

asteel-gsa commented Jun 4, 2024 •
edited
Loading

deploy_backup: Workflow Run
Operations: db to db backup, aws s3 sync
image
Confirmation: Tunneled into fac-snapshot-db and confirmed tables existing. AWS S3 Sync passed as there has been no change of data in either bucket.

scheduled_backup: Workflow Run
Operations: db to s3 backup, aws s3 sync
s3 folder: scheduled/06-04-18/

# stored in scheduled/mm-dd-HH/ format
Alex Steel@DESKTOP-NL4DO24 MINGW64 ~/Code/FAC (as/db-backup-util)
$ aws s3 ls s3://$BUCKET/backups/scheduled/06-04-18/
2024-06-04 14:31:20       5037 public-audit_access.dump
2024-06-04 14:33:02       5595 public-audit_deletedaccess.dump
2024-06-04 14:31:20       4346 public-audit_excelfile.dump
2024-06-04 14:32:50       5649 public-audit_singleauditchecklist.dump
2024-06-04 14:31:38       4754 public-audit_singleauditreportfile.dump
.........

daily_backup: Workflow Run
Operations: db to s3 backup, aws s3 sync
s3 folder: daily/06-04/

# stored in daily/mm-dd/ format
Alex Steel@DESKTOP-NL4DO24 MINGW64 ~/Code/FAC (as/db-backup-util)
$ aws s3 ls s3://$BUCKET/backups/daily/06-04/
2024-06-04 14:41:32       5037 public-audit_access.dump
2024-06-04 14:42:55       5595 public-audit_deletedaccess.dump
2024-06-04 14:42:54       4346 public-audit_excelfile.dump
2024-06-04 14:39:41       5649 public-audit_singleauditchecklist.dump
2024-06-04 14:39:51       4754 public-audit_singleauditreportfile.dump
.........

s3_restore: Workflow Run
Operations: s3 to db restore, aws s3 sync
s3 folder: `daily/06-04/

# Example output of a specific table
2024-06-04T14:51:06.56-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:06 S3 Copying cg-6a89718b-e7b6-4f63-a2ae-78063df394e7 to local disk.  
   2024-06-04T14:51:06.56-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:06 Running command: /home/vcap/app/bin/aws s3 cp s3://cg-6a89718b-e7b6-4f63-a2ae-78063df394e7/backups/daily/06-04/public-census_historical_migration_elecnotes.dump ./pg_dump_tables/
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:07 Completed 256.0 KiB/38.9 MiB (2.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 512.0 KiB/38.9 MiB (5.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 768.0 KiB/38.9 MiB (7.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 1.0 MiB/38.9 MiB (10.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 1.2 MiB/38.9 MiB (12.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 1.5 MiB/38.9 MiB (14.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 1.8 MiB/38.9 MiB (16.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 2.0 MiB/38.9 MiB (18.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 2.2 MiB/38.9 MiB (20.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 2.5 MiB/38.9 MiB (22.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 2.8 MiB/38.9 MiB (24.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 3.0 MiB/38.9 MiB (26.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 3.2 MiB/38.9 MiB (28.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 3.5 MiB/38.9 MiB (30.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 3.8 MiB/38.9 MiB (32.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 4.0 MiB/38.9 MiB (34.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 4.2 MiB/38.9 MiB (35.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 4.5 MiB/38.9 MiB (37.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 4.8 MiB/38.9 MiB (38.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 5.0 MiB/38.9 MiB (40.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 5.2 MiB/38.9 MiB (42.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 5.5 MiB/38.9 MiB (43.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 5.8 MiB/38.9 MiB (45.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 6.0 MiB/38.9 MiB (47.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 6.2 MiB/38.9 MiB (48.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 6.5 MiB/38.9 MiB (50.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 6.8 MiB/38.9 MiB (51.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 7.0 MiB/38.9 MiB (52.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 7.2 MiB/38.9 MiB (54.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 7.5 MiB/38.9 MiB (55.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 7.8 MiB/38.9 MiB (57.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 8.0 MiB/38.9 MiB (58.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 8.2 MiB/38.9 MiB (59.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 8.5 MiB/38.9 MiB (60.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 8.8 MiB/38.9 MiB (62.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 9.0 MiB/38.9 MiB (63.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 9.2 MiB/38.9 MiB (65.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 9.5 MiB/38.9 MiB (65.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 9.8 MiB/38.9 MiB (67.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 10.0 MiB/38.9 MiB (68.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 10.2 MiB/38.9 MiB (69.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 10.5 MiB/38.9 MiB (71.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 10.8 MiB/38.9 MiB (71.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 11.0 MiB/38.9 MiB (73.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 11.2 MiB/38.9 MiB (74.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 11.5 MiB/38.9 MiB (75.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 11.8 MiB/38.9 MiB (76.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 12.0 MiB/38.9 MiB (77.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 12.2 MiB/38.9 MiB (79.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 12.5 MiB/38.9 MiB (79.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 12.8 MiB/38.9 MiB (80.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 13.0 MiB/38.9 MiB (81.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 13.2 MiB/38.9 MiB (82.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 13.5 MiB/38.9 MiB (84.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 13.8 MiB/38.9 MiB (84.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 14.0 MiB/38.9 MiB (86.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 14.2 MiB/38.9 MiB (87.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 14.5 MiB/38.9 MiB (87.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 14.8 MiB/38.9 MiB (89.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 15.0 MiB/38.9 MiB (89.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 15.2 MiB/38.9 MiB (90.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 15.5 MiB/38.9 MiB (91.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 15.8 MiB/38.9 MiB (92.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 16.0 MiB/38.9 MiB (94.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 16.2 MiB/38.9 MiB (94.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 16.5 MiB/38.9 MiB (95.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 16.8 MiB/38.9 MiB (96.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 17.0 MiB/38.9 MiB (97.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 17.2 MiB/38.9 MiB (98.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 17.5 MiB/38.9 MiB (98.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 17.8 MiB/38.9 MiB (99.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 18.0 MiB/38.9 MiB (100.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 18.2 MiB/38.9 MiB (101.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 18.5 MiB/38.9 MiB (102.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 18.8 MiB/38.9 MiB (102.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 19.0 MiB/38.9 MiB (103.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 19.2 MiB/38.9 MiB (104.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 19.5 MiB/38.9 MiB (105.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 19.8 MiB/38.9 MiB (106.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 20.0 MiB/38.9 MiB (106.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 20.2 MiB/38.9 MiB (107.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 20.5 MiB/38.9 MiB (107.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 20.8 MiB/38.9 MiB (108.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 21.0 MiB/38.9 MiB (109.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 21.2 MiB/38.9 MiB (109.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 21.5 MiB/38.9 MiB (110.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 21.8 MiB/38.9 MiB (110.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 22.0 MiB/38.9 MiB (111.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 22.2 MiB/38.9 MiB (111.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 22.5 MiB/38.9 MiB (112.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 22.8 MiB/38.9 MiB (113.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 23.0 MiB/38.9 MiB (114.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 23.2 MiB/38.9 MiB (115.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 23.5 MiB/38.9 MiB (115.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 23.8 MiB/38.9 MiB (116.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 24.0 MiB/38.9 MiB (117.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 24.2 MiB/38.9 MiB (118.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 24.5 MiB/38.9 MiB (119.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 24.8 MiB/38.9 MiB (118.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 25.0 MiB/38.9 MiB (118.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 25.2 MiB/38.9 MiB (119.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 25.5 MiB/38.9 MiB (120.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 25.8 MiB/38.9 MiB (120.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 26.0 MiB/38.9 MiB (120.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 26.2 MiB/38.9 MiB (121.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 26.5 MiB/38.9 MiB (122.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 26.8 MiB/38.9 MiB (122.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 27.0 MiB/38.9 MiB (123.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 27.2 MiB/38.9 MiB (124.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 27.5 MiB/38.9 MiB (124.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 27.8 MiB/38.9 MiB (125.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 27.9 MiB/38.9 MiB (125.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 28.1 MiB/38.9 MiB (125.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 28.4 MiB/38.9 MiB (126.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 28.6 MiB/38.9 MiB (127.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 28.9 MiB/38.9 MiB (127.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 29.1 MiB/38.9 MiB (128.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 29.4 MiB/38.9 MiB (129.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 29.6 MiB/38.9 MiB (129.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 29.9 MiB/38.9 MiB (130.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 30.1 MiB/38.9 MiB (130.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 30.4 MiB/38.9 MiB (131.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 30.6 MiB/38.9 MiB (131.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 30.9 MiB/38.9 MiB (132.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 31.1 MiB/38.9 MiB (133.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 31.4 MiB/38.9 MiB (134.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 31.6 MiB/38.9 MiB (133.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 31.9 MiB/38.9 MiB (133.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 32.1 MiB/38.9 MiB (134.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 32.4 MiB/38.9 MiB (135.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 32.6 MiB/38.9 MiB (134.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 32.9 MiB/38.9 MiB (135.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 33.1 MiB/38.9 MiB (136.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 33.4 MiB/38.9 MiB (136.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 33.6 MiB/38.9 MiB (136.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 33.9 MiB/38.9 MiB (137.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 34.1 MiB/38.9 MiB (138.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 34.4 MiB/38.9 MiB (138.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 34.6 MiB/38.9 MiB (138.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 34.9 MiB/38.9 MiB (139.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 35.1 MiB/38.9 MiB (139.9 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 35.4 MiB/38.9 MiB (139.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 35.6 MiB/38.9 MiB (139.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 35.9 MiB/38.9 MiB (140.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 36.1 MiB/38.9 MiB (141.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 36.4 MiB/38.9 MiB (141.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 36.6 MiB/38.9 MiB (141.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 36.9 MiB/38.9 MiB (142.5 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 37.1 MiB/38.9 MiB (143.1 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 37.4 MiB/38.9 MiB (143.3 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 37.6 MiB/38.9 MiB (144.0 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 37.9 MiB/38.9 MiB (144.6 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 38.1 MiB/38.9 MiB (145.2 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 38.4 MiB/38.9 MiB (145.7 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 38.6 MiB/38.9 MiB (146.4 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT Completed 38.9 MiB/38.9 MiB (146.8 MiB/s) with 1 file(s) remaining
   2024-06-04T14:51:07.48-0400 [APP/TASK/s3_restore_util/0] OUT download: s3://cg-6a89718b-e7b6-4f63-a2ae-78063df394e7/backups/daily/06-04/public-census_historical_migration_elecnotes.dump to pg_dump_tables/public-census_historical_migration_elecnotes.dump
   2024-06-04T14:51:08.43-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:08 TRUNCATE truncating table census_historical_migration_elecnotes    
   2024-06-04T14:51:08.43-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:08 /home/vcap/deps/0/apt/usr/lib/postgresql/15/bin/psql running       
   2024-06-04T14:51:08.47-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:08 RESTORE /home/vcap/deps/0/apt/usr/lib/postgresql/15/bin/pg_restore targeting public.census_historical_migration_elecnotes
   2024-06-04T14:51:18.70-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:18 pg_restore: connecting to database for restore
   2024-06-04T14:51:18.70-0400 [APP/TASK/s3_restore_util/0] OUT pg_restore: processing data for table "public.census_historical_migration_elecnotes"
   2024-06-04T14:51:18.70-0400 [APP/TASK/s3_restore_util/0] OUT pg_restore: executing SEQUENCE SET census_historical_migration_elecnotes_id_seq
   2024-06-04T14:51:18.70-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:18 RESTORE of table census_historical_migration_elecnotes complete.
   2024-06-04T14:51:18.71-0400 [APP/TASK/s3_restore_util/0] OUT INFO: 2024/06/04 18:51:18 REMOVING FILE: public-census_historical_migration_elecnotes.dump

db_restore: Workflow Run
Operations: db to db restore, aws s3 sync
image

@asteel-gsa asteel-gsa requested a review from jadudm June 21, 2024 17:46
jadudm
jadudm approved these changes Jun 21, 2024
View reviewed changes
Copy link
Contributor

@jadudm jadudm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Fixes a rebase collision.

@asteel-gsa asteel-gsa changed the title [Merging Monday] Integration of fac-backup-utility Integration of fac-backup-utility Jun 24, 2024
@GSA-TTS GSA-TTS deleted a comment from github-actions bot Jun 24, 2024
@GSA-TTS GSA-TTS deleted a comment from github-actions bot Jun 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 24, 2024 •
edited
Loading

Terraform plan for dev

Plan: 2 to add, 0 to change, 0 to destroy. 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create

Terraform will perform the following actions:

  # module.dev-backups-bucket.cloudfoundry_service_instance.bucket will be created
+   resource "cloudfoundry_service_instance" "bucket" {
+       id                             = (known after apply)
+       name                           = "backups"
+       replace_on_params_change       = false
+       replace_on_service_plan_change = false
+       service_plan                   = "021bb2a3-7e11-4fc2-b06b-d9f5938cd806"
+       space                          = "06525ba3-19c2-451b-96e9-ea4a9134e8b9"
+       tags                           = [
+           "s3",
        ]
    }

  # module.dev.module.snapshot-database.cloudfoundry_service_instance.rds will be created
+   resource "cloudfoundry_service_instance" "rds" {
+       id                             = (known after apply)
+       json_params                    = jsonencode(
            {
+               storage = 50
            }
        )
+       name                           = "fac-snapshot-db"
+       replace_on_params_change       = false
+       replace_on_service_plan_change = false
+       service_plan                   = "815c6069-289a-4444-ba99-40f0fa03a8f5"
+       space                          = "06525ba3-19c2-451b-96e9-ea4a9134e8b9"
+       tags                           = [
+           "rds",
        ]
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Warning: Argument is deprecated

  with module.dev-backups-bucket.cloudfoundry_service_instance.bucket,
  on /tmp/terraform-data-dir/modules/dev-backups-bucket/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

(and 6 more similar warnings elsewhere)

❌ Error applying plan in Deploy to Development and Management Environment #721

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 24, 2024 •
edited
Loading

Terraform plan for meta

Plan: 0 to add, 1 to change, 0 to destroy. 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # module.s3-backups.cloudfoundry_service_instance.bucket will be updated in-place
!~  resource "cloudfoundry_service_instance" "bucket" {
        id                             = "040c4133-1efe-4281-a485-005960b58405"
        name                           = "backups"
!~      tags                           = [
+           "s3",
        ]
#        (4 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Warning: Argument is deprecated

  with module.s3-backups.cloudfoundry_service_instance.bucket,
  on /tmp/terraform-data-dir/modules/s3-backups/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

❌ Error applying plan in Deploy to Development and Management Environment #721

@asteel-gsa asteel-gsa added this pull request to the merge queue Jun 24, 2024
Merged via the queue into main with commit 60a89b1 Jun 24, 2024
13 checks passed
@asteel-gsa asteel-gsa deleted the as/db-backup-util branch June 24, 2024 13:30
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 24, 2024 •
edited
Loading

Terraform plan for dev

Plan: 1 to add, 0 to change, 0 to destroy. 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create

Terraform will perform the following actions:

  # module.dev-backups-bucket.cloudfoundry_service_instance.bucket will be created
+   resource "cloudfoundry_service_instance" "bucket" {
+       id                             = (known after apply)
+       name                           = "backups"
+       replace_on_params_change       = false
+       replace_on_service_plan_change = false
+       service_plan                   = "021bb2a3-7e11-4fc2-b06b-d9f5938cd806"
+       space                          = "06525ba3-19c2-451b-96e9-ea4a9134e8b9"
+       tags                           = [
+           "s3",
        ]
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Warning: Argument is deprecated

  with module.dev-backups-bucket.cloudfoundry_service_instance.bucket,
  on /tmp/terraform-data-dir/modules/dev-backups-bucket/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

(and 6 more similar warnings elsewhere)

✅ Plan applied in Deploy to Development and Management Environment #721

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 24, 2024 •
edited
Loading

Terraform plan for meta

Plan: 0 to add, 1 to change, 0 to destroy. 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # module.s3-backups.cloudfoundry_service_instance.bucket will be updated in-place
!~  resource "cloudfoundry_service_instance" "bucket" {
        id                             = "040c4133-1efe-4281-a485-005960b58405"
        name                           = "backups"
!~      tags                           = [
+           "s3",
        ]
#        (4 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Warning: Argument is deprecated

  with module.s3-backups.cloudfoundry_service_instance.bucket,
  on /tmp/terraform-data-dir/modules/s3-backups/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

✅ Plan applied in Deploy to Development and Management Environment #721

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jadudm jadudm jadudm approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[MVP] FAC Backup Utility
2 participants
@asteel-gsa @jadudm