CUMULUS-3536 Update Cumulus Core from Aurora Serverless V1 to V2 #3632

tclark-innovim · 2024-04-19T14:29:27Z

Summary: Summary of changes

Addresses CUMULUS-3536: Update Cumulus Core from Aurora Serverless V1 to V2

Changes

Combined updates to support AWS Aurora Serverless v2 for Postgres.

PR Checklist

Update CHANGELOG
Unit tests
Ad-hoc testing - Deploy changes and test manually
Integration tests

* update CL * update terraform templates to serverless v2 * add terraform variable validation * remove upgrade variables * add prevent_destroy = true * add prevent_destroy = true --------- Co-authored-by: Tim Clark <tim.clark@nasa.gov>

# Conflicts: # CHANGELOG.md

…o v2 (#3643) * remove prevent_destroy to allow automated CI migrations --------- Co-authored-by: Tim Clark <tim.clark@nasa.gov>

Co-authored-by: Tim Clark <tim.clark@nasa.gov>

* initial commit * serverless v2 doc updates * Update serverless V2 docs * Fix lint issue

# Conflicts: # CHANGELOG.md

tclark-innovim · 2024-06-28T19:52:00Z

lambdas/db-provision-user-database/src/index.ts

@@ -83,6 +83,7 @@ export const handler = async (event: HandlerEvent): Promise<void> => {
        database: `${dbUser}_db`,
        host: (rootKnexConfig.connection as Knex.PgConnectionConfig).host,
        port: (rootKnexConfig.connection as Knex.PgConnectionConfig).port,
+        disableSSL: 'true',


I had to add disableSSL: 'true' to this file in order for Bamboo CI in order for the Deploy Dev Integration Stack to complete successfully. Perhaps we need to add documentation around this addition?

@Jkovarik This is the additional change I needed to make in order for Bamboo CI in order for the Deploy Dev Integration Stack to complete successfully

We should probably consider making this a user option as part of the incoming event.

That said, the behavior is documented (not as part of this module, but in Core) as part of the ticket CL:

- **CUMULUS-3323** - Added `disableSSL` as a valid database secret key - setting this in your database credentials will disable SSL for all Core database connection attempts. - Added `rejectUnauthorized` as a valid database secret key - setting this to `false` in your database credentials will allow self-signed certs/certs with an unrecognized authority. - Updated the default parameter group for `cumulus-rds-tf` to set `force_ssl` to 1. This setting for the Aurora Serverless v1 database disallows non-SSL connections to the database, and is intended to help enforce security compliance rules. This update can be opted-out by supplying a non-default `db_parameters` set in the terraform configuration.

and

https://github.com/nasa/cumulus/pull/3587/files#diff-7b65a9166b23c12f9723277e9247262dceec1df189b6b67e3a471da1fe8316ab

That said - why do we need to disable SSL for v2? Is this the cert concern, and if so can we leave SSL enabled but allow unverified certs (e.g. rejectUnauthorized)?

I have removed the disableSsl entry and added rejectUnauthorized = false. Bamboo CI ran successfully.

# Conflicts: # CHANGELOG.md

Jkovarik

Good work - just one question re: the current SSL default settings.

# Conflicts: # CHANGELOG.md

# Conflicts: # tf-modules/cumulus-rds-tf/main.tf # tf-modules/cumulus-rds-tf/variables.tf

tclark-innovim · 2024-07-17T13:18:44Z

Good work - just one question re: the current SSL default settings.

@Jkovarik I have removed the disableSsl entry and added rejectUnauthorized = false. Bamboo CI ran successfully.

# Conflicts: # CHANGELOG.md

tclark-innovim · 2024-07-23T15:00:13Z

@Jkovarik Hi Jonathan, it looks like Github is showing "Merging is blocked" due to a change request you have added to this PR. Are there any further changes you are looking for? If not, can you clear the change request so that this feature branch is able to merge, once the time is right? Thanks

made this change

npauzenga

@tclark-innovim the updates look good so I'm gonna go ahead and approve. I do want to be sure we (I) understand the rejectUnauthorized though. It's reasonable to default that the way we have it?

tclark-innovim · 2024-07-23T15:33:01Z

@tclark-innovim the updates look good so I'm gonna go ahead and approve. I do want to be sure we (I) understand the rejectUnauthorized though. It's reasonable to default that the way we have it?

I set rejectUnauthorized = false as default because our Cumulus Lambda does not yet have the AWS RDS SSL root authority certificate installed. We have a future ticket for that. Once that is in place, we can remove the rejectUnauthorized = false.

# Conflicts: # CHANGELOG.md

* Update deployment templates for Aurora Serverless v2 (#3623) * update CL * update terraform templates to serverless v2 * add terraform variable validation * remove upgrade variables * add prevent_destroy = true * add prevent_destroy = true * CUMULUS-3670 Develop upgrade/migration process Aurora Serverless v1 to v2 (#3643) * remove prevent_destroy to allow automated CI migrations * set force_ssl = 0 (#3658) Co-authored-by: Tim Clark <tim.clark@nasa.gov> * [CUMULUS-3671]: Update docs for Serverless V2 (#3666) * initial commit * serverless v2 doc updates * Update serverless V2 docs * Fix lint issue * set DISABLE_PG_SSL: true to support CI * fix lint error * set disableSSL = true * remove DISABLE_PG_SSL * set rejectUnauthorized: 'false' * update CL for v2 changes * fix changelog * add migration notes to changelog, add v2 docs to sidebar * fix changelog --------- Co-authored-by: Tim Clark <tim.clark@nasa.gov> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com>

* CUMULUS-3536 Update Cumulus Core from Aurora Serverless V1 to V2 (#3632) * Update deployment templates for Aurora Serverless v2 (#3623) * update CL * update terraform templates to serverless v2 * add terraform variable validation * remove upgrade variables * add prevent_destroy = true * add prevent_destroy = true * CUMULUS-3670 Develop upgrade/migration process Aurora Serverless v1 to v2 (#3643) * remove prevent_destroy to allow automated CI migrations * set force_ssl = 0 (#3658) Co-authored-by: Tim Clark <tim.clark@nasa.gov> * [CUMULUS-3671]: Update docs for Serverless V2 (#3666) * initial commit * serverless v2 doc updates * Update serverless V2 docs * Fix lint issue * set DISABLE_PG_SSL: true to support CI * fix lint error * set disableSSL = true * remove DISABLE_PG_SSL * set rejectUnauthorized: 'false' * update CL for v2 changes * fix changelog * add migration notes to changelog, add v2 docs to sidebar * fix changelog --------- Co-authored-by: Tim Clark <tim.clark@nasa.gov> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com> * Ecarton/hailian fix sf event sqs to db records CUMULUS-3901 (#3799) * Fix isThrottlingException function to check error name * update changelog and add name/code check in errors * linter fix * changelog * typo fix --------- Co-authored-by: Hailiang Zhang <hailiang.zhang@nasa.gov> Co-authored-by: etcart <amberhosen@gmail.com> * reintroduce Migration Count Report to migrations with async_operations_table (#3805) Co-authored-by: etcart <amberhosen@gmail.com> * Updates CL 18.5.0 release date, removes 19.0, fixes some lint stuff --------- Co-authored-by: Tim Clark <tim.clark@nasa.gov> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com> Co-authored-by: cumulus-bot <141277837+cumulus-bot@users.noreply.github.com> Co-authored-by: Hailiang Zhang <hailiang.zhang@nasa.gov> Co-authored-by: Paul Pilone <paul@element84.com>

….0 release (#3814) * CUMULUS-3536 Update Cumulus Core from Aurora Serverless V1 to V2 (#3632) * Update deployment templates for Aurora Serverless v2 (#3623) * update CL * update terraform templates to serverless v2 * add terraform variable validation * remove upgrade variables * add prevent_destroy = true * add prevent_destroy = true * CUMULUS-3670 Develop upgrade/migration process Aurora Serverless v1 to v2 (#3643) * remove prevent_destroy to allow automated CI migrations * set force_ssl = 0 (#3658) Co-authored-by: Tim Clark <tim.clark@nasa.gov> * [CUMULUS-3671]: Update docs for Serverless V2 (#3666) * initial commit * serverless v2 doc updates * Update serverless V2 docs * Fix lint issue * set DISABLE_PG_SSL: true to support CI * fix lint error * set disableSSL = true * remove DISABLE_PG_SSL * set rejectUnauthorized: 'false' * update CL for v2 changes * fix changelog * add migration notes to changelog, add v2 docs to sidebar * fix changelog --------- Co-authored-by: Tim Clark <tim.clark@nasa.gov> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com> * Ecarton/hailian fix sf event sqs to db records CUMULUS-3901 (#3799) * Fix isThrottlingException function to check error name * update changelog and add name/code check in errors * linter fix * changelog * typo fix --------- Co-authored-by: Hailiang Zhang <hailiang.zhang@nasa.gov> Co-authored-by: etcart <amberhosen@gmail.com> * reintroduce Migration Count Report to migrations with async_operations_table (#3805) Co-authored-by: etcart <amberhosen@gmail.com> * Updates CL 18.5.0 release date, removes 19.0, fixes some lint stuff * Bumps packages versions to 18.5.0 * Adds 18.5.0 docs to website --------- Co-authored-by: etcart <amberhosen@gmail.com> Co-authored-by: Tim Clark <tim.clark@nasa.gov> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com> Co-authored-by: cumulus-bot <141277837+cumulus-bot@users.noreply.github.com> Co-authored-by: Hailiang Zhang <hailiang.zhang@nasa.gov>

tclark-innovim self-assigned this Apr 19, 2024

Tim Clark and others added 3 commits May 1, 2024 08:55

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

1ebeb71

# Conflicts: # CHANGELOG.md

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

b97fc5c

CUMULUS-3670 Develop upgrade/migration process Aurora Serverless v1 t…

9c254b0

…o v2 (#3643) * remove prevent_destroy to allow automated CI migrations --------- Co-authored-by: Tim Clark <tim.clark@nasa.gov>

tclark-innovim changed the title ~~Update Cumulus Core from Aurora Serverless V1 to V2~~ CUMULUS-3536 Update Cumulus Core from Aurora Serverless V1 to V2 May 7, 2024

Tim Clark and others added 10 commits May 15, 2024 08:22

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

ddc2a08

set force_ssl = 0 (#3658)

f6d3480

Co-authored-by: Tim Clark <tim.clark@nasa.gov>

[CUMULUS-3671]: Update docs for Serverless V2 (#3666)

37ef9fa

* initial commit * serverless v2 doc updates * Update serverless V2 docs * Fix lint issue

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

2786ddc

# Conflicts: # CHANGELOG.md

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

24ccfda

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

8de4f01

set DISABLE_PG_SSL: true to support CI

c0f4335

fix lint error

8b91573

set disableSSL = true

7e221e2

remove DISABLE_PG_SSL

953b248

tclark-innovim commented Jun 28, 2024

View reviewed changes

tclark-innovim added the needs review label Jun 28, 2024

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

327a845

Jkovarik added the in review label Jul 3, 2024

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

ac8159b

# Conflicts: # CHANGELOG.md

Jkovarik previously requested changes Jul 8, 2024

View reviewed changes

Jkovarik added pr feedback and removed needs review in review labels Jul 8, 2024

Tim Clark added 4 commits July 15, 2024 11:18

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

e5ec3f5

# Conflicts: # CHANGELOG.md

set rejectUnauthorized: 'false'

a01a28b

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

e94c83d

# Conflicts: # tf-modules/cumulus-rds-tf/main.tf # tf-modules/cumulus-rds-tf/variables.tf

update CL for v2 changes

e602f02

Tim Clark and others added 4 commits July 17, 2024 09:21

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

715b311

# Conflicts: # CHANGELOG.md

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

aeec4a9

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

14d1e8f

# Conflicts: # CHANGELOG.md

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

c361fcc

# Conflicts: # CHANGELOG.md

npauzenga approved these changes Jul 23, 2024

View reviewed changes

Tim Clark and others added 9 commits August 14, 2024 10:11

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

d05ad43

# Conflicts: # CHANGELOG.md

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

de0c69f

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

244cf27

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

8ec939a

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

8ebf49b

fix changelog

3b452ee

add migration notes to changelog, add v2 docs to sidebar

7af1782

fix changelog

908f115

Merge branch 'master' into feature/CUMULUS-3536-serverless-v2-epic

3903df4

tclark-innovim merged commit 7bac340 into master Sep 24, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUMULUS-3536 Update Cumulus Core from Aurora Serverless V1 to V2 #3632

CUMULUS-3536 Update Cumulus Core from Aurora Serverless V1 to V2 #3632

tclark-innovim commented Apr 19, 2024 •

edited

Loading

tclark-innovim Jun 28, 2024

tclark-innovim Jul 8, 2024

Jkovarik Jul 8, 2024 •

edited

Loading

tclark-innovim Jul 17, 2024

Jkovarik left a comment

tclark-innovim commented Jul 17, 2024

tclark-innovim commented Jul 23, 2024

npauzenga left a comment

tclark-innovim commented Jul 23, 2024

CUMULUS-3536 Update Cumulus Core from Aurora Serverless V1 to V2 #3632

CUMULUS-3536 Update Cumulus Core from Aurora Serverless V1 to V2 #3632

Conversation

tclark-innovim commented Apr 19, 2024 • edited Loading

Changes

PR Checklist

tclark-innovim Jun 28, 2024

Choose a reason for hiding this comment

tclark-innovim Jul 8, 2024

Choose a reason for hiding this comment

Jkovarik Jul 8, 2024 • edited Loading

Choose a reason for hiding this comment

tclark-innovim Jul 17, 2024

Choose a reason for hiding this comment

Jkovarik left a comment

Choose a reason for hiding this comment

tclark-innovim commented Jul 17, 2024

tclark-innovim commented Jul 23, 2024

npauzenga left a comment

Choose a reason for hiding this comment

tclark-innovim commented Jul 23, 2024

tclark-innovim commented Apr 19, 2024 •

edited

Loading

Jkovarik Jul 8, 2024 •

edited

Loading