feat(aws): add credentials loading mechanism using sdk config

[iamKunalGupta]

Important function is GetAWSCredentialsProvider in flow/connectors/utils/aws.go

Order (in descending preference):

1. Peer Credentials
2. Explicit PEERDB_<CONNECTOR-NAME>_AWS_CREDENTIALS_<AWS-ENV-NAME> like PEERDB_CLICKHOUSE_AWS_CREDENTIALS_AWS_ACCESS_KEY_ID
3. Default creds loaded by AWS sdk

Pending Items:

• Resolve TODOs
• Clickhouse Test - ONLY WORKS WITH versions after 18th March 2024
• Snowflake Test
• @iskakaushik this is blocked until query s3 with select and aws credentials ClickHouse/ClickHouse#61230 gets to a stable version of clickhouse.
• Retest:

  • incompatible version check

    

  • compatible version check

    
• Check what version clickhouse cloud is currently on
  • Most customers are on 24.2
• Run a long test for 1 day to make sure creds are refreshed

[iskakaushik]

@iamKunalGupta can you file an issue to make an env param PEERDB_DEFAULT_STAGE_LOCATION=<local | s3> and add that to your queue please?

[iamKunalGupta]

@iamKunalGupta can you file an issue to make an env param PEERDB_DEFAULT_STAGE_LOCATION=<local | s3> and add that to your queue please?

@iskakaushik this will lead to issues where s3 staging is supported but not required, like snowflake. It will add additional costs and overhead if enabled by default.

[iskakaushik]

@iamKunalGupta this could be useful for customers who wish to retain the raw streams of CDC for long periods of time, hence having the option would be nice.

[iamKunalGupta]

Tested: