fix: AWS auth fixes

[kujon]

Database name

Any

Pull request description

Fixes #886, #562

Describe what this PR fix

wal-g supports a very limited number of methods of authenticating with AWS. This PR does 2 things:

• it deprecates explicitly passing in aws credentials
• it makes use of aws.credentials.ChainProvider to quack the same as e.g. aws cli.

It's a remake of this PR from a now deprecated repo.

[usernamedt]

Hi! As I can see, this issue has been already addressed in wal-g/storages#45 and #1183. Can you please clarify the difference between this PR and the master branch (except the warning messages)?

[mialinx]

There are two moments:

• S3 is more than AWS object, storage there are other S3-compatible (Yandex Cloud for instance), which provides S3 REST API, but not AWS-like configuration. So, AWS_ACCESS_KEY_ID and others can't be deprecated at the moment.
• There is a scenario with copying backups from one storage to another. In this case, we have to configure multiple storage with different bucket_names (and may be access keys). How can we handle it without AWS_* options in config ?

[mialinx]

It looks like the problem is that we use storage settings (such as AWS_ACCESS_KEY_ID) to choose configured storage.
Maybe we need to add explicit storage type setting like WALG_STORAGE=S3 or WALG_STORAGE=SSH, leaving current method as fallback. We will be able to use S3 storage without any (AWS_...) options in config at all, just as @kujon suggests.

@usernamedt ?

[usernamedt]

It looks like the problem is that we use storage settings (such as AWS_ACCESS_KEY_ID) to choose configured storage. Maybe we need to add explicit storage type setting like WALG_STORAGE=S3 or WALG_STORAGE=SSH, leaving current method as fallback. We will be able to use S3 storage without any (AWS_...) options in config at all, just as @kujon suggests.

@usernamedt ?

We already don't rely on any AWS_* auth setting to determine that storage type is AWS. WAL-G uses the WALG_*_PREFIX for this purpose, i.e. WALG_S3_PREFIX in the case of AWS.

[mialinx]

It looks like the problem is that we use storage settings (such as AWS_ACCESS_KEY_ID) to choose configured storage. Maybe we need to add explicit storage type setting like WALG_STORAGE=S3 or WALG_STORAGE=SSH, leaving current method as fallback. We will be able to use S3 storage without any (AWS_...) options in config at all, just as @kujon suggests.
@usernamedt ?

We already don't rely on any AWS_* auth setting to determine that storage type is AWS. WAL-G uses the WALG_*_PREFIX for this purpose, i.e. WALG_S3_PREFIX in the case of AWS.

Oh, my bad )
Well, then we could just remove deprecation comments warnings (as we plan to AWS_ACCESS_KEY_ID for a long time) and merge this PR.

[kujon]

It looks like the problem is that we use storage settings (such as AWS_ACCESS_KEY_ID) to choose configured storage. Maybe we need to add explicit storage type setting like WALG_STORAGE=S3 or WALG_STORAGE=SSH, leaving current method as fallback. We will be able to use S3 storage without any (AWS_...) options in config at all, just as @kujon suggests.
@usernamedt ?

We already don't rely on any AWS_* auth setting to determine that storage type is AWS. WAL-G uses the WALG_*_PREFIX for this purpose, i.e. WALG_S3_PREFIX in the case of AWS.

Oh, my bad )
Well, then we could just remove deprecation comments warnings (as we plan to AWS_ACCESS_KEY_ID for a long time) and merge this PR.

Will do, give me a day!

[mialinx]

LGTM

[martin31821]

Any chance we can get a release containing this PR?

[x4m]

@usernamedt do you plan release\prerelease soon? Do we have something to show to the community? :)

[x4m]

@martin31821 you can try https://github.com/wal-g/wal-g/releases/tag/v2.0.0-rc1

[Jasper-Ben]

👋 @x4m, I am working with @martin31821. The good news is: v2.0.0-rc1 works for our usecase. See: zalando/postgres-operator#1124 (comment)
@usernamedt a GA release would be great! 👍

[usernamedt]

@usernamedt do you plan release\prerelease soon? Do we have something to show to the community? :)

Hi! Yes, I guess that we can cut a release soon. Most probably, the next week.