Support AWS default credential provider chain

[tekumara]

What happens?

To use S3 (via the HTTPFS extension) requires explicitly setting AWS credentials, eg:

SET s3_region='ap-southeast-2';
SET s3_access_key_id='.....';
SET s3_secret_access_key='....';
SET s3_session_token='....';

The AWS CLI and SDK has a default credential provider chain that will automatically use environment variables if present, or the EC2 metadata endpoint, etc. This removes the need to explicitly provide credentials to applications.

If would be nice if DuckDB could implement the default credentials provider chain (or at least source credentials from environment variables) so that credentials do not need to be explicitly provided.

To Reproduce

eg: from an EC2 instance try accessing an S3 dataset without explicitly providing credentials:

$ duckdb
v0.4.0 da9ee490d
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D INSTALL httpfs;
D LOAD httpfs;
D SELECT * FROM parquet_schema('s3://topsecret/0000_part_00.parquet');
Error: Unable to connect to URL "s3://topsecret/0000_part_00.parquet'": 403 (Forbidden)

Environment (please complete the following information):

• OS: linux
• DuckDB Version: v0.4.0 da9ee49
• DuckDB Client: CLI

Identity Disclosure:

• Full Name: Oliver Mannion
• Affiliation: Xero

If the above is not given and is not obvious from your GitHub profile page, we might close your issue without further review. Please refer to the reasoning behind this rule if you have questions.

Before Submitting

• Have you tried this on the latest master branch?

• Python: pip install duckdb --upgrade --pre
• R: install.packages("https://github.com/duckdb/duckdb/releases/download/master-builds/duckdb_r_src.tar.gz", repos = NULL)
• Other Platforms: You can find binaries here or compile from source.

• Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

[mdagost]

This would be super useful for using DuckDB inside of apps running on kubernetes that get their AWS credentials via service accounts.

[slotrans]

Can't overstate how important this is! It's vital that any tool that talks to AWS use the Default Credentials Provider Chain and prefer that over explicit credentials.

[xerial]

If profile (or instance profile) authentication is used, the profile name also should be able to be set from duckdb or an environment variable. AWS SDK often has such configurations. Here is an example of the Java version of AWS SDK's default credential provider:

https://github.com/aws/aws-sdk-java-v2/blob/a95b16ea35a6b74ec5c9c471eeefb146f96da82c/core/auth/src/main/java/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.java#L92-L109

[samansmink]

This is now supported in nightly builds and the upcoming release through the AWS extension: https://github.com/duckdblabs/duckdb_aws. I'll close this issue for now, if there are any issues or missing features, feel free to open an issue in the aws extension repo!

[xerial]

@samansmink Great work! Thanks for creating this extension!