DuckDB Secrets

[samansmink]

This PR introduces a new concept to DuckDB: Secrets.

Currently in DuckDB, parameters for authentication/configuration of services is now passed through regular settings, e.g for S3:

SET s3_access_key_id='.....'
SET s3_secret_access_key='.....'

Problems with current approach

There are several issues with the current approach:

• Using settings for secrets results in a messy user interface. Currently, there is just a big pile of settings that are added whenever an extension loads that require storing these settings.
• There is no clean way to have scoped secrets currently. This means that queries that require multiple sets of credentials are not possible. For S3 this is currently solved using fully qualified urls, but that is a bit of a hack.
• No clean way to handle the different similar-but-slightly-different configurations for S3/R2/GCS.
• No clean way to handle azure, where different storage accounts + different auth modes make things confusing.
• No clean interface to store secrets like the env/cli authentication in azure.
• It is way too easy to accidentally print sensitive information as user; from duckdb_settings() just spits out secrets.
• Handling these settings is not transactionally safe

Solution: Secrets

The solution to all our problems is to add the concept of Secrets and the SecretManager to DuckDB. This enables us to:

• Have a uniform user interface for these parameters (can be used by httpfs, azure, mysql, postgres, etc.)
• Allow easy scoped secrets
• Persistent secrets between DuckDB restarts
• Support for both serializable and non-serializable secrets.
• Secrets are transactionally safe (for 1 instance of DuckDB)
• Fully separate code path: allows hardening against exploits / bugs
• Have an extensible interface for 3rd parties to build secret management solutions on top of DuckDB

Core Concepts

Essentially, secrets are scoped sets of parameters, that often contain sensitive authentication information. For example, an s3 secret could consists of an S3 api key together with an s3 region. With this secret, duckdb can authenticate request to an s3 compatible api.

Note that secrets are implemented in a very general way: a secret can essentially be anything: a set of key-value pairs, a shared_ptr to some c++ object, or just a random blob of bytes. Basically a the idea is that one part of DuckDB can register a secret, whatever it may look like, and then at a later point another part of duckdb can retrieve it again to use it for authentication/configuration. Note that in many cases secrets will be just a set of key-value pairs though.

Now to fully understand secrets, you will need to understand several core concepts.

Firstly, the secret type is what service/mechanism a secret can be used for. For 0.10.0 the goal is to have at least 4 secret types: S3, GCS, R2, and AZURE. To be clear, anS3 type secret can be used for authenticating to an S3 API. An AZURE type secret will authenticate for an Azure api, etc.

The secret provider is how the secret is created. You could see it as the interface of secret creation. Each secret type will have 1 or more secret providers that can help a user in secret creation. An example of a secret provider would be the (yet to be implemented) aws_default_chain provider, which will use the AWS SDK to automatically detect credentials and create the secret. Another example of a provider would be the connection_string provider, which means the user needs to provide a connection string, which is then used in secret creation. Note that depending on which provider chosen by the user, different parameters to the CREATE SECRET statement will be allowed, see the examples with R2 and Azure below. To ensure this, the SecretManager enforces only the correct parameters are set for the provider during the bind phase.

The secret scope is an optional parameter to every secret that determines which paths the secret applies to. This is the mechanism that allows automatic secret resolution based on the path of a file. When fetching a secret for a path, the secret scopes are compared to the path, returning the matching secret for the path. In the case of multiple matching secrets, the longest prefix is chosen.

Secrets can be temporary or persistent. Temporary secrets are stored in memory for the life span of the duckdb instance. persistent secrets are stored in unecrypted binary format in the ~/.duckdb/stored_secrets directory. On startup of duckdb, persistent secrets are scanned from this directory and automatically loaded. Note that this PR also adds the ability for extensions to implement their own persistent secret storages. By default however, there is only 1 storage for persistent secrets which is called the the local_file storage.

Current Implementation

This PR contains a relatively complete implementation of secrets and the secret manager:

• Temporary and persistent secrets
• Support for serializable and non-serializable secrets
• Secret types: S3, R2 and GCS
• Providers for above type: config(default), and duckdb_settings (for testing)
• Best matching scope resolution

Syntax

Overview

The full syntax for create secret is:

CREATE [OR REPLACE] [PERSISTENT | TEMPORARY] SECRET [IF NOT EXISTS] [secret_name] [IN storage_specifier] (
 	TYPE secret_type,
   	[PROVIDER provider,]
	[SCOPE path_or_list_of_paths,]
	[KEY1 value1],
	[KEY2 value2],
	...
)

And for the DROP:

DROP [PERSISTENT | TEMPORARY] SECRET [IF EXISTS] secret_name [FROM storage_specifier];

Basic Examples

Basic s3 authentication with a temporary secret looks like:

CREATE SECRET (
    TYPE S3,
    KEY_ID 'mykey',
    SECRET 'mysecret',
    REGION 'myregion',
)

R2 secrets will automatically set the correct endpoint and region, and require users to pass an extra parameter called account_id

statement ok
CREATE SECRET (
    TYPE R2,
    ACCOUNT_ID 'some_account',
    KEY_ID 'my_r2_key',
    SECRET 'my_r2_secret'
)

GCS secrets will automatically set endpoint, url style, region, only requiring a key and secret.

statement ok
CREATE SECRET (
    TYPE GCS,
    KEY_ID 'my_key',
    SECRET 'my_secret'
);

Secret providers

Each secret type has a default provider. For the S3 secret type, this is the config provider. The config provider simply allows manually setting all s3 authentication settings that previously would be set through SET s3_<...>. When no provider is specified, the default config provider is chosen. This PR also contains a demo provider called duckdb_settings. What the duckdb_settings provider does is very simple: it parses the current s3 settings like s3_region and s3_access_key_id and creates a secret from them. So for example:

CREATE SECRET (
	TYPE S3,
	PROVIDER duckdb_settings
);

will grab all s3 related settings we use now, and wrap it in a secret. In this PR we use it for testing. When this PR is merged however, the aws extension will implement different providers based on the aws sdk's functionality for credential acquisition. One of them will be aws_default_chain which does the same thing as the current CALL load_aws_credentials() function.

Scoped secrets.

When a secret is requested in DuckDB from the secret manager, the secret with the longest matching prefix as scope will be chosen. This allows finegrained, but intuitive control over scoping.

For example when 2 secrets are created:

CREATE SECRET s1 (
    TYPE GCS,
    KEY_ID 'my_key1',
    SECRET 'my_secret1'
    SCOPE 'gcs://my-container'
)
CREATE SECRET s2 (
    TYPE GCS,
    KEY_ID 'my_key2',
    SECRET 'my_secret2'
    SCOPE 'gcs://'
)

and the user queries: SELECT * FROM "gcs://my-container", secret s1 will be chosen.

Note that the scope parameter can contain a list of scope prefixes as well:

CREATE SECRET s2 (
    TYPE GCS,
    KEY_ID 'my_key2',
    SECRET 'my_secret2',
    SCOPE: ['gcs://container1', 'gcs://container2']
)

Listing secrets

Secrets are listed using:

SELECT * FROM duckdb_secrets();

Which prints:

• name
• type
• provider
• storage (how the secret is stored: either in-memory or path/to/secret)
• scope
• secret_string (the stringified token, with sensitive info redacted (can be turned of))

Persistent secrets

Default secret storage: local_file

By default, persistent secrets are stored in the ~/.duckdb/stored_secrets directory and loaded automatically on startup. Persistent secrets are enabled by default, but secrets are by default temporary, so to create a persistent secret, the keyword PERSISTENT needs to be explicitly passed:

statement ok
CREATE PERSISTENT SECRET my_persistent_secret (
    TYPE S3,
    KEY_ID 'key',
    SECRET 'secret'
)

Custom secret storage

The SecretManager allows extending secret storage by providing the ability to custom secret storage. Users can then choose in which storage to store their secret, for example:

CREATE PERSISTENT SECRET p1 IN my_custom_secret_storage (
    TYPE S3,
    KEY_ID 'key',
    SECRET 'secret'
)

configuration

DuckDB currently has 3 configuration params related to persistent secrets: SET secret_directory='' can be used to specify in which local directory the secrets are stored, and SET allow_persistent_secrets=false; can be used to completely disable use of persistent secrets. Note that these setting can only be changed before the secret manager is used; so after any sort of secret creation/deletion/list/use these settings are fixed for the lifetime of the dbinstance. Finally, SET default_secret_storage='' is used to switch the default storage for persistent secrets

Yak shaving

Patch disabling substrait tests

The CI for extension tests was not properly ran so these breakages were hidden. I've disabled the breaking tests for now with a patch

Fix small DefaultGenerator issue

DefaultGenerators would not always be added to the dependency manager

Followup:

• AWS and Azure implementations of secrets follow soon
• Deprecation of s3_<..> and azure_<..> credential related settings
• Deprecation of fully qualified urls (query params in s3 urls)
• Small cleanup with regards to serialization of stored secrets
  • serialization refactor?
  • add test for backwards compatibility
  • add version number to files

Even later:

• mysql and postgres
• remove s3_<..> and azure_<..> settings

[Mytherin]

Thanks! Looks excellent - some very minor comments and then this is good to go

[Mytherin]

TODO for a later PR