Skip to content

bug: AWS STS throttling when using S3-backed stages with IAM Roles #18012

Closed
@dixel

Description

@dixel

Search before asking

  • I had searched in the issues and found no similar issues.

Version

v1.2.680-9e71e4d2df (rust-1.85.0-nightly-2025-01-02T06:59:02.500834814Z)

What's Wrong?

We're using Databend with S3-backed external stages configured via IAM Roles, as per the official documentation: https://docs.databend.com/guides/load-data/stage/aws-iam-role

When running large COPY queries that pull data from a stage, we frequently hit AWS STS throttling limits. Example error:

LoadCredential, service: s3, path: ....parquet, range: 0- } => loading credential to sign http request, source: request to AWS STS Services failed: <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
<Error>
<Type>Sender</Type>
<Code>Throttling</Code>
<Message>Rate exceeded</Message>
</Error>
<RequestId>...</RequestId>
</ErrorResponse>

We reproduced this with just one databend-query instance, so it doesn't seem to be a horizontal scaling issue. The failing query is an INSERT INTO ... WITH stage_data AS (SELECT ... FROM @stage) style query—complex but accessing a single stage only.

Can Databend or reqsign cache credentials to reduce or avoid the rate of calls to AWS STS, especially in large workloads?

Let me know if you need logs or further information. Thanks!

How to Reproduce?

Here's a snippet of a sample query we used to reproduce it without actually copying/inserting the data too:

CREATE OR REPLACE STAGE example_stage
URL='s3://your-bucket/path/to/data/'
CONNECTION=(ROLE_ARN='arn:aws:iam::<account-id>:role/...')
FILE_FORMAT=(TYPE='PARQUET')
COMMENT='';
SELECT DISTINCT
    col_1,
    NULLIF(col_2, '')::TIMESTAMP as col_2,
    col_3,
    ...
    NULLIF(col_n, '')::DATE as col_n
FROM @example_stage
WHERE col_n >= TODAY() - 1
AND col_key IN (
    SELECT col_key
    FROM reference_config_table
    WHERE active
)
LIMIT 100;

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Metadata

Metadata

Assignees

Labels

C-bugCategory: something isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions