Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
v1.2.680-9e71e4d2df (rust-1.85.0-nightly-2025-01-02T06:59:02.500834814Z)
What's Wrong?
We're using Databend with S3-backed external stages configured via IAM Roles, as per the official documentation: https://docs.databend.com/guides/load-data/stage/aws-iam-role
When running large COPY queries that pull data from a stage, we frequently hit AWS STS throttling limits. Example error:
LoadCredential, service: s3, path: ....parquet, range: 0- } => loading credential to sign http request, source: request to AWS STS Services failed: <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
<Error>
<Type>Sender</Type>
<Code>Throttling</Code>
<Message>Rate exceeded</Message>
</Error>
<RequestId>...</RequestId>
</ErrorResponse>
We reproduced this with just one databend-query
instance, so it doesn't seem to be a horizontal scaling issue. The failing query is an INSERT INTO ... WITH stage_data AS (SELECT ... FROM @stage)
style query—complex but accessing a single stage only.
Can Databend or reqsign
cache credentials to reduce or avoid the rate of calls to AWS STS, especially in large workloads?
Let me know if you need logs or further information. Thanks!
How to Reproduce?
Here's a snippet of a sample query we used to reproduce it without actually copying/inserting the data too:
CREATE OR REPLACE STAGE example_stage
URL='s3://your-bucket/path/to/data/'
CONNECTION=(ROLE_ARN='arn:aws:iam::<account-id>:role/...')
FILE_FORMAT=(TYPE='PARQUET')
COMMENT='';
SELECT DISTINCT
col_1,
NULLIF(col_2, '')::TIMESTAMP as col_2,
col_3,
...
NULLIF(col_n, '')::DATE as col_n
FROM @example_stage
WHERE col_n >= TODAY() - 1
AND col_key IN (
SELECT col_key
FROM reference_config_table
WHERE active
)
LIMIT 100;
Are you willing to submit PR?
- Yes I am willing to submit a PR!