Skip to content

Configure S3Hook transfer_config_args via service_config#65963

Merged
o-nikolas merged 1 commit intoapache:mainfrom
stephen-bracken:s3-transfer-config
Apr 28, 2026
Merged

Configure S3Hook transfer_config_args via service_config#65963
o-nikolas merged 1 commit intoapache:mainfrom
stephen-bracken:s3-transfer-config

Conversation

@stephen-bracken
Copy link
Copy Markdown
Contributor

Enable configuring the S3 transfer config arguments using the service config

Previously the transfer_config_args could only be configured by passing in arguments to the S3Hook object. This PR makes the S3Hook also read arguments from the s3 service_config in the connection's extras to override this behaviour. e.g.

"extras": {
  "service_config": {
    "s3": {
      "transfer_config_args": {
        "use_threads": true
      }
    }
  }
}

This allows deployment managers to change the transfer config options via configuration, for example in S3TaskHandler.io

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
  • No

@boring-cyborg boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Apr 27, 2026
@stephen-bracken stephen-bracken force-pushed the s3-transfer-config branch 2 times, most recently from c2be3f4 to ca9d389 Compare April 27, 2026 19:53
@stephen-bracken stephen-bracken marked this pull request as ready for review April 27, 2026 21:00
Comment thread providers/amazon/docs/connections/aws.rst
@o-nikolas o-nikolas merged commit 5795508 into apache:main Apr 28, 2026
93 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants