Skip to content

fix: Restore respect for druid configs controlling s3 multipart upload configs - #19890

Merged
gianm merged 2 commits into
apache:masterfrom
capistrant:aws-multipart-config-restore
Aug 6, 2026
Merged

fix: Restore respect for druid configs controlling s3 multipart upload configs#19890
gianm merged 2 commits into
apache:masterfrom
capistrant:aws-multipart-config-restore

Conversation

@capistrant

@capistrant capistrant commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Description

Fix a regression in that we were not honoring our s3 client defaults for multipart uploads + not honoring any operator override. Without this change we use multipart but use the sdk default which appears to be 8MB chunks instead of our default of 20.

I have personally felt this impact during MSQ compaction with s3 throttling during the segment pushing at the end of a task. In investigating, I noticed that even if I tried to modify the configs, they were not being honored by the client.

Release note

Fixes a regression that was preventing s3 multipart upload configs from taking effect.


Key changed/added classes in this PR
  • ServerSideEncryptingAmazonS3.java

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@capistrant capistrant changed the title fix: Restore respect for druid configs controlling s3 multipart upload configs bug: Restore respect for druid configs controlling s3 multipart upload configs Aug 5, 2026
{
final String unquoted = StringUtils.replace(eTag, "\"", "");
final int dash = unquoted.lastIndexOf('-');
return dash < 0 ? 1 : Integer.parseInt(unquoted.substring(dash + 1));
@capistrant capistrant changed the title bug: Restore respect for druid configs controlling s3 multipart upload configs fix: Restore respect for druid configs controlling s3 multipart upload configs Aug 5, 2026
@gianm
gianm merged commit 5c65e19 into apache:master Aug 6, 2026
29 of 35 checks passed
@github-actions github-actions Bot added this to the 39.0.0 milestone Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants