Skip to content

Conversation

NickLarsenNZ
Copy link
Member

@NickLarsenNZ NickLarsenNZ commented Mar 11, 2025

Description

Part of stackabletech/issues#696.

Caution

Blocked on testing. Need to test how the global region setting works for buckets in different regions.

Per s3 extension docs:

  • Add -Daws.region=us-east-1 to the jvm.config file for all Druid services.
  • Add -Daws.region=us-east-1 to druid.indexer.runner.javaOpts in Middle Manager configuration so that the property will be passed to Peon (worker) processes.

Note

I did not add it to jvm.config, as it will be specific to the bucket being accessed.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes
# Author
- [x] CRD changes approved
- [x] CRD documentation for all fields, following the [style guide](https://docs.stackable.tech/home/nightly/contributor/docs/style-guide).
- [ ] Helm chart can be installed and deployed operator works
- [ ] Integration tests passed (for non trivial changes)
# Reviewer
- [ ] Code contains useful comments
- [ ] Code contains useful logging statements
- [ ] (Integration-)Test cases added
- [ ] Documentation added or updated. Follows the [style guide](https://docs.stackable.tech/home/nightly/contributor/docs/style-guide).
- [ ] Changelog updated
- [ ] Cargo.toml only contains references to git tags (not specific commits or branches)
# Acceptance
- [ ] Feature Tracker has been updated
- [ ] Proper release label has been added
- [ ] [Roadmap](https://github.com/orgs/stackabletech/projects/25/views/1) has been updated

@NickLarsenNZ NickLarsenNZ moved this to Development: Waiting for Review in Stackable Engineering Mar 11, 2025
};

if let Some(s3) = s3_conn {
conf.insert(S3_REGION_NAME.to_string(), Some(s3.region.name.to_string()));
Copy link
Member

@sbernauer sbernauer Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think according to https://druid.apache.org/docs/latest/development/extensions-core/s3/#aws-region aws.region needs to go into the JVM args, not the Druid config.
Via env var AWS_REGION sounds even easier to me

Copy link
Member Author

@NickLarsenNZ NickLarsenNZ Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does, but that also seems strange to be since you might want to connect to buckets in multiple regions.

The S3 extension needs testing with AWS S3 buckets:

  • Deep storage
  • S3 Ingestion

I'm just trying to get a working baseline.

@NickLarsenNZ
Copy link
Member Author

This won't be done because:

  • Druid uses the AWS SDK v1 which ignores the region setting if the endpoint has been specified.
  • Our CRDs require the hostname (therefore the endpoint will be set), so the region must be ignored.
    • I considered adding a warning if the region is set, however it will always appear set since it defaults to us-east-1. If it were a pub const in stackable-operator, it would have been ok to check if it has been set differently to the default and emit a warning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants