Skip to content

[fix][io] Fix DynamoDB source client construction with a custom endpoint#79

Merged
david-streamlio merged 3 commits into
apache:masterfrom
david-streamlio:fix/dynamodb-endpoint-region
Jul 10, 2026
Merged

[fix][io] Fix DynamoDB source client construction with a custom endpoint#79
david-streamlio merged 3 commits into
apache:masterfrom
david-streamlio:fix/dynamodb-endpoint-region

Conversation

@david-streamlio

Copy link
Copy Markdown
Contributor

Fixes #78

Motivation

The DynamoDB source's awsEndpoint / dynamoEndpoint / cloudwatchEndpoint settings are unusable. Any configuration supplying an endpoint fails at client construction:

java.lang.IllegalStateException: Only one of Region or EndpointConfiguration may be set.

All three builders in DynamoDBSourceConfig set an endpoint configuration and then, in a separate non-else branch, a region:

if (!this.getAwsEndpoint().isEmpty()) {
    builder.setEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(...));
}
if (!this.getAwsRegion().isEmpty()) {   // not an else
    builder.setRegion(this.getAwsRegion());
}

AWS SDK v1's AwsClientBuilder forbids that pairing — the EndpointConfiguration already carries the signing region. And awsRegion is mandatory: DynamoDBSource.open() enforces checkArgument(isNotBlank(getAwsRegion()), "The aws-region must be set"). So whenever an endpoint is configured, both setters run and build() throws. The endpoint path could never have worked.

It went unnoticed because production usage against real AWS leaves the endpoint empty, and the module had no test beyond config parsing (#50). Master carries a comment that reads like a previous encounter with this, resolved at the wrong layer:

// Even if the endpoint is set, it seems to require a region to go with it
checkArgument(isNotBlank(dynamodbSourceConfig.getAwsRegion()), "The aws-region must be set");

Modifications

  1. Make region and endpoint mutually exclusive (else if) in all three builders.
  2. Gate buildDynamoDBClient and buildCloudwatchClient on their own endpoint fields. They read dynamoEndpoint / cloudwatchEndpoint while gating on awsEndpoint, so setting awsEndpoint alone built an EndpointConfiguration from an empty string.

Compatibility

I do not believe any working behavior is removed:

  • Endpoint + region set → previously threw; now the endpoint wins (carrying the region as its signing region). Strictly an improvement.
  • awsEndpoint alone, for the DynamoDB/CloudWatch clients → previously built an endpoint configuration from an empty string; now falls through to the region-based default endpoint.
  • Both awsEndpoint and dynamoEndpoint set → unchanged.
  • No endpoint set (the common case against real AWS) → unchanged.

Verifying this change

Existing config tests pass:

./gradlew :dynamodb:test
DynamoDBSourceConfigTest > 5 tests PASSED
BUILD SUCCESSFUL

The fix is exercised end-to-end by the LocalStack integration test in #77, which cannot run without it — that PR is now test-only and depends on this one.

All three client builders set an endpoint configuration and then, in a
separate branch, a region. AWS SDK v1 rejects that combination with
'Only one of Region or EndpointConfiguration may be set', and
DynamoDBSource.open() requires awsRegion, so every configuration that
supplies an endpoint failed at client construction. The endpoint path
could not have worked.

Make region and endpoint mutually exclusive. Also gate the DynamoDB and
CloudWatch builders on their own endpoint fields rather than on
awsEndpoint: they were reading dynamoEndpoint and cloudwatchEndpoint
while gating on awsEndpoint, so setting awsEndpoint alone built an
endpoint configuration from an empty string.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes DynamoDB connector AWS SDK v1 client construction when custom endpoints are configured by ensuring region and endpoint configuration are not both set on the same builder, and by gating each client’s endpoint configuration on the correct endpoint field.

Changes:

  • Make setEndpointConfiguration(...) and setRegion(...) mutually exclusive (else if) for DynamoDB Streams, DynamoDB, and CloudWatch client builders.
  • Fix endpoint gating so buildDynamoDBClient uses dynamoEndpoint and buildCloudwatchClient uses cloudwatchEndpoint (instead of incorrectly gating on awsEndpoint).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +187 to 191
// EndpointConfiguration already carries the signing region, and the AWS SDK v1 forbids
// setting both an endpoint configuration and a region, so they must be mutually exclusive.
if (!this.getAwsEndpoint().isEmpty()) {
builder.setEndpointConfiguration(
new AwsClientBuilder.EndpointConfiguration(this.getAwsEndpoint(), this.getAwsRegion()));
…ndling

Cover the crash directly: constructing each client with an endpoint and
the (mandatory) region must not throw, each builder must select its own
endpoint field, and setting awsEndpoint alone must not leak an empty
endpoint into the DynamoDB and CloudWatch clients.

Three of the four fail against the unfixed builders with the original
IllegalStateException; the fourth pins the unaffected region-only path.
@david-streamlio

Copy link
Copy Markdown
Contributor Author

Good catch — a fix for a user-facing crash with no test guarding it is how this bug survived in the first place. Added DynamoDBSourceConfigClientBuilderTest (unit, no containers or network) covering exactly what you asked for:

  • testBuildersDoNotThrowWhenEndpointAndRegionAreBothSet — constructs all three clients with an endpoint and the mandatory region; construction itself is the assertion.
  • testEachBuilderSelectsItsOwnEndpoint — asserts each client resolves to its intended endpoint field (awsEndpoint → Streams, dynamoEndpoint → DynamoDB, cloudwatchEndpoint → CloudWatch), read back from the SDK's resolved endpoint rather than inferred.
  • testAwsEndpointAloneDoesNotLeakIntoTheOtherClients — pins the second defect: setting awsEndpoint alone must leave the DynamoDB and CloudWatch clients on their regional default endpoints instead of building one from an empty string.
  • testRegionOnlyUsesRegionalEndpoints — pins the unaffected common case (no endpoint override, real AWS).

Verified they actually catch the bug rather than just passing alongside the fix. Reverting DynamoDBSourceConfig to the master version makes three of the four fail with the original exception:

testBuildersDoNotThrowWhenEndpointAndRegionAreBothSet FAILED
    java.lang.IllegalStateException: Only one of Region or EndpointConfiguration may be set.
testEachBuilderSelectsItsOwnEndpoint FAILED
    java.lang.IllegalStateException: Only one of Region or EndpointConfiguration may be set.
testAwsEndpointAloneDoesNotLeakIntoTheOtherClients FAILED
    java.lang.IllegalStateException: Only one of Region or EndpointConfiguration may be set.

testRegionOnlyUsesRegionalEndpoints passes either way, as it should — it covers the path the bug never touched.

With the fix restored, the full :dynamodb:test suite is green. End-to-end coverage against a real endpoint lands separately in #77, whose LocalStack test cannot run without this fix.

The regression tests added in the previous commit were committed
alongside an accidental revert of DynamoDBSourceConfig, and correctly
failed CI with the original IllegalStateException.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] DynamoDB source cannot use a custom endpoint: region and endpoint are both set, which AWS SDK v1 forbids

2 participants