Problem
Amazon Kinesis has no resource-based (resource) policy, so a Hudi ingestion job cannot read a Kinesis stream that lives in a different AWS account than the application — the default credential chain (the job's own pod/node/task role) gets AccessDenied. The only way to read cross-account is IAM role assumption (sts:AssumeRole).
Today JsonKinesisSource / KinesisOffsetGen build the KinesisClient with region + the default credential chain (or static access/secret keys for LocalStack-style endpoints), with no way to assume a role in the stream's account.
Proposed enhancement
Add an optional config hoodie.streamer.source.kinesis.role.arn. When set, the Kinesis client is built with an auto-refreshing StsAssumeRoleCredentialsProvider (AWS SDK v2) for that role; the base STS client uses the default credential chain (which must be granted sts:AssumeRole on the ARN). When empty/absent, behavior is unchanged (same-account read via the default chain). Static access/secret keys continue to take precedence. Naming aligns with the existing HoodieAWSConfig.hoodie.aws.role.arn.
The assume-role provider is cached per region|roleArn so a long-lived streaming executor reuses one StsClient per distinct role rather than leaking one per mapPartitions task.
Environment
- Component: hudi-utilities (DeltaStreamer/Streamer Kinesis source)
A PR implementing this is attached.
Problem
Amazon Kinesis has no resource-based (resource) policy, so a Hudi ingestion job cannot read a Kinesis stream that lives in a different AWS account than the application — the default credential chain (the job's own pod/node/task role) gets
AccessDenied. The only way to read cross-account is IAM role assumption (sts:AssumeRole).Today
JsonKinesisSource/KinesisOffsetGenbuild theKinesisClientwith region + the default credential chain (or static access/secret keys for LocalStack-style endpoints), with no way to assume a role in the stream's account.Proposed enhancement
Add an optional config
hoodie.streamer.source.kinesis.role.arn. When set, the Kinesis client is built with an auto-refreshingStsAssumeRoleCredentialsProvider(AWS SDK v2) for that role; the base STS client uses the default credential chain (which must be grantedsts:AssumeRoleon the ARN). When empty/absent, behavior is unchanged (same-account read via the default chain). Static access/secret keys continue to take precedence. Naming aligns with the existingHoodieAWSConfig.hoodie.aws.role.arn.The assume-role provider is cached per
region|roleArnso a long-lived streaming executor reuses oneStsClientper distinct role rather than leaking one permapPartitionstask.Environment
A PR implementing this is attached.