Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scala Stream Collector: configure the ability to use IP address as partition key #2331

Closed
fblundun opened this issue Jan 13, 2016 · 3 comments
Assignees

Comments

@fblundun
Copy link
Contributor

The SSC doesn't guarantee that records will be sent to Kinesis in the same order they arrive, which means sending events from the same IP to the same shard doesn't do anything.

So we should either stop using the IP address as a partition key or start guaranteeing preservation of record order.

@fblundun
Copy link
Contributor Author

In the end we decided to make this a configurable setting, defaulting to random partitioning.

@fblundun fblundun changed the title Decide whether to keep using IP address as partition key Scala Stream Collector: configure the ability to use IP address as partition key Jan 13, 2016
@fblundun fblundun self-assigned this Jan 13, 2016
@fblundun fblundun added this to the Release 78 [KIN] Bird TBC milestone Jan 13, 2016
@alexanderdean
Copy link
Member

The default will be random sharding - but if you need events sharded by ipAddress (so that users are approximately tied to one shard), you can still switch that on.

@fblundun
Copy link
Contributor Author

I'll add an optional boolean configuration setting called collector.sink.kinesis.useIpAddressAsPartitionKey defaulting to false.

fblundun added a commit that referenced this issue Feb 15, 2016
fblundun added a commit that referenced this issue Feb 15, 2016
fblundun added a commit that referenced this issue Feb 15, 2016
fblundun added a commit that referenced this issue Feb 15, 2016
fblundun added a commit that referenced this issue Feb 15, 2016
fblundun added a commit that referenced this issue Feb 15, 2016
fblundun added a commit that referenced this issue Feb 15, 2016
fblundun added a commit that referenced this issue Feb 15, 2016
fblundun added a commit that referenced this issue Feb 15, 2016
fblundun added a commit that referenced this issue Feb 29, 2016
fblundun added a commit that referenced this issue Feb 29, 2016
fblundun added a commit that referenced this issue Feb 29, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants