Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

configurable kafka partitioner #2284

Merged
merged 6 commits into from
Aug 24, 2016
Merged

Conversation

urso
Copy link

@urso urso commented Aug 16, 2016

  • Store kafka message wrapper in outputs.Data.Values for re-use on retry.
  • Add support for configuring the partitioner strategy
  • configurable support ignoring unavailable partitions
  • partitioning is mostly stable in error case (try to schedule events to same partition if returned to libbeat on error)

@urso urso added enhancement in progress Pull request is currently in progress. libbeat labels Aug 16, 2016
@urso
Copy link
Author

urso commented Aug 17, 2016

Requires #2295 to fix test build

@urso urso force-pushed the enh/kafka-partitioner branch 2 times, most recently from bb9c463 to 8fa7888 Compare August 18, 2016 11:46
@urso urso changed the title [WIP] configurable kafka partitioner configurable kafka partitioner Aug 18, 2016
@urso urso added review and removed in progress Pull request is currently in progress. labels Aug 18, 2016
@urso
Copy link
Author

urso commented Aug 18, 2016

Will update docs in another PR, as a number of other settings have been added/changed for kafka output.

var testOptions = outputs.Options{}
func TestKafkaPublish(t *testing.T) {
single := modetest.SingleEvent
// multi := modetest.MultiEvent
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the idea with these 2 commented out lines?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they can be removed as not being used. events of type []modetest.EventInfo hold a number of events + setting to bulk or not bulk publish those events. It's used to drive sending events through output plugin.

In test randMulti is used.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

@ruflin
Copy link
Member

ruflin commented Aug 22, 2016

LGTM. What about the updates to the config file? Will this also follow in a separate PR?

@urso
Copy link
Author

urso commented Aug 22, 2016

config files can be done in this PR.

@urso
Copy link
Author

urso commented Aug 22, 2016

@ruflin added configs

@@ -35,6 +35,10 @@ packetbeat.protocols.amqp:
# the AMQP protocol by commenting out the list of ports.
ports: [5672]

packetbeat.protocols.cassandra:
#Cassandra port for traffic monitoring.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not part of this PR, but needs a space in front of Cassandra.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, this should be fixed by another PR. will have to rebase.

@tsg
Copy link
Contributor

tsg commented Aug 22, 2016

LGTM, left a minor comment/question.

@andrewkroh
Copy link
Member

LGTM

urso added 4 commits August 23, 2016 13:50
- add config settings to configure partition strategy
- add config settings to configure partitioner selecting only reachable
  partitions
- Store kafka message in data.values so original message including partition
  selection is still available on retry
- make kafka message key configurable
- update changelog
- replace kafka integration tests with table driven integration tests
- add integrations tests using batch publishing
- add integration tests using different partitioner configurations
@urso
Copy link
Author

urso commented Aug 23, 2016

renamed switch to group_events.

@tsg tsg merged commit a025e36 into elastic:master Aug 24, 2016
@urso urso deleted the enh/kafka-partitioner branch February 19, 2019 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants