Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: kafka output #942

Merged
merged 3 commits into from Feb 10, 2016

Conversation

Projects
None yet
7 participants
@urso
Copy link
Collaborator

commented Feb 7, 2016

Kafka output basically wrapping github.com/Shopify/sarama library in libbeat + make it configurable.

Depends on: #927 and #940

@urso urso force-pushed the urso:feature/kafka-output branch from f046234 to 0b3d584 Feb 8, 2016

@monicasarbu monicasarbu referenced this pull request Feb 8, 2016

Closed

Add more outputs than Elasticsearch and Logstash #943

3 of 3 tasks complete

@urso urso force-pushed the urso:feature/kafka-output branch 3 times, most recently from a905db4 to 1b938be Feb 9, 2016

@urso urso added review and removed in progress labels Feb 9, 2016

}
debugf("hosts: %v", hosts)

useType := kafkaDefaultTopicsFromType

This comment has been minimized.

Copy link
@tsg

tsg Feb 9, 2016

Collaborator

I think this configuration option is not documented.

This comment has been minimized.

Copy link
@urso

urso Feb 9, 2016

Author Collaborator

oh yes, forgot this one.


Configuration options for TLS parameters like the certificate authority to use
for HTTPS-based connections. If the `tls` section is missing, the host CAs are used for HTTPS connections to
Elasticsearch.

This comment has been minimized.

Copy link
@tsg

tsg Feb 9, 2016

Collaborator

s/Elasticsearch/Kafka? Do all the TLS parameters work for Kafka?

This comment has been minimized.

Copy link
@urso

urso Feb 9, 2016

Author Collaborator

ups and yes

@tsg

This comment has been minimized.

Copy link
Collaborator

commented Feb 9, 2016

LGTM. I suggest some integration tests for a few of the common configuration (TLS/no-TLS, ACK/no-ACK), but those can come in a different PR.

urso added some commits Feb 7, 2016

@urso urso force-pushed the urso:feature/kafka-output branch from 1ab0527 to 2f64b7e Feb 10, 2016

@urso

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 10, 2016

History cleanup finished

tsg added a commit that referenced this pull request Feb 10, 2016

@tsg tsg merged commit 1a6c611 into elastic:master Feb 10, 2016

4 checks passed

CLA Commit author is a member of Elasticsearch
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
default Merged build finished.
Details

@urso urso deleted the urso:feature/kafka-output branch Feb 11, 2016

@biswajit86

This comment has been minimized.

Copy link

commented Feb 18, 2016

Is there a need for implementing other PR's or can we start using kafka output in beats framework.

@anefassa

This comment has been minimized.

Copy link

commented Mar 23, 2016

This is great news on providing a Kafka output for beats. When is v5.0.0-GA targeted to be released?

In the meantime I have built from master to start testing it out. Any tips on how to configure the Kafka output?

@andrewkroh

This comment has been minimized.

Copy link
Member

commented Mar 23, 2016

@anefassa

This comment has been minimized.

Copy link

commented Mar 24, 2016

Thanks for sharing the docs. Kafka output with filebeat is working for me.

@joschi joschi referenced this pull request May 9, 2016

Open

Add support for Kafka #2

@mostolog

This comment has been minimized.

Copy link

commented Jul 12, 2016

Could anyone @elastic confirms if beat -> kafka is going to be available for version 5?

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Jul 12, 2016

Yes, Kafka will be available in the 5.0 release. You can already tests it with the 5.0.0-alpha releases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.