file beat Alpha (5.0)

[osmancis]

I need to push my logs to kafka and for that I might need filebeat 5.0, though it is in alpha. But I am not sure about the repo and other settings and I am relying heavy on readymade cookbooks! Is there any cook available? Help is greatly appreciated.

[bradq]

@osmancis you can use the default['filebeat']['yum'] or default['filebeat']['apt'] settings documented in the readme to set repositories. 5.0-alpha repository and keys are documented here. I use "https://packages.elastic.co/kibana/5.0.0-alpha/debian stable main" for installing 5.0.

As for use, v5 largely extends syntax. This cookbook doesn't yet support the extended features of 5.0 in the LWRPs (e.g. no JSON input for prospectors), but should work fine if your only use case for 5 is kafka output. If you're not in a place to add a pull request enabling them, you can always simply use the default['filebeat']['conf_dir'] attribute and add your own output YAML as a raw template rather than a LWRP.

The filebeat output configuration is not restrictive (nor is any other part of filebeat.yml); it's just a straight to-YAML rendering of the node['filebeat']['config'] object, and you're free to add any arbitrary elements via attributes.

I doubt you'll have any issues using the cookbook with 5 here. I'll try to send in a PR to support some of the newer parameters if I have time this week.

[vkhatri]

@osmancis Currently beats 5.0.0-alpha* is not available to install via repository, at least not in yum.

# yum search filebeat --show-duplicate
Loaded plugins: priorities, update-motd, upgrade-helper
193 packages excluded due to repository priority protections
=========================================================================================================== N/S matched: filebeat ============================================================================================================
filebeat-1.0.0~beta4-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.0.0~rc1-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.0.0~rc2-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.0.0-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.0.1-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.1.0-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.1.1-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.1.2-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.2.0-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.2.1-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.2.2-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.2.3-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.
filebeat-1.2.3-1.x86_64 : Sends log files to Logstash or directly to Elasticsearch.

Perhaps adding an attribute to skip package install via this cookbook could be useful when using wrapper cookbook to install beats 5.x.

I will also give it a try to add support for 5.x configuration parameters.

[bradq]

@vkhatri Actually it is available via yum; you just need to use the docs that I linked to and set the repository parameter within the cookbook. They're using an alternate repo for alpha releases.

[kibana-5.0.0-alpha] 
name=Kibana repository for 5.0.0-alpha packages 
baseurl=https://packages.elastic.co/kibana/5.0.0-alpha/centos 
gpgcheck=1 
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch 
enabled=1 

[vkhatri]

@bradq For some reason, it did not work for me. Does it still work for you?

[chopmo]

This is not working for me either, unfortunately. I can't seem to find the right repo.

Added this as /etc/yum.repos.d/beats.repo:

[beats]
name=Elastic Beats Repository
baseurl=https://packages.elastic.co/kibana/5.0.0-alpha/centos
enabled=1
gpgcheck=1
gpgkey=https://packages.elasticsearch.org/GPG-KEY-elasticsearch
metadata_expire=3h
sslverify=true

Then I did a yum clean all and a yum makecache.

But it seems that only Kibana is avaible in the repo:

$ yum --disablerepo="*" --enablerepo="beats" list available
Loaded plugins: priorities, update-motd, upgrade-helper, versionlock
Available Packages
kibana.i386                                                          5.0.0_alpha3-1                                                         beats
kibana.i686                                                          5.0.0_alpha5-1                                                         beats
kibana.x86_64                                                        5.0.0_alpha5-1                                                         beats

Since I'm using JSON logging, I need a 5.x version of Filebeat, but I can't think of any way to install it with this recipe.

[bradq]

@chopmo @vkhatri Sorry for the delayed response...I'm in a new job that doesn't use EL or Filebeat. I'll validate these in a few hours and get back to you, as it seems that their documentation might have changed?

I definitely used this without issue, though...there may have been a repo change as they get ready to shift, or I just recorded it here in a dumb way.

[vkhatri]

@bradq Looks like it was indeed removed for alpha versions but planning to add it back for 5.x beta.

@chopmo As beats are supposed to be available to install via official repository, i prefer to keep it that way. Adding capability to download and install from a package url is certainly possible, but it will increase the complexity.

However, i am open to suggestions.

[chopmo]

@vkhatri I think it definitely makes sense to stick to the repos.

I think my solution for now will have to be writing a custom recipe to upload a binary package and install it. Then I can go back to using this recipe once 5.x is available from the repos.

Thanks for the responses, both!

[vkhatri]

Beats docs updated. Will update cookbook shortly.

https://www.elastic.co/guide/en/elasticsearch/reference/5.0/rpm.html
https://www.elastic.co/guide/en/elasticsearch/reference/5.0/deb.html

[vkhatri]

working on adding 5.x support

[vkhatri]

Added v5.x support iv v0.4.3.