[dev.icinga.com #11436] Add LogstashWriter for Logstash in addition to GelfWriter #4054

Open
icinga-migration opened this Issue Mar 22, 2016 · 15 comments

Comments

Projects
None yet
3 participants
Member

icinga-migration commented Mar 22, 2016

This issue has been migrated from Redmine: https://dev.icinga.com/issues/11436

Created by twidhalm on 2016-03-22 15:49:55 +00:00

Assignee: mfriedrich
Status: Assigned
Target Version: 2.7.0
Last Update: 2016-11-09 15:11:53 +00:00 (in Redmine)

Backport?: Not yet backported
Include in Changelog: 1

The GELF protocol can use UDP or TCP according to the documentation.

Since some GELF receivers (like Logstash) can only receive UDP and the Icinga 2 GELF writer can only send TCP we have the situation that 2 tools talking the same protocol still can't talk to each other.

Could you add a protocol switcher to the GELF writer object?

Changesets

2016-06-15 13:21:28 +00:00 by kgoller 8ad95bd

Delete unused files

refs #11436

2016-06-15 13:27:26 +00:00 by kgoller ce7a23c

Delete more unused files

refs #11436

2016-07-06 11:51:59 +00:00 by kgoller cd785ae

add UDP support

- icinga can now use UDP to communicate with logstash;
	switch protocol in logstash.conf

refs #11436

2016-07-06 12:07:15 +00:00 by kgoller 50e4c35

delete comment

refs #11436

2016-07-11 19:03:41 +00:00 by kgoller ea5aadf

fix several small issues

-removed some unneeded changes
-fixed several typo errors
-moved TcpSocket::Connect() into class Socket

refs #11436

2016-07-14 07:21:20 +00:00 by kgoller 1db4852

Remove pointer

refs #11436

Relations:

Member

icinga-migration commented Mar 22, 2016

Updated by mfriedrich on 2016-03-22 20:31:33 +00:00

  • Category set to Perfdata
  • Target Version set to Backlog
Member

icinga-migration commented Mar 22, 2016

Updated by mfriedrich on 2016-03-22 20:31:33 +00:00

Part of Kai's internship.

Member

icinga-migration commented Mar 24, 2016

Updated by mfriedrich on 2016-03-24 10:46:13 +00:00

  • Status changed from New to Assigned
  • Assigned to set to kgoller

Hi Kai,

this will be part of your project in the next months. We will discuss it further then.

Kind regards,
Michael

Member

icinga-migration commented Mar 24, 2016

Updated by kowalskimn on 2016-03-24 16:23:30 +00:00

widhalmt wrote:

The GELF protocol can use UDP or TCP according to the documentation.

Since some GELF receivers (like Logstash) can only receive UDP and the Icinga 2 GELF writer can only send TCP we have the situation that 2 tools talking the same protocol still can't talk to each other.

Could you add a protocol switcher to the GELF writer object?

I can already tell you it won't be enough to get logstash working.

There is a plugin for logstash that grabs gelf over tcp, and it's having issues with format of various fields. Not sure if that's for logstash or icinga2 to fix.

Member

icinga-migration commented Mar 29, 2016

Updated by mfriedrich on 2016-03-29 15:08:33 +00:00

Thanks, that will also be part of Kai's task to evaluate whether the existing GELF specification works with Logstash or if a different format specification is required.

Member

icinga-migration commented Mar 29, 2016

Updated by mfriedrich on 2016-03-29 15:09:09 +00:00

  • Subject changed from Add UDP communication to GELF writer to _Evaluate whether the GelfWriter works with Logstash _
Member

icinga-migration commented Apr 18, 2016

Updated by mfriedrich on 2016-04-18 11:46:34 +00:00

Logstash GELF Input

https://github.com/logstash-plugins/logstash-input-gelf/blob/master/lib/logstash/inputs/gelf.rb

# The main use case for this input is to leverage existing GELF
# logging libraries such as the GELF log4j appender. A library used
# by this plugin has a bug which prevents it parsing uncompressed data.
Member

icinga-migration commented Sep 28, 2016

Updated by mfriedrich on 2016-09-28 13:49:04 +00:00

  • Target Version changed from Backlog to 2.7.0
Member

icinga-migration commented Nov 9, 2016

Updated by mfriedrich on 2016-11-09 15:11:53 +00:00

  • Assigned to changed from kgoller to mfriedrich
Member

icinga-migration commented Nov 9, 2016

Updated by mfriedrich on 2016-11-09 15:13:30 +00:00

  • Blocked set to 13111

icinga-migration added this to the 2.7.0 milestone Jan 17, 2017

Owner

dnsmichi commented Jan 24, 2017

Code was not in a good shape, and at least the performance data part is still incomplete.

Tests

Use the icinga2x-elastic box. Logstash 5.x requires the udp plugin to be manually installed.

/usr/share/logstash/bin/logstash-plugin install logstash-input-udp logstash-codec-json
systemctl restart logstash

Configure the input and output.

vim /etc/logstash/conf.d/icinga2.conf

input {
  udp {
    port => 5555
    codec => json
  }
}
output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
  }
}

Then point Icinga 2 to it inside features-enabled/logstash.conf.

object LogstashWriter "logstash" {
  host = "192.168.33.7"
  port = 5555
  //port = 9201
  //socket_type = "udp"
}

Note: The logstash input port is freely assignable, there is no sane default.

Discussion

I'm not entirely convinced if the udp input makes sense since both logstash and elasticsearch provide a HTTP API. Elastic beats are providing output methods for both interfaces, so they probably do not differ that much.

The JSON blob is entirely written to Logstash, so you'll need to use the json codec on your own if your prefer just that.

screen shot 2017-01-24 at 18 15 29

@dnsmichi dnsmichi added a commit that referenced this issue Feb 8, 2017

@dnsmichi Kai Goller + dnsmichi Add LogstashWriter feature
This adds the UdpSocket class.

refs #4054
444dfc3

dnsmichi referenced this issue in Icinga/icingabeat Feb 20, 2017

Closed

Export performance data as separate fields #3

@dnsmichi dnsmichi added a commit that referenced this issue Feb 22, 2017

@dnsmichi Kai Goller + dnsmichi Add LogstashWriter feature
This adds the UdpSocket class.

refs #4054
eed7dbe

@dnsmichi Any update about this?

Owner

dnsmichi commented Mar 17, 2017 edited

Still on my TODO list. Mainly tests and docs are missing.

@dnsmichi dnsmichi added a commit that referenced this issue Mar 20, 2017

@dnsmichi Kai Goller + dnsmichi Add LogstashWriter feature
This adds the UdpSocket class.

refs #4054
f5a971f

@dnsmichi dnsmichi added a commit that referenced this issue Mar 20, 2017

@dnsmichi dnsmichi Add documentation for the LogstashWriter feature
This includes Logstash and Icinga 2 configuration.

refs #4054
71d5ee7

dnsmichi closed this in #5085 Mar 20, 2017

dnsmichi changed the title from [dev.icinga.com #11436] Evaluate whether the GelfWriter works with Logstash to [dev.icinga.com #11436] Add LogstashWriter for Logstash in addition to GelfWriter Mar 20, 2017

Owner

dnsmichi commented May 23, 2017

I'm postponing this feature until there is a general decision about an ElasticWriter feature. There's a possible sponsor for that.

dnsmichi reopened this May 23, 2017

@dnsmichi dnsmichi modified the milestone: 2.8.0, 2.7.0 May 23, 2017

@dnsmichi dnsmichi added a commit that referenced this issue May 23, 2017

@dnsmichi dnsmichi Revert "Add documentation for the LogstashWriter feature"
This reverts commit 71d5ee7.

refs #4054
22752e6

@dnsmichi dnsmichi added a commit that referenced this issue May 23, 2017

@dnsmichi dnsmichi Revert "Review LogstashWriter feature implementation"
This reverts commit bd5ff81.

refs #4054
4c76601

@dnsmichi dnsmichi added a commit that referenced this issue May 23, 2017

@dnsmichi dnsmichi Revert "Add LogstashWriter feature"
This reverts commit f5a971f.

refs #4054
52d986d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment