add GELF support, at least for output

[rgerhards]

As mentioned on Twitter, a couple of folks have interest in adding GELF support to rsyslog. This tracker is intended to help scope what actually needs to be done.

[rgerhards]

A first review shows that "what needs to be done" may actually be not nothing. If I understand the GELF spec [ https://www.graylog.org/resources/gelf-2/ ] correctly, rsyslog already has everything that's needed to write and transport GELF, it "just" needs to be configured correctly:

• we can generate JSON
• we can send a plain TCP stream, and we can also zip-compress it

So it looks like the question boils more down to "what exactly is needed?". Any feedback on that would be appreciated.

[rgerhards]

Here is an example of how this is done with Elasticsearch: http://www.rsyslog.com/output-to-elasticsearch-in-logstash-format-kibana-friendly/ This is probably useful for the JSON part. Note that there are other ways to write the JSON, but to suggest anything specific, I would need to know what should be mapped to what...

[henrikjohansen]

From the GELF specs page (https://www.graylog.org/resources/gelf-2/) the following fields should be set by the client :

version string (UTF-8) :
GELF spec version – “1.1”; MUST be set by client library

host string (UTF-8) :
name of the host, source or application that sent this message; MUST be set by client library.)

short_message string (UTF-8) :
a short descriptive message; MUST be set by client library.

full_message string (UTF-8) :
a long message that can i.e. contain a backtrace; optional.

timestamp number :
Seconds since UNIX epoch with optional decimal places for milliseconds; SHOULD be set by client library. Will be set to NOW by server if absent.

level number :
the level equal to the standard syslog levels; optional, default is 1 (ALERT).

facility string (UTF-8) :
optional, deprecated. Send as additional field instead.

[rgerhards]

I read the spec, but some questions still remain open.

What should go in "host" - we can add the hostname, we can add the tag, or we can add hostname and tag concatenated .. or anything else.

What should go in short_messages vs. full_message? We only have a single msg object in rsyslog, and that's the message as it is. We could put that into full_message. And maybe shorten short_message to e.g. the first 128 chars?

Is it correct that the facility should now be discarded?

We could create a template e.g. to do as follows:

"1.1" --> version
hostname --> host
msg, first 128 chars --> short_message
msg --> full_message
timegenerated --> timestamp
severity --> level

[janmejay]

Is 128 char truncated message really useful? How is it used? And considering it's easy to build denormalization, isn't t best done by consumer?

Is it useful to potentially have tags pushed in short message field?

[bernd]

I will try to answer some of @rgerhards questions.

• host should contain the ip address or the hostname of the host that sends the message. In case of forwarding syslog messages from other hosts, the original host.
• Just put the message in short_message and ignore full_message. The latter was meant for things like stack traces etc. The Graylog system shows the short_message as "message" by default.
• The facility field is deprecated to slim down the pre defined fields. The syslog facility could be sent as _facility or _syslog_facility instead.

Hope that helps. 😃 Let me know if you have any further questions.

[jpmens]

While the spec says GELF spec version – “1.1”; MUST be set by client library I've not seen Graylog deny a message which had no version string in it. @bernd ?

[bernd]

@jpmens True, we currently do not check this. 😕

[rgerhards]

Well, the version shouldn't be any problem at all, it's just constant text. As it looks, all we need to do in rsyslog is to configure a proper template - and that's it. So it looks like it's no programming required, just doing a small config snippet. Then, of course, it would be helpful to have someone who actually tries it out.

[bernd]

What about structured data? It would be nice if that can be broken up into GELF additional fields. Is that also possible with templates?

[rgerhards]

@bernd I need to check for structured data, maybe we need some extra properties for it. But I don't think so mmpstrucdata should take care of that.

Is there anything else that a "GELF output" should handle? Remember that I am trying to spec up things...

[bernd]

Some more notes:

• The Graylog GELF TCP input currently uses a null-byte as message delimiter. This means using gzip compression does not work. 😕
• The GELF spec contains a section about "Chunked GELF". GELF chunking can be used with UDP to send huge messages. (https://www.graylog.org/resources/gelf-2/)

[friedl]

As far as I understood, the Graylog only needs the null-byte as message delimiter when using TCP, right? So it should work with default UDP!? This null-byte delimiter is an issue, that Rainer needs to look at.

Anyway, I have prepared a template to reflect the GELF format as described above and I left out the full_message since it is not really needed. This gave me messages like this:

{"version":"1.1","host":"localhost","short_message":" [origin software="rsyslogd" swVersion="8.9.0" x-pid="62054" x-info="http://www.rsyslog.com"] start","timestamp":"1428508461","level":"6"}

Can someone please confirm if the template works for them with UDP transport?

Here is the template:

template(name="gelf" type="list") {
        constant(value="{\"version\":\"1.1\",")
        constant(value="\"host\":\"")
        property(name="hostname")
        constant(value="\",\"short_message\":\"")
        property(name="msg" format="json")
        constant(value="\",\"timestamp\":\"")
        property(name="timegenerated" dateformat="unixtimestamp")
        constant(value="\",\"level\":\"")
        property(name="syslogseverity")
        constant(value="\"}")
}

[bernd]

@friedl Yes, the null-byte delimiter is only for TCP connections. Default UDP should work, yes.

I will test the template tomorrow.

[lennartkoopmann]

👍 for breaking up structured syslog data into GELF fields

[rgerhards]

@lennartkoopmann I'll look at breaking up structured data asap. However, it's still very seldomly used. I guess the best approach is to say "submit normalized data to GELF properties". That's a much broader thing and plays well with rsyslog's normalization. But let me see first if the simple things work.

[bernd]

The template that @friedl posted works for me on 7.4.4.

[rgerhards]

That sounds great. So let me ask about additional fields, e.g. for structured data. I just re-read the GELF spec page. It sounds to me that you can add additional fields with prefixing them by underscore (e.g. "_some_field"), but it sounds like no hierarchy is supported. Is that correct? I ask because we have a 2-layer hierarchy in RFC5424 structured data, and potentially many more layers when receiving JSON data. So I need to find out how to put this into GELF.

[bernd]

Yes, the current GELF version does not support hierarchies. For parsing RFC5424 in Graylog we are currently prefixing the keys with the SD-ID. (if configured) This isn't really nice but the simplest thing that came to our mind at that time.

[rgerhards]

Thx. I need to think how to shuffle the definitions inside the templates. It's probably a bit tricky and maybe requires some code to be written.

[colttt]

Hello,

any news about the null-byte as message delimiter when using TCP?

[ghost]

Still no null-byte for TCP?
Effectively renders rsyslog as useless for centralized logging over Internet!

[rgerhards]

@colttt @opensysnotes msg terminator for TCP is totally off-topic for this issue, thus no response here

[zombah]

@friedl template worked fine for a long time, but latest versions of graylog now always says warning on timestamp, like this:

WARN : org.graylog2.inputs.codecs.GelfCodec - GELF message <993703c0-f7de-11e8-a47f-3a3172c4aecf> (received from <127.0.0.1:60732>) has invalid "timestamp": 1543939606  (type: STRING)

it is only warning and message processed fine itself.

[guard43ru]

I made this template, it's more neat. Some fields optional.

# gelf template
template(name="gelf" type="list" option.jsonf="on") {
  constant(outname="version" value="1.1" format="jsonf")
  property(outname="host" name="hostname" format="jsonf")
  property(outname="timestamp" name="timereported" dateFormat="unixtimestamp" datatype="number" format="jsonf")
  property(outname="_application_name" name="app-name" format="jsonf")
  property(outname="message" name="msg" format="jsonf")
  property(outname="severity" name="syslogseverity-text" caseConversion="upper" format="jsonf")
  property(outname="facility" name="syslogfacility-text" format="jsonf")
}