Sync behaviour of GELF plugin and Graylog (re: has empty mandatory "short_message")

[hangstl]

My setup: Logstash -> GELF -> Graylog 2.4.6

Some of my logfiles contain lines with only whitspace characters (TAB + end of line). This leads to lots of "has empty mandatory "short_message" field" Exceptions in Graylog.

imho this is a bug, because there is a message (the TAB character). GelfCodec.java has this line here:

if (StringUtils.isBlank(shortMessageNode.asText()) && StringUtils.isBlank(messageNode.asText())) {
                throw new IllegalArgumentException(prefix + "has empty mandatory \"short_message\" field.");
}

Would it make sense to change StringUtils.isBlank() to StringUtils.isEmpty()

If not, I would appreciate to sync the behaviour of the GELF Plugin and the GelfCodec in Graylog. I mean, why send messages to Graylog if it rejects it anyway?

[jalogisch]

REF https://community.graylog.org/t/errors-empty-mandatory-short-message-field/7060

[eduault]

I have the same problem with my Graylog 2.4.6 server on Ubuntu Linux (created in AWS from The AMI provided by Graylog).

The problematic messages are sent by Graylog itself (or a plugin).

root@graylog:/data/log/graylog/server# netstat | grep 36028
tcp        0      0 graylog:36028           graylog:12202           ESTABLISHED
tcp6       0     68 graylog:12202           graylog:36028           ESTABLISHED

Here are the logs :

2018-10-15_09:43:24.46613 java.lang.IllegalArgumentException: GELF message <a304ef41-d054-11e8-bee0-0600ea37eac6> (received from <graylog_ip:36028>) is missing mandatory "short_message" or "message" field.
2018-10-15_09:43:24.46615 	at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:262) ~[graylog.jar:?]
2018-10-15_09:43:24.46616 	at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:134) ~[graylog.jar:?]
2018-10-15_09:43:24.46617 	at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
2018-10-15_09:43:24.46617 	at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
2018-10-15_09:43:24.46617 	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
2018-10-15_09:43:24.46618 	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
2018-10-15_09:43:24.46618 	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
2018-10-15_09:43:24.46618 	at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
2018-10-15_09:43:24.46618 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
2018-10-15_09:43:24.46619 WARN  [GelfCodec] GELF message <a30627c1-d054-11e8-bee0-0600ea37eac6> (received from <graylog_ip:36028>) is missing mandatory "host" field.
2018-10-15_09:43:24.46619 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=a30627c1-d054-11e8-bee0-0600ea37eac6, journalOffset=450585286, codec=gelf, payloadSize=239, timestamp=2018-10-15T08:30:56.572Z, remoteAddress=/graylog_ip:36028} on input <5b8cfa3d61d4970cb039a638>.
2018-10-15_09:43:24.46620 ERROR [DecodingProcessor] Error processing message RawMessage{id=a30627c1-d054-11e8-bee0-0600ea37eac6, journalOffset=450585286, codec=gelf, payloadSize=239, timestamp=2018-10-15T08:30:56.572Z, remoteAddress=/graylog_ip:36028}

Those logs sent by "runit-service" facility may be the same :

facility
    runit-service
from_gelf
    true
level
    6
message
    ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=61ee7a20-d050-11e8-bee0-0600ea37eac6, journalOffset=449355463, codec=gelf, payloadSize=308, timestamp=2018-10-15T08:00:29.378Z, remoteAddress=/172.20.128.41:36028} on input <5b8cfa3d61d4970cb039a638>.
source
    graylog-server
timestamp
2018-10-15T09:11:11.089Z

[eduault]

In GELFMessageTest.java, GELF_JSON has no "short_message".
In the test methods of this class, some new GELFMessage(...) are created with json data having no "short_message".

Idem in GELFMessageChunkTest.java : GELF_JSON has no "short_message" nor "version".

And Graylog GELF inputs apparently accepts GELF JSON having no "version", which is said to be mandatory !

[piellick]

Hi everyone,

there is a workaround to this before waiting the bug correction ?

[n-rodriguez]

Hi there! Any news?

[alitoufighi]

Any updates?

[DilaraSSS]

I have the same problem with my Graylog 4.0.11+e4e88a4, codename Noir. Any update?

Here are the logs :

2022-01-14 09:48:04,082 ERROR [DecodingProcessor] - Unable to decode raw message RawMessage{id=119053b1-751f-11ec-96f4-7af235b1e3ab, journalOffset=198692353, codec=gelf, payloadSize=514, timestamp=2022-01-14T09:48:04.075Z, remoteAddress=/10.40.0.0:19396} on input <607e93166eddf7582244c00f>. - {} 2022-01-14 09:48:04,082 ERROR [DecodingProcessor] - Error processing message RawMessage{id=119053b1-751f-11ec-96f4-7af235b1e3ab, journalOffset=198692353, codec=gelf, payloadSize=514, timestamp=2022-01-14T09:48:04.075Z, remoteAddress=/10.40.0.0:19396} - {} java.lang.IllegalArgumentException: GELF message <119053b1-751f-11ec-96f4-7af235b1e3ab> (received from <10.40.0.0:19396>) has empty mandatory "short_message" field. at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:258) ~[graylog.jar:?] at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:149) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:90) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?] at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]

[storm1kk]

I have a lot of such messages too. I read somewhere that it's kinda a white noise. But can we do something with this? It bothers me when I see [ERROR]...