LogstashUdpSocketAppender with prefix not working

[nikosmoum]

jackson: 2.12.3
logback-classic: 1.2.9
logstash-logback-encoder:7.0.1
java: 11
On Centos7

After upgrading to 7.0.1 from 5.3, I tried to use the new UDP appender for syslog based on the example here, but it is not working (even tried that exact example but same error is shown).

logback.xml:

    <appender name="SyslogAppender" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
        <host>127.0.0.1</host>
        <layout>
            <prefix class="ch.qos.logback.classic.PatternLayout">
                <pattern>@cee:</pattern>
            </prefix>
        </layout>
    </appender>
...
    <root level="WARN">
        <appender-ref ref="SyslogAppender" />
    </root>

the log output:

...
INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [net.logstash.logback.appender.LogstashUdpSocketAppender]
INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SyslogAppender]
INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.PatternLayout] for [layout] property
ERROR in ch.qos.logback.core.joran.spi.Interpreter@73:66 - no applicable action for [prefix], current ElementPath  is [[configuration][appender][layout][prefix]]
INFO in net.logstash.logback.appender.LogstashUdpSocketAppender[SyslogAppender] - Defaulting maxMessageSize to [65000]
...
INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SyslogAppender] to Logger[ROOT]

I also tried the following, which produced no log output (so it seems as if a correct configuration):

    <appender name="SyslogAppender" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
        <host>127.0.0.1</host>
        <layout>
            <pattern>@cee:</pattern>
        </layout>
    </appender>
...
    <root level="WARN">
        <appender-ref ref="SyslogAppender" />
    </root>

But it does not actually work, since all I see in /var/log/messages is:

Dec 21 10:06:22 myhostname.com @cee:
Dec 21 10:06:22 myhostname.com @cee:
Dec 21 10:06:22 myhostname.com @cee:
Dec 21 10:06:22 myhostname.com @cee:
...

Before upgrading, when using logstash-logback-encoder 7.0.1 and logback-classic 1.2.3, this worked fine:

    <appender name="SyslogAppender" class="net.logstash.logback.appender.LogstashSocketAppender">
        <host>127.0.0.1</host>
        <port>514</port>
        <prefix class="ch.qos.logback.classic.PatternLayout">
            <pattern>@cee:</pattern>
        </prefix>
    </appender>
...
    <root level="WARN">
        <appender-ref ref="SyslogAppender" />
    </root>

with output in /var/log/messages like:

Dec 21 10:21:32 myhostname.com @cee: {"@timestamp":"2021-12-21T10:21:32.111+00:00","@version":"1","message":"Request: verb=GET, uri=/myapp/status","logger_name":"org.myapp...","thread_name":"http-bio-8443-exec-10","level":"INFO","level_value":20000,"requestType":"req","requestUuid":"bdeb101b-def9-4318-b901-83046e661042"}

Given that the exact example in the docs does not work, it looks like a bug, but I was not sure if it is.

[brenuart]

Hi Nikos,

Thanks for the feedback. You are right, the documentation is very poor on this subject if not confusing.
As you noticed, the LogstashSocketAppender has been removed in 6.0 and the documentation isn't clear since then. In addition, the behaviour of its replacement components is slightly different which makes the transition even more tricky - sorry about that.

To start, here is a configuration that should (hopefully) produce the same results as before:

    <conversionRule conversionWord="syslogStart" converterClass="ch.qos.logback.classic.pattern.SyslogStartConverter"/>

    <appender name="SyslogAppender" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
        <host>127.0.0.1</host>
        <port>514</port>
        <layout class="net.logstash.logback.layout.LogstashLayout">
            <prefix class="ch.qos.logback.classic.PatternLayout">
                <pattern>%syslogStart{USER}@cee: </pattern>
            </prefix>
        </layout>
    </appender>

    <root>
        <appender-ref ref="SyslogAppender" />
    </root>

LogstashUdpSocketAppender accepts a Layout rather than an Encoder as most of the other appenders. No layout is provided by default so you need to declare one. In this example I configured the appender with the LogstashLayout which serialises the log event in JSON format with standard Logstash conventions for the field names (this is what was used by default before 6.0). You can find more information in the section dedicated to the UDP Appender.

Since you are sending to a syslog server you also need a prefix with the syslog special fields in front of the JSON payload. This is done with the <prefix> configuration property. The <prefix> and <suffix> configuration parameters both accepts any kind of Logback Layout - the actual implementation class must therefore be specified. Here we are using a standard Logback PatternLayout with the special %syslogStart{} conversion word. This conversion word outputs the standard syslog prefix (facility, date/time, host). It is not defined by default and must therefore be explicitly registered with the logback configuration with a <conversionRule> directive (see Logback documentation for more information).

@philsttr probably knows more than me on this particular topic - no doubt he will correct me if I'm wrong. Any way, I hope my explanations are clear enough to help you with your upgrade to LLE 7.0.1. In any case, don't hesitate to post additional comments - we'll do our best to help you.

We are well aware the configuration of the UDP appender can be confusing and not yet fully aligned with the other components of the LLE eco-system. Feedbacks like yours help us to identify the areas needing our attention.

[nikosmoum]

This configuration works well; thank you for the detailed and quick reply!

[mmuruev]

So still not clear how to limit message size with UDP. All examples with encoder.. why UPD cant have it? Why it's even so tricky? Max message length is one of the commonly used parameter.