Log4j 2.x plugin for customizable and Logstash-friendly JSON layout.
Clone or download
vy #17 Parse template tree once to create necessary resolvers.
Provides more stable 29k ops/sec performance in the benchmark. At this
stage, Node-to-String generation is the bottleneck. See the following
JMH snippet:

    50.0%  50.4% java.net.SocketInputStream.socketRead0
    14.8%  14.9% java.lang.Class.getEnclosingMethod0
    12.3%  12.4% sun.nio.cs.UTF_8.updatePositions
     5.1%   5.2% java.lang.StringCoding$StringDecoder.decode
     2.8%   2.8% java.security.AccessController.doPrivileged
     2.0%   2.0% com.fasterxml.jackson.databind.node.ObjectNode.serialize
     1.7%   1.8% java.lang.String.valueOf
     1.1%   1.1% java.lang.Throwable.getLocalizedMessage
     1.0%   1.0% com.fasterxml.jackson.core.json.JsonWriteContext.createChildObjectContext
     0.8%   0.8% com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers$4.resolve
     7.5%   7.5% <other>
Latest commit 591ed46 Oct 12, 2018

README.md

Build Status Maven Central License

LogstashLayout plugin provides a Log4j 2.x layout with customizable and Logstash-friendly JSON formatting.

By default, LogstashLayout ships the official JSONEventLayoutV1 stated by log4j-jsonevent-layout Log4j 1.x plugin. Compared to JSONLayout included in Log4j 2.x and log4j-jsonevent-layout, LogstashLayout provides the following additional features:

  • Additional fields can be added. (See template and templateUri parameters.)
  • JSON schema structure can be customized. (See template and templateUri parameters.)
  • Timestamp formatting can be customized. (See dateTimeFormatPattern and timeZoneId parameters.)

Usage

Add the log4j2-logstash-layout dependency to your POM file

<dependency>
    <groupId>com.vlkan.log4j2</groupId>
    <artifactId>log4j2-logstash-layout</artifactId>
    <version>${log4j2-logstash-layout.version}</version>
</dependency>

together with a valid log4j-core dependency:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>${log4j2.version}</version>
</dependency>

Below you can find a sample log4j2.xml snippet employing LogstashLayout.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
    <Appenders>
        <Console name="CONSOLE" target="SYSTEM_OUT">
            <LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"
                            templateUri="classpath:LogstashJsonEventLayoutV1.json"
                            prettyPrintEnabled="true"
                            stackTraceEnabled="true"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="CONSOLE"/>
        </Root>
    </Loggers>
</Configuration>

This generates an output as follows:

{
  "exception": {
    "exception_class": "java.lang.RuntimeException",
    "exception_message": "test",
    "stacktrace": "java.lang.RuntimeException: test\n\tat com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo.main(LogstashLayoutDemo.java:11)\n"
  },
  "line_number": 12,
  "class": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo",
  "@version": 1,
  "source_host": "varlik",
  "message": "Hello, error!",
  "thread_name": "main",
  "@timestamp": "2017-05-25T19:56:23.370+02:00",
  "level": "ERROR",
  "file": "LogstashLayoutDemo.java",
  "method": "main",
  "logger_name": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo"
}

LogstashLayout is configured with the following parameters:

Parameter Name Type Description
prettyPrintEnabled boolean enables pretty-printer (defaults to false)
locationInfoEnabled boolean includes the filename and line number in the output (defaults to false)
stackTraceEnabled boolean includes stack traces (defaults to false)
emptyPropertyExclusionEnabled boolean exclude empty and null properties (defaults to true)
dateTimeFormatPattern String timestamp formatter pattern (defaults to yyyy-MM-dd'T'HH:mm:ss.SSSZZZ)
timeZoneId String time zone id (defaults to TimeZone.getDefault().getID())
mdcKeyPattern String regex to filter MDC keys (does not apply to direct mdc:key access)
ndcPattern String regex to filter NDC items
template String inline JSON template for generating the output (has priority over templateUri)
templateUri String JSON template for generating the output (defaults to classpath:LogstashJsonEventLayoutV1.json)

templateUri denotes the URI pointing to the JSON template that will be used while formatting the log events. By default, LogstashLayout ships the JSONEventLayoutV1 in LogstashJsonEventLayoutV1.json within the classpath:

{
  "mdc": "${json:mdc}",
  "ndc": "${json:ndc}",
  "exception": {
    "exception_class": "${json:exceptionClassName}",
    "exception_message": "${json:exceptionMessage}",
    "stacktrace": "${json:exceptionStackTrace}"
  },
  "line_number": "${json:sourceLineNumber}",
  "class": "${json:sourceClassName}",
  "@version": 1,
  "source_host": "${hostName}",
  "message": "${json:message}",
  "thread_name": "${json:threadName}",
  "@timestamp": "${json:timestamp}",
  "level": "${json:level}",
  "file": "${json:sourceFileName}",
  "method": "${json:sourceMethodName}",
  "logger_name": "${json:loggerName}"
}

In case of need, you can create your own templates with a structure tailored to your needs. That is, you can add new fields, remove or rename existing ones, change the structure, etc. Please note that templateUri parameter only supports file and classpath URI schemes.

Below is the list of known template variables that will be replaced while rendering the JSON output.

Variable Name Description
exceptionClassName logEvent.getThrown().getClass().getCanonicalName()
exceptionMessage logEvent.getThrown().getMessage()
exceptionStackTrace logEvent.getThrown().printStackTrace() (inactive when stackTraceEnabled=false)
exceptionRootCauseClassName the innermost exceptionClassName in causal chain
exceptionRootCauseMessage the innermost exceptionMessage in causal chain
exceptionRootCauseStackTrace the innermost exceptionStackTrace in causal chain
level logEvent.getLevel()
loggerName logEvent.getLoggerName()
mdc Mapped Diagnostic Context Map<String, String> returned by logEvent.getContextData()
mdc:<key> Mapped Diagnostic Context String associated with key (mdcKeyPattern is discarded)
message logEvent.getFormattedMessage()
message:json if logEvent.getMessage() is of type MultiformatMessage and supports JSON, its read value, otherwise, {"message": <formattedMessage>} object
ndc Nested Diagnostic Context String[] returned by logEvent.getContextStack()
sourceClassName logEvent.getSource().getClassName()
sourceFileName logEvent.getSource().getFileName() (inactive when locationInfoEnabled=false)
sourceLineNumber logEvent.getSource().getLineNumber() (inactive when locationInfoEnabled=false)
sourceMethodName logEvent.getSource().getMethodName()
threadName logEvent.getThreadName()
timestamp logEvent.getTimeMillis() formatted using dateTimeFormatPattern and timeZoneId

JSON field lookups are performed using the ${json:<variable-name>} scheme where <variable-name> is defined as <resolver-name>[:<resolver-key>]. Characters following colon (:) are treated as the resolver-key of which as of now only supported in forms mdc:<key> and message:json.

Log4j 2.x Lookups (e.g., ${java:version}, ${env:USER}, ${date:MM-dd-yyyy}) are supported in templates too. Though note that while ${json:...} template variables is expected to occupy an entire field, that is, "level": "${json:level}", a lookup can be mixed within a regular text: "myCustomField": "Hello, ${env:USER}!".

See layout-demo directory for a sample application using the LogstashLayout.

Fat JAR

Project also contains a log4j2-logstash-layout-fatjar artifact which includes all its transitive dependencies in a separate shaded package (to avoid the JAR Hell) with the exception of log4j-core, that you need to include separately.

This might come handy if you want to use this plugin along with already compiled applications, e.g., Elasticsearch 5.x, which requires Log4j 2.x.

Appender Support

log4j2-logstash-layout is all about providing a highly customizable JSON schema for your logs. Though this does not necessarily mean that all of its features are expected to be supported by every appender in the market. For instance, while prettyPrintEnabled=true works fine with log4j2-redis-appender, it should be turned off for Logstash's log4j-json file input type. (See Pretty printing in Logstash issue.) Make sure you configure log4j2-logstash-layout properly in a way that is aligned with your appender of preference.

Performance

The source code ships a JMH benchmark, where you can measure the rendering performance of your preferred platform. As of this writing, using a single thread on an Intel i7 2.70GHz processor powering Java HotSpot 1.8.0_161, the log4j2-logstash-layout can render ~29k LogEvent/sec with a GC allocation rate of ~3 GB/sec.

Contributors

License

Copyright © 2017-2018 Volkan Yazıcı

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.