Passing a JsonTemplateLayout'ed Log message to a Clickhouse

[emelyanovkr]

My main goal is to log all my messages with LOGGER.info(...) or something like this in JSON format. I've reached that by using this in my .xml configuration : <JsonTemplateLayout eventTemplateUri="classpath:GcpLayout.json"/>.
I got 3 questions:

1. At the start of my program I got this message: 2024-04-05T16:08:34.125664500Z main WARN for URI classpath:GcpLayout.json found 2 resources, using the first one: ..., however, in my resource folder I have only one GcpLayout.json file. So where is the second one? I guess it's some default generated implementation template, but I couldn't find this template in files, so I could disable this WARN message.
2. I have a custom Appender:

@Plugin(
    name = "ClickHouseAppender",
    category = Core.CATEGORY_NAME,
    elementType = Appender.ELEMENT_TYPE,
    printObject = true)
public class ClickHouseAppender extends AbstractAppender {
  private final ClickHouseDAO clickHouseDAO;

  private ClickHouseAppender(
    String name, Filter filter, Layout<String> layout, boolean ignoreExceptions, Property[] properties) {
    super(name, filter, layout, false, properties);
    clickHouseDAO = ClickHouseDAO.getInstance();
  }

  @PluginFactory
  public static ClickHouseAppender createAppender(
    @PluginAttribute("name") String name,
    @PluginElement("Filters") Filter filter,
    @PluginElement("layout") Layout<String> layout,
    @PluginAttribute("ignoreExceptions") boolean ignoreExceptions)
  {
    if (name == null)
    {
      LOGGER.info("No name provided for ClickHouseAppender, default name is set");
      name = "ClickHouseAppender";
    }

    if (layout == null)
    {
      layout = PatternLayout.createDefaultLayout();
    }

    return new ClickHouseAppender(name, filter, layout, ignoreExceptions, null);
  }

  @Override
  public void append(LogEvent event) {
    
  }
}

So, I want to pass a whole JSON Log message

{"timestamp":"2024-04-05 16:56:17","severity":"INFO","message":"Compression of 20.80 MB of data with rate 26.30 MB/sec finished in 0.79 sec.","logging.googleapis.com/labels":{"compression_rate":"26.29693268825372","data_size":"20.80087375640869"},"_exception":{"stackTrace":""},"_thread":"pool-2-thread-13"}

As it is to my clickhouse Database, so I just need to recover this message as String, possibly.
I've seen about this discussion - #2013, but I find putting all my parameters in MDC might be a little bit strange to do, so I'm asking about other possibilites.

3. Clickhouse accepts InputStream as well, so I could pass my message straight in the stream and send it to DB. Is it possible with Log4J2?

[ppkarwasz]

1. There is a GcpLayout.json embedded in log4j-layout-template-json. If it is missing something you can submit a PR, otherwise you can rename your template for the warning to disappear. Since the order of the classpath is usually random, this will prevent your application from randomly switching between the two templates.
2. To retrieve the formatted message you can use:

   final String message = logEvent.getMessage().getFormattedMessage();

   Note: if the string must be a valid JSON you must prevent users from configuring a different layout. Certainly using PatternLayout will lead to data corruption.

[vy]

1. At the start of my program I got this message: 2024-04-05T16:08:34.125664500Z main WARN for URI classpath:GcpLayout.json found 2 resources, using the first one: ..., however, in my resource folder I have only one GcpLayout.json file. So where is the second one? I guess it's some default generated implementation template, but I couldn't find this template in files, so I could disable this WARN message.

I suspect you have multiple GcpLayout.json files in your classpath. If you'd look at JsonTemplateLayout.java, eventTemplateUri="classpath:GcpLayout.json" translates to LoaderUtil.findResources("GcpLayout.json") call. You can connect a debugger, put a breakpoint there, and see what causes multiple GcpLayout.json resource hits.

2. I have a custom Appender: ... So, I want to pass a whole JSON Log message

Make the layout a requirement and avoid surprises. That is, no need for a if (layout == null) { layout = ... }. If there is no layout, simply fail. Then, in your appender, the rest is easy:

@Override
public void append(LogEvent event) {
    String serializedEvent = layout.toSerializable(event);
    clickHouseDb.write(serializedEvent);
}

As it is to my clickhouse Database, so I just need to recover this message as String, possibly.
I've seen about this discussion - #2013, but I find putting all my parameters in MDC might be a little bit strange to do, so I'm asking about other possibilites.

I am not able to follow you. Can you explain in detail what do you want to write to your ClickHouse database/table?

3. Clickhouse accepts InputStream as well, so I could pass my message straight in the stream and send it to DB. Is it possible with Log4J2?

Good question. Writing an efficient and resilient appender is not trivial. You can check out Encoder<T>#encode(T,ByteBufferDestination) method (Layout<T> extends from Encoder<LogEvent>) and how it is employed in AbstractOutputStreamAppender. In a nutshell, your sink (i.e., ClickHouse) needs to implement ByteBufferDestination. I would not suggest this much sophistication; simply stick to serialize-to-String-and-write-to-db, unless it constitutes an efficiency a bottleneck.

[emelyanovkr]

I suspect you have multiple GcpLayout.json files in your classpath. If you'd look at JsonTemplateLayout.java, eventTemplateUri="classpath:GcpLayout.json" translates to LoaderUtil.findResources("GcpLayout.json") call. You can connect a debugger, put a breakpoint there, and see what causes multiple GcpLayout.json resource hits.

file:/../gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-layout-template-json/2.23.1/fa3b683c257fabc4cb025c5b7c2b382c9487bed3/log4j-layout-template-json-2.23.1.jar!/GcpLayout.json

He is getting a layout from .jar file itself, so I guess I should just ignore this WARN cause there is nothing to do with it.

Make the layout a requirement and avoid surprises. That is, no need for a if (layout == null) { layout = ... }. If there is no layout, simply fail. Then, in your appender, the rest is easy:

That's done, thanks for a hint. I am getting a JSONFormatted message within LogEvent like this:

  @Override
  public void append(LogEvent event) {
    String serializedEvent = (String) getLayout().toSerializable(event);
    System.out.print(serializedEvent);
  }

I am not able to follow you. Can you explain in detail what do you want to write to your ClickHouse database/table?

My ClickHouse DB contains a table with one field - (log JSON), so there is a format named JSON. I want just to pass my JSON'ed string to a db so I could store all my logs there with JSON format of the field.

I would not suggest this much sophistication; simply stick to serialize-to-String-and-write-to-db, unless it constitutes an efficiency a bottleneck.

Yeah, seems like it's working like that, mentioned here - ClickHouse-Java Issues/Passing JSON to a DB. I'm going to try that.

[emelyanovkr]

1. So I've managed to store my logs in a database, but somehow all fields positions are reversed:

{
  "exception": {
    "stackTrace": ""
  },
  "mdc_fields": {
    "compression_rate": "4.671450264565971",
    "data_size": "40.24454402923584"
  },
  "message": "Compression of 40.24 MB of data with rate 4.67 MB/sec finished in 8.62 sec.",
  "severity": "INFO",
  "thread": "pool-2-thread-4",
  "timestamp": ""
}

2. Also, timestamp field is not included in GrcLayout.json file, but somehow it's appeared in database record and somehow it's not appearing in System.out.println(...) output in IDE. I solved that by removing this field with .replaceAll(...), but maybe there is another solution to it.
3. Is it possible to include 'exception' field in the template only if there are any exceptions happened or severityLevel == ERROR?
4. Is there an option to encode messages or something in pattern with %enc{%m}{CRLF} for a template layout? I've tried something like this:

"message": {
    "$resolver": "pattern",
    "pattern": "%enc{%m}{CRLF}",
    "stackTraceEnabled": true
  },

But it results in having \r\n, however I need UNIX style sequences for inserting my string. One of the solutions is like this - .replaceAll("\\r\\n", "\n"), but maybe there is more appropriate way to do this. I found about parameter eventDelimeter, however when using like this: <JsonTemplateLayout eventTemplateUri="classpath:GcpLayout.json" eventDelimiter="\n"/> my delimeters are still the same as from System.lineSeparator(), so for Windows it's just \r\n.
5. How to ensure pattern for exception message or exception classname? I want to encode them with CRLF, but I haven't found anything useful in here about patterns for specific fields - https://logging.apache.org/log4j/2.x/manual/json-template-layout.html#event-template-resolvers. Because I'm encountering an error while getting ERROR level of severity -

{"severity":"ERROR","thread":"main","message":"FAILED SEARCH DIRECTORY java.lang.NullPointerException: null\r\n\tat java.base/java.util.Objects.requireNonNull(Objects.java:233) ~[?:?]\r\n\tat com.crypto.service.util.TicketsDataReader.getFilesInDirectory(TicketsDataReader.java:54) [main/:?]\r\n\tat com.crypto.service.util.TicketsDataReader.readExecutor(TicketsDataReader.java:66) [main/:?]\r\n\tat com.crypto.service.LoaderApplication.main(LoaderApplication.java:10) [main/:?]\r\n","exception":{"class":"java.lang.NullPointerException","message":null}}
2024-04-06T12:55:29.700753500Z main ERROR An exception occurred processing Appender ClickHouseAppender java.lang.RuntimeException: FAILED TO INSERT STRING - Code: 117. DB::Exception: Cannot parse object: (at row 1)
: 
Row 1:
Column 0,   name: timestamp, type: DateTime,       parsed text: "1712408129148"
Column 1,   name: log,       type: Object('json'), parsed text: "{<DOUBLE QUOTE>severity<DOUBLE QUOTE>:<DOUBLE QUOTE>ERROR<DOUBLE QUOTE>,<DOUBLE QUOTE>thread<DOUBLE QUOTE>:<DOUBLE QUOTE>main<DOUBLE QUOTE>,<DOUBLE QUOTE>message<DOUBLE QUOTE>:<DOUBLE QUOTE>FAILED SEARCH DIRECTORY java.lang.NullPointerException: null<BACKSLASH>r<BACKSLASH>n<BACKSLASH>tat java.base/java.util.Objects.requireNonNull(Objects.java:233) ~[?:?]<BACKSLASH>r<BACKSLASH>n<BACKSLASH>tat com.crypto.service.util.TicketsDataReader.getFilesInDirectory(TicketsDataReader.java:54) [main/:?]<BACKSLASH>r<BACKSLASH>n<BACKSLASH>tat com.crypto.service.util.TicketsDataReader.readExecutor(TicketsDataReader.java:66) [main/:?]<BACKSLASH>r<BACKSLASH>n<BACKSLASH>tat com.crypto.service.LoaderApplication.main(LoaderApplication.java:10) [main/:?]<BACKSLASH>r<BACKSLASH>n<DOUBLE QUOTE>,<DOUBLE QUOTE>exception<DOUBLE QUOTE>:{<DOUBLE QUOTE>class<DOUBLE QUOTE>:<DOUBLE QUOTE>java.lang.NullPointerException<DOUBLE QUOTE>,<DOUBLE QUOTE>message<DOUBLE QUOTE>:null}}"ERROR
Code: 117. DB::Exception: Cannot parse object. (INCORRECT_DATA) (version 24.1.2.10900 (official build))

[emelyanovkr]

So, a total answer for anyone who might find this discussion useful:

He is getting a layout from .jar file itself, so I guess I should just ignore this WARN cause there is nothing to do with it.

Rename a layout file and passing it to a .xml configuration or somewhere else totally fix the WARN issue.

---

So I've managed to store my logs in a database, but somehow all fields positions are reversed:

ClickHouse sorting all data automatically, that why it's seems to be reversed, it's just sorted.

---

Also, timestamp field is not included in GrcLayout.json file, but somehow it's appeared in database record and somehow it's not appearing in System.out.println(...) output in IDE. I solved that by removing this field with .replaceAll(...), but maybe there is another solution to it.

That appears to be because clickhouse stores a row structure under JSON format for a first input record, so if other records doesn't contain any info for a field, then it just puts "" there. That's why timestamp is "" - it was removed from client-side template, but still remained in the server table structure. TRUNCATE TABLE seems to fix the structure, if needed.

---

Is there an option to encode messages or something in pattern with %enc{%m}{CRLF} for a template layout? I've tried something like this:

I don't know still about making a pattern for a resolver, however about delimeters - two ways:

1. Just use eventDelimeter property in configuration, "" blank or space is possible.
2. .replaceAll(System.lineSeparator(), " ")

---

Because I'm encountering an error while getting ERROR level of severity -

The issue on the clickhouse site, somehow it's just can't accept escape sequences in the message, so all \r\n\t was replaced with some other delimeter, for example, or could be just removed.