Log statements with stack trace get truncated due to Logstash 8 KB limit

[gmarziou]

Overview of the issue

When app logs an exception stack trace and the JSON encoded log statement is larger than 8 KB, Logstash is confused and truncate the statement resulting in invalid JSON fragment.

This limit is the default one in Logstash: elastic/logstash#1505

Motivation for or Use Case

Logs with exceptions are very important and should contain all required info for investigation while possibly optimizing the volume of data transferred to Logstash.

JHipster Version(s)

3.0 master

Reproduce the error

Log an error with very long text and exception:

log.error("Very log text .....", exception );

Suggest a Fix

logstash-logback-encoder has few options to limit stack trace depth.

In particular, the rootFirst option which prints the root cause first could be combined with maxLength in bytes.

[PierreBesson]

@gmarziou, I think I solved the issue.

First, I did some test by creating a super long stacktrace. Like that:

        Exception exception = new Exception();
        StackTraceElement st = new StackTraceElement("very long message...","","",1);
        StackTraceElement[] arraySt = new StackTraceElement[1];
        arraySt[0] = st;
        exception.setStackTrace(arraySt);
        log.error("exception",exception);

That's because, although your "reproduce the error" example is enough to get the jsonparse failure in logstash but it does not simulate the real case correctly as it's the log message that is too long not the stacktrace itself.

Then as is proposed in the docs, I defined a `ShortenedThrowableConverter| with setMaxLength(7500) and as expected it's fixing the jsonparsefailure in logstash ! I chose a 7500 bytes stacktrace limit as the 8K limit is for the full log (stacktrace + other fields).
I also tried to apply rootCauseFirst but it didn't change anything in logstash output. Do you think I should still be using this option ?

[gmarziou]

Well truncating the stack trace will avoid the JSON error but it will not help people trying to debug their app as the root cause will be missing. That's why I was thinking of combining truncation and root first.
A good test would be to create a deeply nested exception with something like this:

Exception root = new Exception("I'm the root cause");
Exception high = new Exception("Second level", root);

[gmarziou]

BTW, I got such deeply nested exception from a ribbon timeout on a gateway.

[PierreBesson]

Sorry, I'm just seing your reply now. I will run some tests to see if adding rootFirst helps. Your idea is good ! If this works, I don't see a problem to have the root cause first, it would be an improvement.