ClastCastException in 1.11.6 by LogLevel Level.ERROR

[JanGoerss]

Hi,

logging in 1.11.6 throw a ClassCastException and every error log is not sent over HttpEventCollector to SplunkServer

java.lang.ClassCastException: class org.apache.logging.log4j.core.impl.Log4jLogEvent cannot be cast to class org.apache.logging.log4j.core.impl.MutableLogEvent (org.apache.logging.log4j.core.impl.Log4jLogEvent and org.apache.logging.log4j.core.impl.MutableLogEvent are in unnamed module of loader 'app')
at com.splunk.logging.HttpEventCollectorLog4jAppender.generateErrorDetail(HttpEventCollectorLog4jAppender.java:275)

Environment:
Java 17
Splunklogging 1.11.6
log4j 2.17.2
slf4j 1.7.36

[bparmar-splunk]

Hi @JanGoerss,
Thank you for reaching out.

We have tried to reproduce the issue which you faced in our local, but it worked well. We have used all above versions which you mentioned, in our local test.
It seems there are multiple versions of Log4J library on your machine.
Please try to remove all dependencies from local maven repo and re-check again the same use case.

If it does not work, then please do share us the detail steps to reproduce.

[JanGoerss]

Hi,

i created a simple Gradleproject. This project reproduce the error. Use Gradletask "run"

Environment:
Java 17
Splunklogging 1.11.6
log4j 2.17.2
slf4j 1.7.36
splunk-logging-failed.zip

Log of Gradle Task "run"
06:56:52: Executing 'run'...

Task :app:compileJava UP-TO-DATE
Task :app:processResources
Task :app:classes

Task :app:run
2022.09.22 06:56:53 - INFO - App:INFO
2022.09.22 06:56:53 - ERROR - App:ERROR
2022-09-22 06:56:53,701 Log4j2-AsyncAppenderEventDispatcher-1-ASYNC ERROR An exception occurred processing Appender http-input java.lang.ClassCastException: class org.apache.logging.log4j.core.impl.Log4jLogEvent cannot be cast to class org.apache.logging.log4j.core.impl.MutableLogEvent (org.apache.logging.log4j.core.impl.Log4jLogEvent and org.apache.logging.log4j.core.impl.MutableLogEvent are in unnamed module of loader 'app')
at com.splunk.logging.HttpEventCollectorLog4jAppender.generateErrorDetail(HttpEventCollectorLog4jAppender.java:275)
at com.splunk.logging.HttpEventCollectorLog4jAppender.append(HttpEventCollectorLog4jAppender.java:245)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
at org.apache.logging.log4j.core.appender.AsyncAppenderEventDispatcher.dispatch(AsyncAppenderEventDispatcher.java:130)
at org.apache.logging.log4j.core.appender.AsyncAppenderEventDispatcher.dispatchRemaining(AsyncAppenderEventDispatcher.java:110)
at org.apache.logging.log4j.core.appender.AsyncAppenderEventDispatcher.run(AsyncAppenderEventDispatcher.java:75)

[bparmar-splunk]

Hi @JanGoerss,
We have tried using your application and below are our findings:

• The configurations in log4j2.xml file was not proper as per Splunk doc. Please refer Splunk doc.
• We have tried changing the configuration as per doc and it started working.
  Also, we are attaching a file with revised configuration settings. Please verify your changes and let us know about your test results.
  File:
  log4j2.xml.zip

[JanGoerss]

Hi,

it runs without async, but why shouldn't you be allowed to use async logging (low-latency logging) with splunk.

It is an implementation change from 1.11.5 to 1.11.6.

https://github.com/splunk/splunk-library-javalogging/blob/main/src/main/java/com/splunk/logging/HttpEventCollectorLog4jAppender.java

private String generateErrorDetail(final LogEvent event) {

LogEvent is an interface and has many implementations.

Why is the event hard cast to MutableLogEvent in line 275 in Method generateErrorDetail?

        if (event.getThrown() == null && (((MutableLogEvent) event).getParameters()).length <= 0) {

[bparmar-splunk]

HI @JanGoerss,
Thank you for your response.

We have used MutableLogEvent because we want exception details from parameters when it does not have any information in getThrown() method. And by using its parent, we cannot fetch its parameters.

Also, We are open for any suggestion OR PR which can support for your use-case.

[MartinHeinen]

We are running into the same issue after upgrading to version 1.11.6
This issue causes our threads reading from a RabbitMQ Queue to be cancelled and slowly stopping all our processing, so we had to downgrade back to 1.11.5.
Is there a way to address this?
In general, I was surprised, that a logging library would throw an exception in the first place. Is it the expectation to always try/catch logging or are there other ways we can avoid running into an exception?
Additionally, I would suggest, to firstly check if the cast to MutableLogEvent is possible and if not log the event as done before the version upgrade.

Kind regards,

Martin

[bparmar-splunk]

Hi @MartinHeinen,
Thank you for raising this up.

We will look into this issue and will update you and @JanGoerss for the fix.
Thank you.

[bparmar-splunk]

Hi @MartinHeinen & @JanGoerss,
We have fixed the issue in our local. We are yet to push and release with new minor version. But before that we would request you guys to test the build in your application using a jar file (attached here) just to make sure that the issue has been resolved in both the use-cases.
JAR:
splunk-library-javalogging-1.11.6.jar.zip

Please share your results once you test your application.

Thank you.

[JanGoerss]

Hi,

i tested with the jar and it works in our application. Thanks

[bparmar-splunk]

Hi @JanGoerss,
Thank you for the update.

We are still waiting for the response from @MartinHeinen, if it works well then we will plan a minor release post that.

Thank you.

[MartinHeinen]

Hi,

For us it's not easy to reproduce the issue in Pre-Production, but during our testing we did not run into any issues with the jar you provided and the exception did not occur again so far.

Thanks

[bparmar-splunk]

Hi @MartinHeinen,
Thank you for the update.
We will plan for a release soon and will post new version link here.

Thanks.

[bparmar-splunk]

Hi @JanGoerss & @MartinHeinen,
We have successfully released new version 1.11.7 of this library.

Please update your library to this version.
We are closing this issue.

Thank you.