Logging MDC tags stopped working in 2.5.10

[scheleaap]

After upgrading Kamon from 2.5.9 to 2.5.10, tags set in the Kamon context are no longer included in log messages.

At our company, we write log messages in JSON format and add information to the messages through the MDC through Kamon.
We use Kamon with logback-classic and logstash-logback-encoder.

Expected behavior

For example, we add the tag application-startup-marker to a log message when the application is starting up:

object Application extends App with StrictLogging {
  Kamon.init()
  Kamon.runWithContextTag("application-startup-marker", value = true)(logger.info("Application starting..."))
}

The log message then looks as follows (for Kamon ≤ 2.5.9):

{"@timestamp":"2022-11-02T08:42:52.879Z","level":"INFO","level_value":20000,"message":"Application starting...",
"logger_name":"foo.Application$","thread_name":"main","caller_class_name":"foo.Application$","caller_method_name":"$anonfun$new$1",
"caller_file_name":"Application.scala","caller_line_number":8,"application-startup-marker":"true"}
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Actual behavior

However, when we upgrade to 2.5.10, the tag suddenly disappears:

{"@timestamp":"2022-11-02T08:43:24.648Z","level":"INFO","level_value":20000,"message":"Application starting...",
"logger_name":"foo.Application$","thread_name":"main","caller_class_name":"foo.Application$","caller_method_name":"$anonfun$new$1",
"caller_file_name":"Application.scala","caller_line_number":8}
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Minimal example

I have created a minimal example here: https://github.com/scheleaap/kamon-issue. Run with sbt run.

If you change the line

      "io.kamon" %% "kamon-bundle" % "2.5.10",

to

      "io.kamon" %% "kamon-bundle" % "2.5.9",

in build.sbt, you will see the tag reappears.

Conclusion

This looks like a regression in Kamon 2.5.10.

When looking at the changes made for that release, I see there were changes in MDC-related code. Perhaps a bug was introduced there?

[antoine-morisseau]

Same issue here quit annoying since it stop us to use kamon above of 2.5.9 because we heavily rely on MDC for logging purpose.

[ivantopo]

Hey @scheleaap, thank you so much for reporting this with a reproducible, it made it super easy to fix the problem! 🎉

Indeed, the regressing came with v2.5.10 where I rewrote a part of the instrumentation and it would basically just append things to the MDC if there was already something in the MDC. If the MDC was empty, Kamon wouldn't add anything to it 🤦

Fixed in #1227 and should be released pretty soon.

[scheleaap]

@ivantopo thank you for investigating & fixing the issue so fast, that's great! Looking forward to the next release :-)

[Dichotomia]

Hello!
Any news about a release containing the fix?

Thanks!