ECS structured logging logs some ECS fields with incorrect types

[N4zroth]

Hey,

we're using Spring Boot 4.0.2 with structured logging in ECS format enabled:

logging:
  structured:
    format:
      console: ecs
      file: ecs

According to ECS's documentation, there are fields that should not be a String, e. g. url.port. Problem is, this and probably others are logged as String as can be seen here.

I understand that this is not easy solvable as all MDC fields are String, but this violates the ECS standard and produces lots of problems when ingested in an Elastic cluster without transformation.
It'd probably best if the logger checked ECS field types and only logged them if they were convertible to the type required by the spec.

We're currently solving this by using special transformations in the spring-logback.xml which also doesn't feel good.

[philwebb]

This is indeed quite a hard problem to fix in a generic way. The best idea I can think of is to update org.springframework.boot.logging.structured.ContextPairs so that ElasticCommonSchemaStructuredLogFormatter could pass some hints about the expected type.

In the meantime, perhaps you can create your own JsonWriterStructuredLogFormatter that's based on ElasticCommonSchemaStructuredLogFormatter and adds the url.port entry directly using the correct type.

[N4zroth]

Thanks for the format ;)

And thanks for the hint, I've added an example formatter that transforms MDC fields before adding them, discarding 'wrong' fields. The println when discarding a field might be problematic for our setup but I'll test that out. Doesn't really matter, at least we have a solution that works for us 👍

[philwebb]

We discussed this today and we'd like to fix this by automatically converting to the correct type. This is likely to be quite an involved bit of work so I've flagged the issues as team-only.