Adding custom properties, that change during processing

[fire-papaya]

Description
I want to add custom fields into logging object, but adding it into ThreadContext is not an option. I have to log HTTP requests to external services, and I want to log fields like url, verb, headers, etc. There might be several sequential requests with different urls, verbs, so adding those fields into ThreadContext makes no sense. As of now i log them as part of message.

Configuration
log4j2.xml:

<Configuration status="WARN" monitorInterval="30">
    <Properties>
        <Property name="LOG_PATTERN">
            %d{yyyy-MM-dd HH:mm:ss.SSS} [%X{traceId},%X{spanId}] %highlight{ %5p }{FATAL=red blink, ERROR=red, WARN=yellow bold, INFO=green, DEBUG=green bold, TRACE=blue}${hostName} --- [%15.15t] %-40.40c{1.} : %m%n%ex
        </Property>
    </Properties>
    <Appenders>
        <Elasticsearch name="ELASTIC_SEARCH_BATCH">
            <RollingIndexName indexName="api-io" pattern="yyyy-MM" />
            <JacksonJsonLayout>
                <VirtualProperty name="trace_id" value="$${ctx:traceId:-}" dynamic="true" />
            </JacksonJsonLayout>
            <AsyncBatchDelivery>
                <IndexTemplate name="api-io-template" path="classpath:index_template.json" />
                <JestHttp serverUris="http://localhost:9200" />
            </AsyncBatchDelivery>
        </Elasticsearch>
    </Appenders>
    <Loggers>
        <Logger name="ApiLogFilter" level="INFO" additivity="false" >
            <AppenderRef ref="ELASTIC_SEARCH_BATCH" />
        </Logger>
    </Loggers>
</Configuration>

Note: Other appenders and their configs are omitted
Note: traceId is injected into Thread context

index_template.json

{
  "index_patterns": [
    "api-io-*"
  ],
  "settings": {
    "index": {
      "refresh_interval": "5s"
    }
  },
  "mappings": {
    "properties": {
      "trace_id": {
        "type": "keyword"
      },
      "url": {
        "type": "keyword"
      },
      "http_verb": {
        "type": "keyword"
      },
      "loggerName": {
        "type": "keyword"
      },
      "message": {
        "type": "text"
      },
      "timeMillis": {
        "type": "date",
        "format": "epoch_millis"
      },
      "thread": {
        "type": "keyword"
      },
      "level": {
        "type": "keyword"
      }
    }
  }
}

Runtime (please complete the following information):

• log4j2-elasticsearch-jest:1.5.2
• SpringBoot 2.2.1
• JVM 11

Additional context
I have surfed through other related issues. Most of the time people got satisfied with ThreadContext solution, unfortunately this is not my case. As far as I understood writing my own mix-in will not work either. I was thinking to implement my own custom lookup just wanted to be sure that I am on the right path and there are not more trivial solutions that I have missed.

[rfoltyns]

@burning-papaya I don't see many other options than ThreadContext tbh.

If you're tied to Log4j2 and need to log these events with other appenders, custom MessageFactory and mixins for Message may be a way to go. Otherwise, you can use AsyncBatchDelivery directly and serialize your own classes instead of LogEvent.

[fire-papaya]

@rfoltyns Thank you, for your reply.

Could you please elaborate on the last sentence (direct usage of AsyncBatchDelivery with my own class)? I have a class which i have to log, every field has to be written in elasticsearch separately for ease of searching (it's Kotlin data class):

@JsonAutoDetect(fieldVisibility = JsonAutoDetect.Visibility.ANY)
 data class RequestLog (
    @JsonProperty("verb") var verb: String = "",
    @JsonProperty("uri") var uri: String = "",
    @JsonProperty("query") var query: String = "",
    @JsonProperty("body") var body: String = "",
    @JsonProperty("io_type") var type: String = "OUT_REQUEST"
)

[rfoltyns]

You can use parts of programmatic config test and use this library like this:

asyncBatchDelivery.add(indexName, itemSourceFactory.create(instanceOfYourClass, objectWriter));

ItemSourceFactory and AsyncBatchDelivery must be started explicitly in this case.

[rfoltyns]

@burning-papaya If you haven't made any progress yet, I think that you might be interested in this. What you described is achievable with Log4j2 params and unwrapping serializers.