Additional fields Elasticsearch

[thilankadileepa]

This is the way that I can add logs in to Elasticsearch
{
"_index" : "log4j2",
"_type" : "index",
"_id" : "JpTcx20Bt2rAkaoY4EfT",
"_score" : 1.0,
"_source" : {
"timeMillis" : 1571016201881,
"loggerName" : "Test",
"level" : "INFO",
"message" : "Test Message",
"thread" : "main"
}
}

I want to add a new field as sessionId

I have gone through all the available contents in the web and the source code but could not locate the way to do that.

Is it possible to share an example where I can do this ?

[rfoltyns]

@thilankadileepa You can use ThreadContext as decribed here.

Also, JacksonJsonLayout:VirtualProperty[] will be added in 1.4 (to be released this year) - it will allow to define properties similar to JsonLayout:KeyValuePair[]. It's already in master and it's documented here.

[thilankadileepa]

Thanks a lot.

To use the JacksonJsonLayout:VirtualProperty, what is the library that I should use ?
Is it

org.appenders.log4j
log4j2-elasticsearch-jest
1.3.5

or

org.appenders.log4j log4j2-elasticsearch-core 1.3.5

Please note that I have ES version 7.

Best Regards,
Thilanka

[rfoltyns]

@thilankadileepa It's not released yet (not available in 1.3.5). It will be available as part of 1.4 Core later this year.

[thilankadileepa]

Can I build it by myself and use ?
I am working on a system which plan to go live this year and this library is extremely useful for me. If I can add one additional field (sessionId) my requirement is fully covered.
I have tried it for 2 days but couldn't locate it yet, if you can share an example with me will be a great.

[rfoltyns]

Yes, you can build your own fork if you can't wait for the release.
You can find the example of JacksonJsonLayout:VirtualProperty[] config here.

If you don't want to maintain your own fork internally, JsonLayout:KeyValuePair[] should give you the same functionality - it will have a bigger memory footprint though as buffer pools can't be configured in this case.

[thilankadileepa]

Thanks

I have build it and add the following dependancy

org.appenders.log4j
log4j2-elasticsearch-core
1.4.0-SNAPSHOT

My log4j2.xml looks similar to the example.
And I have added the provided dependencies as well.

I am getting the following error on startup.

2019-10-16 06:54:09,215 main ERROR Error processing element JestBufferedHttp ([AsyncBatchDelivery: null]): CLASS_NOT_FOUND 2019-10-16 06:54:10,096 main ERROR No Elasticsearch client factory [JestHttp|ElasticsearchBulkProcessor] provided for AsyncBatchDelivery: clientObjectFactory 2019-10-16 06:54:10,097 main ERROR Could not create plugin of type class org.appenders.log4j2.elasticsearch.AsyncBatchDelivery for element AsyncBatchDelivery org.apache.logging.log4j.core.config.ConfigurationException: Arguments given for element AsyncBatchDelivery are invalid: field 'clientObjectFactory' has invalid value 'null' at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:209) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)

[rfoltyns]

Use log4j2-elasticsearch-jest module. log4j2-elasticsearch-core is required only if you use programmatic config or extend Core modules.

[thilankadileepa]

Thank You :)
I have changed it to log4j2-elasticsearch-jest as advised.
Seems its goes in the correct path and I am getting the following exception.

**Exception in thread "BatchNotifier" java.lang.NoSuchMethodError: org.appenders.log4j2.elasticsearch.jest.BufferedBulk.getURI()Ljava/lang/String;** at org.appenders.log4j2.elasticsearch.jest.BufferedJestHttpClient.prepareRequest(BufferedJestHttpClient.java:63) at org.appenders.log4j2.elasticsearch.jest.BufferedJestHttpClient.executeAsync(BufferedJestHttpClient.java:47) at org.appenders.log4j2.elasticsearch.jest.JestHttpObjectFactory$1.apply(JestHttpObjectFactory.java:194) at org.appenders.log4j2.elasticsearch.jest.JestHttpObjectFactory$1.apply(JestHttpObjectFactory.java:177)

[rfoltyns]

It caused by latest Jest jars. io.seachbox:jest:6.3.0 replaced io.searchbox.action.AbstractAction.getURI() with io.searchbox.action.AbstractAction.getURI(ElasticsearchVerision).

It will work once you remove dependency on io.seachbox:jest:6.3.0 or provide an explicit dependency on io.seachbox:jest:2.4.0 (as specified by this plugin's POM).

Other option is to switch to log4j2-elasticsearch-hc (and compatible XML config) which doesn't have a dependency on Jest jars.

[thilankadileepa]

Finally I managed to do that :)
Thanks a lot for the help with quick responses.

[rfoltyns]

Great news! Glad to help! :)

Feel free to close this issue if you feel that everything was addressed.

[thilankadileepa]

Final Question

How can I add this dynamically per each request ? For example a session id for each transaction ?
I was managed to add a constant and an env variable as a VirtualProperty

[rfoltyns]

Try using ThreadContext and

<VirtualProperty name="propName" value="$${ctx:sessionId}" dynamic="true" />.

Options available in the value field are well documented in Log4j2 documentation.

[thilankadileepa]

Thanks :)