Add support to log directly into Elasticsearch

[michaelhyatt]

It should be possible to use appenders to log directly into Elasticsearch without even going through Filebeat using something, like this one:

https://github.com/rfoltyns/log4j2-elasticsearch

[xeraa]

Is that really a pattern we want to promote? While I see the convenience of being able to skip Filebeat it has the downsides of:

1. Coupling:
   • Suddenly your application needs to know the details of your logging backend (URI, credentials,...)
   • Version compatibility is also getting an issue — IMO the library doesn't really support 7. It supports the deprecated Transport client (with a pretty tight coupling to major versions) and JEST, which I thought was dead but just got some recent updates though I think it doesn't support 7.x officially yet.
2. Outages:
   • I'm not sure how this one behaves, but Logback will IMO not retry to connect when the logging destination is down during the application startup (intermittent outages will be recovered though).
   • When your network or logging system is having issues you are blind without a fallback to log files (though there seems to be a fallback planned for the next release).
   • The default buffer size seems to be 10,000 after which you'd lose the oldest messages (I assume blocking isn't an option).

[felixbarny]

Another thing is that we'd be missing the metadata Filebeat appends to the logs, for example, the host, docker and Kubernetes metadata as well as the index template provided by Filebeat.

But I don't see the two libraries as competing. It seems like log4j2-elasticsearch provides an Appender. This library provides a Layout. So it seems it would be possible to use the EcsLayout in combination with the Elasticsearch appender.

[michaelhyatt]

That was exacttly what I wanted: to combine the Layout this library provides with Elasticsearch Appender. For additional safety, the users can use additional appenders, such as rolling log file, but there is value in offering them a choice to bypass filebeat and go to ES directly.

What is needed to make it work is making sure the appender can receive the fields from the layout definition to push the data into ES.

Hope it makes sense.

[felixbarny]

What is needed to make it work is making sure the appender can receive the fields from the layout definition to push the data into ES.

In theory, that should not require any additional effort. But I haven't tested the two in combination yet.

[felixbarny]

I've summarized some of the advantages in #8

[michaelhyatt]

Tried it and it didn't work. I don't think I fully grasp how appenders and layout interact. This is the config I tried:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="DEBUG" packages="co.elastic.logging.log4j2,org.appenders.log4j2.elasticsearch">
	<Appenders>
		<Console name="LogToConsole" target="SYSTEM_OUT">
			<EcsLayout serviceName="my-app" />
		</Console>
		<!-- <File name="LogToFile" fileName="logs/app.log"> <EcsLayout serviceName="my-app"/> 
			</File> -->
		<Elasticsearch name="elasticsearchAsyncBatch">
			<IndexName indexName="log4j2" />
			<AsyncBatchDelivery>
				<IndexTemplate name="log4j2" path="classpath:indexTemplate.json" />
				<JestHttp serverUris="http://localhost:9200" />
			</AsyncBatchDelivery>
		</Elasticsearch>
	</Appenders>
	<Loggers>
		<Root level="info">
			<!-- <AppenderRef ref="LogToFile"/> -->
			<AppenderRef ref="LogToConsole" />
			<AppenderRef ref="elasticsearchAsyncBatch" />
		</Root>
	</Loggers>
</Configuration>

My dependencies:

		<dependency>
			<groupId>co.elastic.logging</groupId>
			<artifactId>log4j2-ecs-layout</artifactId>
			<version>${java-ecs-logging.version}</version>
		</dependency>
		<dependency>
			<groupId>org.appenders.log4j</groupId>
			<artifactId>log4j2-elasticsearch-jest</artifactId>
			<version>1.3.5</version>
		</dependency>

The error I am getting:

Caused by: java.lang.IllegalAccessError: tried to access class org.apache.logging.log4j.core.jackson.StackTraceElementMixIn from class org.apache.logging.log4j.core.jackson.ExtendedLog4j2JsonModule
	at org.apache.logging.log4j.core.jackson.ExtendedLog4j2JsonModule.setupModule(ExtendedLog4j2JsonModule.java:37) ~[?:?]
	at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:747) ~[jackson-databind-2.8.9.jar:2.8.9]
	at org.appenders.log4j2.elasticsearch.JacksonJsonLayout$Builder.createConfiguredWriter(JacksonJsonLayout.java:122) ~[?:?]
	at org.appenders.log4j2.elasticsearch.JacksonJsonLayout$Builder.build(JacksonJsonLayout.java:114) ~[?:?]
	at org.appenders.log4j2.elasticsearch.ElasticsearchAppender$Builder.build(ElasticsearchAppender.java:126) ~[?:?]
	at org.appenders.log4j2.elasticsearch.ElasticsearchAppender$Builder.build(ElasticsearchAppender.java:86) ~[?:?]
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122) ~[log4j-core-2.8.2.jar:2.8.2]

[felixbarny]

That's an exception related to log4j2-elasticsearch and does not seem to be caused by this project. Seems like a Jackson version mismatch or log4j2-elasticsearch ships with a Jackson version which is not compatible with your Java version. Does the error also happen when removing the java-ecs-logging dependency?

I have not tried it out but the configuration would look something like this:

<Elasticsearch name="elasticsearchAsyncBatch">
	<EcsLayout serviceName="my-app" />
	<IndexName indexName="log4j2" />
	<AsyncBatchDelivery>
		<IndexTemplate name="log4j2" path="classpath:indexTemplate.json" />
		<JestHttp serverUris="http://localhost:9200" />
	</AsyncBatchDelivery>
</Elasticsearch>

Each appender needs a layout which formats a log event into a String. The appender is only responsible for, well, appending that string to a specific output, like a file, or Elasticsearch.

[wolframhaussig]

We currently use logback for logging. As our applications are running on HP-UX we cannot use FileBeat to read the logfiles and we cannot use LogStash either(JNI native library). So we currently are between a rock and a hard place... The only way currently available would be to use the SyslogAppender with LogStash on the ElasticStack server being the Syslog server.

With this issue here being solved it would be a lot easier for us...

[Marx2]

Following is working for me.

<Elasticsearch name="elasticsearchAsyncBatch">
	<EcsLayout serviceName="my-app" />
	<IndexName indexName="log4j2" />
	<AsyncBatchDelivery>
                <JestHttp serverUris="http://localhost:9200/_bulk"/>
	</AsyncBatchDelivery>
</Elasticsearch>

However removal of
<IndexTemplate name="log4j2" path="classpath:indexTemplate.json" />
is probably responsible for lack of many important fields, especially no date/timestamp is available to be used for sorting/filtering in Kibana

Having this line attached issues an error:

2020-01-13 17:42:56,999 main ERROR layout EcsLayout has no parameter that matches element EcsLayout
2020-01-13 17:42:57,001 main ERROR Could not create plugin of type class org.appenders.log4j2.elasticsearch.IndexTemplate for element IndexTemplate org.apache.logging.log4j.core.config.ConfigurationException
	at org.appenders.log4j2.elasticsearch.IndexTemplate$Builder.loadClasspathResource(IndexTemplate.java:132)
	at org.appenders.log4j2.elasticsearch.IndexTemplate$Builder.loadSource(IndexTemplate.java:96)
	at org.appenders.log4j2.elasticsearch.IndexTemplate$Builder.build(IndexTemplate.java:86)
	at org.appenders.log4j2.elasticsearch.IndexTemplate$Builder.build(IndexTemplate.java:65)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:964)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:904)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
	at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:548)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:620)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:637)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
	at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
	at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
	at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:67)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:59)
	at org.springframework.boot.SpringApplication.<clinit>(SpringApplication.java:195)
	at com.ydc.discovery.Application.main(Application.java:26)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:543)
	at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: java.lang.NullPointerException
	at java.base/java.io.Reader.<init>(Reader.java:167)
	at java.base/java.io.InputStreamReader.<init>(InputStreamReader.java:97)
	at org.appenders.log4j2.elasticsearch.IndexTemplate$Builder.loadClasspathResource(IndexTemplate.java:121)
	... 32 more

[rfoltyns]

As @felixbarny said - ecs-logging-java provides a layout, not an ES client.

@michaelhyatt @wolframhaussig If you'd like to use this layout, you can use it with log4j2-elasticsearch and JestHttp or (for high throughput scenarios) HCHttp. However, HCHttp is not compatible with EcsLayout, so feel free to open an issue in my repo if you need any support.

@Marx2 I've just tested EcsLayout with JestHttp without an IndexTemplate - works just fine. EscLayout did a great job here. Kibana can find a date column during index pattern definition. Could you verify it again, please?

@xeraa Addressing some of your concerns:

• fallback to another appender (e.g. file appender) was added in 2017
• retry was added in latest releases
• log4j2-elasticsearch uses Jest only for _bulk API. It's compatible with 7.x. as there's been no major changes there since the last few major releases. However, since Jest module allocates a ton of memory - log4j2-elasticsearch-hc would be my best recommendation atm due to much lower allocation rate.

[felixbarny]

FYI: we're planning to add support in the Elastic APM Java agent to automatically ship the application logs: elastic/apm#252

TL;DR: The agent would create a "shadow log file" for every of the application's log file. That file will then be tailed by the agent and sent to the APM Server which does some processing, like adding metadata to the log events and sends them to Elasticsearch.

[wolframhaussig]

@felixbarny Thank you so much for your work! I know this might take a while but if you need a tester just send me a note and I would be happy to give it a try.

[felixbarny]

Closing this as there are currently no plans to provide appenders that write to Elasticsearch.

But there are the alternatives of using with 3rd party appenders and there are plans to let APM agents ship the logs.