Mitigate: RCE 0-day exploit found in log4j, a popular Java logging package https://github.com/elastic/elasticsearch/issues/81618

[PrajaktaPurohit]

Signed-off-by: Prajakta Purohit prajakta@chef.io

[netlify]

👷 Deploy Preview for chef-server processing.

🔨 Explore the source changes: 768f12d

🔍 Inspect the deploy log: https://app.netlify.com/sites/chef-server/deploys/61b3c447fd637100074bdecf

[PrajaktaPurohit]

https://buildkite.com/chef/chef-chef-server-main-omnibus-adhoc/builds/3569
https://buildkite.com/chef/chef-umbrella-main-chef-server/builds/949
https://buildkite.com/chef/chef-umbrella-main-chef-server-full/builds/95

[lbakerchef]

Looks good. Is there anything showing this working?

[jasonwbarnett]

I'm using chef-server-core v14.9.23 on RHEL 7.

I manually added this into the Chef Infra Server (/var/opt/opscode/elasticsearch/config/jvm.options) and after restarting elastic search, it fails to start.

‐Dlog4j2.formatMsgNoLookups=true

You see it try to start with this:

/opt/opscode/embedded/open-jre//bin/java -cp /opt/opscode/embedded/elasticsearch/lib/* org.elasticsearch.tools.launchers.JvmOptionsParser /var/opt/opscode/elasticsearch/config/jvm.options

and after that it doesn't actually successfully start the service. It ends up executing the above over and over and over.

How to reproduce

1. Switch to root user on chef infra server

   sudo su -
   

2. Stop elasticsearch

   chef-server-ctl stop elasticsearch
   

3. Add config to /var/opt/opscode/elasticsearch/config/jvm.options

4. Impersonate startup process

   Do the following as root user:

   ulimit -n 65536
   
   export ES_HOME=/var/opt/opscode/elasticsearch            #/var/opt/opscode/elasticsearch
   export ES_DATA=/var/opt/opscode/elasticsearch/data       #/var/opt/opscode/elasticsearch/data
   
   export JAVA_HOME=/opt/opscode/embedded/open-jre/
   export ES_PATH_CONF=/var/opt/opscode/elasticsearch/config
   export PATH=/opt/opscode/embedded/bin:$JAVA_HOME/bin:$ES_HOME:$PATH #/opt/opscode/embedded/bin
   
   export TMPDIR=/var/opt/opscode/elasticsearch/tmp
   export ES_TMPDIR=/var/opt/opscode/elasticsearch/tmp
   
   cd $ES_HOME
   
   exec /opt/opscode/embedded/bin/chpst -P -u opscode -U opscode /opt/opscode/embedded/elasticsearch/bin/elasticsearch
   encountered [1] error parsing [/var/opt/opscode/elasticsearch/config/jvm.options]
   [1]: encountered improperly formatted JVM option line [‐Dlog4j2.formatMsgNoLookups=true] on line number [51]
   

p.s. I used viddy at 200ms to capture the following command retrying:

/opt/opscode/embedded/open-jre//bin/java -cp /opt/opscode/embedded/elasticsearch/lib/* org.elasticsearch.tools.launchers.JvmOptionsParser /var/opt/opscode/elasticsearch/config/jvm.options

[PrajaktaPurohit]

I am currently testing the build off of this PR and it elasticsearch service comes up ok. I will try the manual steps by editing the config after testing the upgrades.

[jasonwbarnett]

I am currently testing the build off of this PR and it elasticsearch service comes up ok. I will try the manual steps but editing the config after testing the upgrades.

@PrajaktaPurohit What version of elastic is used in the latest version of Chef Infra Server? I'm obviously running on a bit older version of Chef Infra Server so maybe that's why I'm getting different results.

[jasonwbarnett]

@PrajaktaPurohit in parallel I'm going to try to edit /opt/opscode/embedded/cookbooks/private-chef/templates/default/elasticsearch_jvm.opts.erb and then run chef-server-ctl reconfigure followed by chef-server-ctl restart to see if that yields success.

[jasonwbarnett]

@PrajaktaPurohit it worked! No idea why that's different, but it works!

/opt/opscode/embedded/open-jre//bin/java -Xmx3467m -Xms3467m -XX:NewSize=216M -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/var/opt/opscode/elasticsearch/tmp -XX:+HeapDumpOnOutOfMemoryError -Dlog4j2.formatMsgNoLookups=true -Des.path.home=/opt/opscode/embedded/elasticsearch -Des.path.conf=/var/opt/opscode/elasticsearch/config -Des.distribution.flavor=oss -Des.distribution.type=tar -cp /opt/opscode/embedded/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch

[PrajaktaPurohit]

14.9.23 and the latest 14.15.11 on chef-server uses elasticsearch 6.8.18 (internally)

[PrajaktaPurohit]

@PrajaktaPurohit in parallel I'm going to try to edit /opt/opscode/embedded/cookbooks/private-chef/templates/default/elasticsearch_jvm.opts.erb and then run chef-server-ctl reconfigure followed by chef-server-ctl restart to see if that yields success.

yep - that would be the path I would test out to see if the config works! Glad to hear that is working! Thank you for checking!

[tas50]

Closing this out since we went to Elasticsearch 6.8.21 which does the same thing out of the box