Possible performance degradation due to log4j cve 2021 44228 hotpatch service on Amazon Linux 2
Starting Friday, 2021-12-17, JDKs shipped in Amazon Linux automatically install a patcher agent (log4j-cve-2021-44228-hotpatch) that runs in the background and takes care of hot-fixing running Java application that are affected by log4j cve 2021-44228.
We have identified that some HPC applications and workloads can suffer performance degradation when executed on compute nodes that have such patcher agent running in the background.
In Amazon Linux 2 clusters deployed with ParallelCluster 2 (versions <= 2.11.3) and ParallelCluster 3 (versions < 3.1.0) such service is installed at node bootstrap time as part of the cloud-init automatic upgrades logic.
log4j-cve-2021-44228-hotpatch package is being installed as a dependency of java-1.8.0-openjdk-headless and java-17-amazon-corretto-headless packages, that are both installed in ParallelCluster AMIs as dependencies of the package R. ParallelCluster does not run any Java application on cluster nodes and does not require such patching service to be running in the background. However users might be running Java applications that are affected by the mentioned vulnerability.
In order to verify if the patcher service is running you can run sudo systemctl status log4j-cve-2021-44228-hotpatch and check if the status is running.
In ParallelCluster version 2.11.4, cloud-init is not performing automatic upgrades at boot, hence the patching service is not installed and enabled by default at instance provisioning time. Note that in case an explicit upgrade of the jdk packages is performed, the patcher agent will be installed and started automatically on the cluster nodes.
Starting from ParallelCluster 2.11.5 and 3.1.0 we are going to disable the log4j patcher agent by default. In case your application requires log4j patching we highly recommend to address the vulnerability by upgrading the log4j library used in your application. If you want to re-enable the log4j patcher agent, please run the following commands on cluster nodes:
sudo systemctl unmask log4j-cve-2021-44228-hotpatch
sudo systemctl enable log4j-cve-2021-44228-hotpatch
sudo systemctl start log4j-cve-2021-44228-hotpatch
In case you don't run Java applications in your cluster or you already patched your application and you don't need the hot patching functionality provided by log4j-cve-2021-44228-hotpatch, you can run the following commands to disable the service on cluster nodes:
sudo systemctl mask log4j-cve-2021-44228-hotpatch
sudo systemctl stop log4j-cve-2021-44228-hotpatch
The execution of such commands can be automated as part of a custom bootstrap action and needs to be ran on every compute node.
In case you want to remove all Java dependencies from ParallelCluster AMI you can run the following commands:
sudo yum install yum-plugin-remove-with-leaves
sudo yum remove R --remove-leaves
Please note that additional packages installed by the user might depend on JDK. In that case it won’t be enough to remove the R package in order to remove all Java dependencies from the AMI.
The execution of such commands can be automated as part of a custom bootstrap action or as part of a custom AMI creation.