Log4j bump to 2.18 due to [LOG4J2-3419]

[cryptoe]

Symptom:

Mappers/Reducers spawned by the 'hadoop_index` tasks on EMR cluster 6.3.xx were running in debug mode.
They were not respecting the args passed via job properties.

Solution:

As Hadoop is still on the old log4j 1.xx jars, there is some incompatibility when both log4j 1.xx and log4j 2.xx jars are present.
log4j 2.17.1 does not support log4j1.x custom levels. Hence it was defaulting to DEBUG.
This PR: https://github.com/apache/logging-log4j2/pull/789​ adds the support for log4j 1.x levels.
Post upgrading the jar to log4j 2.18, the mapper/reducer are picking the correct logger level ie : INFO

DEBUG StatusLogger PluginManager 'Lookup' found xx plugins
DEBUG StatusLogger PluginManager 'Log4j Builder' found xx plugins
DEBUG StatusLogger Parsing for [root] with value=[INFO,CLA,EventCounter].
DEBUG StatusLogger Level token is [INFO].
DEBUG StatusLogger Logger root level set to INFO
DEBUG StatusLogger Parsing appender named "CLA".

How to check if your cluster is affected by this

Add the following properties in your ingestion spec:

{
  "type": "index_hadoop",
  "spec": {
    "dataSchema": "xx",
    "ioConfig": "xx",
    "tuningConfig": {
       "xx": "xx",
       "jobProperties": {
       "mapreduce.map.java.opts": "-server -Dlog4j.debug=true -Dorg.apache.logging.log4j.simplelog.StatusLogger.level=TRACE",,
        "mapreduce.reduce.java.opts": "-server  -Dlog4j.debug=true -Dorg.apache.logging.log4j.simplelog.StatusLogger.level=TRACE"
        }
      }
  }
}

Grab the application id from the peon task logs.

Go to yarn main nodes and do
yarn logs --applicationId xxx

If you see output like :

DEBUG StatusLogger PluginManager 'Log4j Builder' found xx plugins
DEBUG StatusLogger Parsing for [root] with value=[DEBUG,CLA,EventCounter].
DEBUG StatusLogger Level token is [DEBUG].
DEBUG StatusLogger Logger root level set to DEBUG

then your mappers and reducers are running in debug mode.

Another way is to take the flame graph of the mappers/reducers and see if any code path is hitting debug sections.

---

Key changed/added classes in this PR

• pom.xml

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.