Rolling problem -Dlog4j.configurationFile with two log4j2.xml files

[mariusv5]

Description

I use the Apache Tomcat.
We have configured the log4j2 Java Options Arguments: -Dlog4j.configurationFile={/path1}/log4j2.xml,{/path2}/log4j2.xml.
The two files are related one for Web Application and the other for WebServices (so the JDK is the same), on startup the logs collected are:

ll wss_*
-rw-r----- 1 root mf 0 1 feb 12.55 wss_alarms.log
-rw-r----- 1 root mf 5368 1 feb 12.55 wss_axis.log
-rw-r----- 1 root mf 1082 1 feb 12.55 wss_database.log
-rw-r----- 1 root mf 123172 1 feb 13.12 wss_main.log
-rw-r----- 1 root mf 0 1 feb 12.55 wss_warrants.log
-rw-r----- 1 root mf 0 1 feb 12.55 wss_wfa.log

ll webapp_*
-rw-r----- 1 root mf 8172 1 feb 13.13 webapp_actions.log
-rw-r----- 1 root mf 176 1 feb 12.55 webapp_database.log
-rw-r----- 1 root mf 0 1 feb 12.55 webapp_grids.log
-rw-r----- 1 root mf 0 1 feb 12.55 webapp_json.log
-rw-r----- 1 root mf 4796 1 feb 12.55 webapp_struts.log

I use the DefaultRolloverStrategy max="5", with the number of retention days, below is an example:

		<RollingFile name="RollingAlarms" fileName="${log-path-wss}/${service-suffix-wss}_alarms.log" append="true" filePattern="${log-path-wss}/${service-suffix-wss}_alarms-%d{yyyy-MM-dd}-%i.log">
			<PatternLayout pattern="%d %5p (%F:%L) %m%n" />

			<Policies>
				<OnStartupTriggeringPolicy />
				<SizeBasedTriggeringPolicy size="50 MB" />
				<CronTriggeringPolicy schedule="0 0 0 * * ?" />
			</Policies>

			<DefaultRolloverStrategy max="${retention-days-wss}">
				<Delete basePath="${log-path-wss}" maxDepth="1">
					<IfFileName glob="${service-suffix-wss}_alarms-*.log" />
					<IfLastModified age="${retention-days-wss}d" />
				</Delete>
			</DefaultRolloverStrategy>
		</RollingFile>

The problem is that in after the file rolling, no more log are produced and the rotated file stays at zero length:

-rw-r--r-- 1 root mf 92521 31 gen 23.59 wss_wfa-2023-01-31-1.log
-rw-r--r-- 1 root mf 0 1 feb 00.00 wss_wfa.log

Configuration

Operating system:

• Red Hat Enterprise Linux Server release 7.2

Application server:

• apache-tomcat-8.5.9

Log4j2 Version:

• log4j-api-2.17.1.jar
• log4j-core-2.17.1.jar

JDK:

• JDK 1.8.0_301

Logs

No logs

Reproduction

Configure tomcat with two different log4j2.xml file and different attributes, when log4j start rolling, the lines log not add in the new file and no more log are produced.

[rgoers]

What do you see with -Dlog4j2.debug=true set? Can you please provide those logs

[mariusv5]

In attach the logs retrieved after Dlog4j2.debug=true setting.
catalina.log

[ppkarwasz]

@mariusv5,

Your problem is caused by multiple log4j-api and log4j-core jars being loaded into your Tomcat system (your logs show 3 instances of ClassLoaderContextSelector, one for each web application). Each one opens a file descriptor to your log file and each one tries to rotate it. As you can imagine, this does not end well.

Log4j2 is perfectly capable of sharing the same log file between logging context, if a single copy of log4j-core is deployed in your system. Technically each logging context will then have its own copy of RollingFileAppender, but the engine that opens files and performs the rotation (RollingFileManager) will be shared.

Unlike full application servers such as Wildfly, Tomcat has not easy way to share a single library between web applications. In order to do that you need to install a set of Log4j2 artifacts in the $CATALINA_BASE/lib folder and do one of the following:

1. Remove the log4j-* artifacts from your web applications (personally I don't like the idea of modifying web applications),
2. Switch the delegation model of all web application classloaders to "parent first" (this might break some apps if they need different versions of the same library), by adding:

   <Loader delegate="true"/>
   

   to their context configuration (e.g. in $CATALINA_BASE/conf/context.xml). Cf. Tomcat documentation.
3. Use a custom classloader that looks up org.apache.logging.log4j.* classes in the parent classloader first, but uses the usual delegation for the other classes. This is basically what Tomcat does with org.apache.catalina.* and other internal classes.

You can find such a classloader in my personal repository (cf. Log4jWebappClassloader). You can find it on Maven Central under the coordinates:

<dependency>
    <groupId>eu.copernik</groupId>
    <artifactId>log4j-tomcat</artifactId>
    <version>2.18.0</version>
</dependency>

To use it, place it in the $CATALINA_BASE/lib folder together with log4j-api and log4j-core and add:

<Loader loaderClass="eu.copernik.log4j.tomcat.Log4jWebappClassLoader"/>

to your $CATALINA_BASE/conf/context.xml file.