Log4j don't rollover nor delete files after the introduction of log per pipeline

[andsel]

Logstash information:

Please include the following information:

1. Logstash version: >= 7.5.0
2. Logstash installation source: tar/deb
3. How is Logstash being run: command line

Plugins installed: (bin/logstash-plugin list --verbose)

JVM (e.g. java -version):

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

1. JVM version (java -version)

openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.10+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.10+9, mixed mode)

2. JVM installation source: sdkman!
3. Value of the JAVA_HOME environment variable if set. /home/andrea/.sdkman/candidates/java/current

OS version (uname -a if on a Unix-like system):

Linux kalispera 5.3.0-64-generic #58-Ubuntu SMP Fri Jul 10 19:33:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
When logstash-plain.log file reached the size configured in log4j2.properties the file has to rollover, and when the rolling file appender contains a strategy that defines an action (for example delete all file older that 1 minute) it has to execute the action.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1.configure pipeline with

input {
  java_generator {
    lines => [
      "line 1",
      "line 2",
      "line 3"
    ]
    eps => 1
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

2. change log4 configuration to rollover at 1Kb, no time related triggering policy and a strategy to require deletion of log files older than 1 minute (config/log4j2.properties):

appender.rolling.type = RollingFile
appender.rolling.name = plain_rolling
appender.rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
appender.rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}-%i.log.gz
appender.rolling.policies.type = Policies
#appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
#appender.rolling.policies.time.interval = 1
#appender.rolling.policies.time.modulate = true
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]} %m%n
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size = 1KB
####
appender.rolling.strategy.delete.type = Delete
appender.rolling.strategy.delete.basePath = logs/
appender.rolling.strategy.delete.maxDepth = 1
appender.rolling.strategy.delete.ifLastModified.type = IfLastModified
appender.rolling.strategy.delete.ifLastModified.age = 1M
####
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 30

3. run logstash in debug to produce some log traffic bin/logstash -f pipeline.conf --debug and verify inlogs/ folder that the logstash-plain.log becomes bigger than 1Kb and never rollover nor the rolled over files are deleted after 1 minute

Provide logs (if relevant):

• logs/ content with version 7.4.2

andrea@kalispera logs $ date
Fri 21 May 2021 09:45:33 AM CEST
andrea@kalispera logs $ ls -lart
total 28
drwxr-xr-x 13 andrea andrea 4096 May 21 09:45 ..
-rw-r--r--  1 andrea andrea  251 May 21 09:45 logstash-plain-2021-05-21-18.log.gz
-rw-r--r--  1 andrea andrea  250 May 21 09:46 logstash-plain-2021-05-21-19.log.gz
-rw-r--r--  1 andrea andrea  246 May 21 09:46 logstash-plain-2021-05-21-20.log.gz
-rw-r--r--  1 andrea andrea  249 May 21 09:46 logstash-plain-2021-05-21-21.log.gz
drwxr-xr-x  2 andrea andrea 4096 May 21 09:46 .
-rw-r--r--  1 andrea andrea  768 May 21 09:46 logstash-plain.log
andrea@kalispera logs $ ls -lart
total 28
drwxr-xr-x 13 andrea andrea 4096 May 21 09:45 ..
-rw-r--r--  1 andrea andrea  246 May 21 09:46 logstash-plain-2021-05-21-20.log.gz
-rw-r--r--  1 andrea andrea  249 May 21 09:46 logstash-plain-2021-05-21-21.log.gz
-rw-r--r--  1 andrea andrea  244 May 21 09:46 logstash-plain-2021-05-21-22.log.gz
-rw-r--r--  1 andrea andrea  246 May 21 09:47 logstash-plain-2021-05-21-23.log.gz
drwxr-xr-x  2 andrea andrea 4096 May 21 09:47 .
-rw-r--r--  1 andrea andrea  225 May 21 09:47 logstash-plain.log

• logs/ content with version 7.5.1

andrea@kalispera logs $ ls -lart
total 32
drwxr-xr-x 13 andrea andrea  4096 May 21 09:48 ..
-rw-r--r--  1 andrea andrea     0 May 21 09:48 logstash-slowlog-plain.log
drwxr-xr-x  2 andrea andrea  4096 May 21 09:48 .
-rw-r--r--  1 andrea andrea 22590 May 21 09:48 logstash-plain.log
andrea@kalispera logs $ ls -lart
total 44
drwxr-xr-x 13 andrea andrea  4096 May 21 09:48 ..
-rw-r--r--  1 andrea andrea     0 May 21 09:48 logstash-slowlog-plain.log
drwxr-xr-x  2 andrea andrea  4096 May 21 09:48 .
-rw-r--r--  1 andrea andrea 33844 May 21 09:51 logstash-plain.log

the logstash-plain.log keeps growing.

An heavy suspect it that's related to #11108