EJB persistent timers that were deferred during app start do not run when app finishes starting

[njr-11]

Describe the bug
EJB Persistent Timers that get deferred due the application not being started yet do not run after the application finishes starting.

Steps to Reproduce
Have a large, slowly-starting application with a number of persistent EJB timers around from a previous run. Start the server and if timing is right, the timers will never start running.

Expected behavior
EJB persistent timers that were deferred due to an unavailable application should run after the application becomes available again.

Diagnostic information:

• OpenLiberty Version: unknown - 22.0.0.7
• Affected feature(s) ejbPersistentTimer-3.2, enterpriseBeansPersistentTimer-4.0
• Java Version: Java(TM) SE Runtime Environment (8.0.7.5 - pmz6480sr7fp5-20220208_01(SR7 FP5))

Additional context

Timers are first put into a queue because they attempt to start while the server config is still being processed,

[6/30/22 11:58:47:077 PDT] 00000045 id=3f305e12 com.ibm.ws.concurrent.persistent.internal.InvokerTask        > run[1] Entry  
                                                                                                               com.ibm.ws.concurrent.persistent.internal.PersistentExecutorImpl@b07b2010
[6/30/22 11:58:47:078 PDT] 00000045 id=b07b2010 ibm.ws.concurrent.persistent.internal.PersistentExecutorImpl 3 execution deferred while config update in progress
[6/30/22 11:58:47:078 PDT] 00000045 id=3f305e12 com.ibm.ws.concurrent.persistent.internal.InvokerTask        < run[1] Exit  
                                                                                                               attempted to run during a configuration update.

When the config update finishes, they are resubmitted,

[6/30/22 11:58:58:282 PDT] 00000055 id=b07b2010 ibm.ws.concurrent.persistent.internal.PersistentExecutorImpl > configUpdateCompleted Entry  
                                                                                                               ApplicationsInstallCalled
[6/30/22 11:58:58:283 PDT] 00000055 id=b07b2010 ibm.ws.concurrent.persistent.internal.PersistentExecutorImpl < configUpdateCompleted Exit  
                                                                                                               1
[6/30/22 11:58:58:283 PDT] 00000055 id=aafae42d istent.internal.PersistentExecutorImpl$1MyCompletionListener < successfulCompletion Exit 

[6/30/22 11:58:58:283 PDT] 00000045 id=3f305e12 com.ibm.ws.concurrent.persistent.internal.InvokerTask        > run[1] Entry  
                                                                                                               com.ibm.ws.concurrent.persistent.internal.PersistentExecutorImpl@b07b2010

Unfortunately, the application still isn't started at this point, and the timers are once again deferred,

[6/30/22 11:59:01:982 PDT] 00000045 id=3f305e12 com.ibm.ws.concurrent.persistent.internal.InvokerTask        3 classloader identifier 
                                                                                                               WebModule:db2zai#DB2ZAI.1.5.0.0.jar
[6/30/22 11:59:01:982 PDT] 00000045 id=3f305e12 com.ibm.ws.concurrent.persistent.internal.InvokerTask        3 classloader 
                                                                                                               com.ibm.ws.classloading.internal.ThreadContextClassLoader@3a8b44e8

[6/30/22 11:59:01:982 PDT] 00000045 id=bccdf591 com.ibm.ws.concurrent.persistent.internal.ApplicationTracker > isStarted Entry  
                                                                                                               db2zai
[6/30/22 11:59:01:983 PDT] 00000045 id=bccdf591 com.ibm.ws.concurrent.persistent.internal.ApplicationTracker < isStarted Exit  
                                                                                                               false

[6/30/22 11:59:01:983 PDT] 00000045 id=3f305e12 com.ibm.ws.concurrent.persistent.internal.InvokerTask        < run[1] Exit  
                                                                                                               unavailable - deferred

Eventually the application starts, and the timers should be reattempted after this, but they aren't,

[6/30/22 11:59:03:265 PDT] 00000061 id=00000000 com.ibm.ws.app.manager.AppMessageHelper                      A CWWKZ0001I: Application db2zai started in 48.273 seconds.

[6/30/22 11:59:03:267 PDT] 0000004d id=bccdf591 com.ibm.ws.concurrent.persistent.internal.ApplicationTracker > addStartedApplication Entry  
                                                                                                               {com.ibm.wsspi.application.Application}={location=/u/home/DB2ZAI/db2zaiv1r3.4050/DB2Z-ML-Akira/servers/akira/apps/DB2ZAI.1.5.0.0.jar, service.id=520, service.bundleid=63, service.scope=singleton, application.state=STARTED, config.overrides=true, name=db2zai, config.id=com.ibm.ws.app.manager[db2zai], context-root=/db2zai, config.source=file, service.factoryPid=com.ibm.ws.app.manager, autoStart=true, id=db2zai, config.displayId=application[db2zai], service.pid=com.ibm.ws.app.manager_66, classloader=com.ibm.ws.classloading.classloader_117, type=war}
[6/30/22 11:59:03:267 PDT] 0000004d id=bccdf591 com.ibm.ws.concurrent.persistent.internal.ApplicationTracker < addStartedApplication Exit 

A good guess is that ApplicationTracker.addStartedApplication doesn't see any timers to run because ApplicationTracker.deferTask wasn't ever invoked for the tasks. There is no evidence in the trace of deferTask being invoked.

I only see one place in code where deferTask can be invoked,

                    } else if (ownerForDeferredTask != null)
                        appTracker.deferTask(this, ownerForDeferredTask, persistentExecutor);

It requires a non-null ownerForDeferredTask.

I see 2 places (both in the same method) where ownerForDeferredTask is assigned,

                    if (!appTracker.isStarted(owner)) {
                        ownerForDeferredTask = owner;
                        if (trace && tc.isEntryEnabled())
                            Tr.exit(this, tc, "run[" + taskId + ']', "unavailable - deferred");
                        return; // Ignore, we are deferring the task because the application or module is unavailable

            if (loader == null || !appTracker.isStarted(owner)) {
                ownerForDeferredTask = owner;
                if (trace && tc.isEntryEnabled())
                    Tr.exit(this, tc, "run[" + taskId + ']', "unavailable - deferred");
                return; // Ignore, we are deferring the task because the application or module is unavailable
            }

Notice the return statment from both of those if blocks. It prevents code flow from continuing, meaning that appTracker.deferTask is unreachable.