Application install may deadlock in cluster

[rymsha]

regression of #7832

qtp1968303782-7609 id=7609 state=WAITING
    - waiting on <0x1e54de6c> (a java.util.concurrent.CountDownLatch$Sync)
    - locked <0x1e54de6c> (a java.util.concurrent.CountDownLatch$Sync)
    at java.base@11.0.6/jdk.internal.misc.Unsafe.park(Native Method)
    at java.base@11.0.6/java.util.concurrent.locks.LockSupport.park(Unknown Source)
    at java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
    at java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown Source)
    at java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source)
    at java.base@11.0.6/java.util.concurrent.CountDownLatch.await(Unknown Source)
    at com.enonic.xp.core.impl.app.ApplicationRegistry$ApplicationWrapper.getConfig(ApplicationRegistry.java:311)
    at com.enonic.xp.script.impl.function.ApplicationInfoBuilder.buildConfig(ApplicationInfoBuilder.java:39)
    at com.enonic.xp.script.impl.function.ApplicationInfoBuilder.build(ApplicationInfoBuilder.java:32)
    at com.enonic.xp.script.impl.executor.ScriptExecutorImpl.buildAppInfo(ScriptExecutorImpl.java:123)
    at com.enonic.xp.script.impl.executor.ScriptExecutorImpl.initialize(ScriptExecutorImpl.java:114)
    at com.enonic.xp.script.impl.executor.ScriptExecutorManager.createExecutor(ScriptExecutorManager.java:53)
    at com.enonic.xp.script.impl.executor.ScriptExecutorManager$$Lambda$296/0x000000010095a840.apply(Unknown Source)

qtp1968303782-7658 id=7658 state=BLOCKED
    - waiting to lock <0x36b021ca> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
     owned by qtp1968303782-7609 id=7609
    at java.base@11.0.6/java.util.concurrent.ConcurrentHashMap.replaceNode(Unknown Source)
    at java.base@11.0.6/java.util.concurrent.ConcurrentHashMap.remove(Unknown Source)
    at com.enonic.xp.script.impl.executor.ScriptExecutorManager.invalidate(ScriptExecutorManager.java:59)
    at com.enonic.xp.script.impl.standard.ScriptRuntimeImpl.invalidate(ScriptRuntimeImpl.java:42)
    at com.enonic.xp.script.impl.standard.ScriptRuntimeFactoryImpl.lambda$invalidate$0(ScriptRuntimeFactoryImpl.java:54)
    at com.enonic.xp.script.impl.standard.ScriptRuntimeFactoryImpl$$Lambda$442/0x0000000100bcd040.accept(Unknown Source)
    at java.base@11.0.6/java.util.concurrent.CopyOnWriteArrayList.forEach(Unknown Source)
    at com.enonic.xp.script.impl.standard.ScriptRuntimeFactoryImpl.invalidate(ScriptRuntimeFactoryImpl.java:54)
    at com.enonic.xp.script.impl.ScriptRuntimeFactoryDelegate.invalidate(ScriptRuntimeFactoryDelegate.java:55)
    at com.enonic.xp.core.impl.app.ApplicationRegistry.lambda$callInvalidators$3(ApplicationRegistry.java:138)
    at com.enonic.xp.core.impl.app.ApplicationRegistry$$Lambda$437/0x0000000100bcbc40.accept(Unknown Source)
    at java.base@11.0.6/java.util.concurrent.CopyOnWriteArrayList.forEach(Unknown Source)
    at com.enonic.xp.core.impl.app.ApplicationRegistry.callInvalidators(ApplicationRegistry.java:135)
    at com.enonic.xp.core.impl.app.ApplicationRegistry.lambda$invalidate$0(ApplicationRegistry.java:69)
    at com.enonic.xp.core.impl.app.ApplicationRegistry$$Lambda$436/0x0000000100bcb840.apply(Unknown Source)
    at java.base@11.0.6/java.util.concurrent.ConcurrentHashMap.computeIfPresent(Unknown Source)
    at com.enonic.xp.core.impl.app.ApplicationRegistry.invalidate(ApplicationRegistry.java:68)
    at com.enonic.xp.core.impl.app.ApplicationServiceImpl.handleUpdate(ApplicationServiceImpl.java:564)
    at com.enonic.xp.core.impl.app.ApplicationServiceImpl.installOrUpdateApplication(ApplicationServiceImpl.java:455)
    at com.enonic.xp.core.impl.app.ApplicationServiceImpl.doInstallGlobalApplication(ApplicationServiceImpl.java:140)
    at com.enonic.xp.core.impl.app.ApplicationServiceImpl.lambda$installGlobalApplication$3(ApplicationServiceImpl.java:121)
    at com.enonic.xp.core.impl.app.ApplicationServiceImpl$$Lambda$917/0x0000000101497840.call(Unknown Source)
    at com.enonic.xp.context.ContextImpl.callWith(ContextImpl.java:102)
    at com.enonic.xp.core.impl.app.ApplicationHelper.callWithContext(ApplicationHelper.java:148)
    at com.enonic.xp.core.impl.app.ApplicationServiceImpl.installGlobalApplication(ApplicationServiceImpl.java:118)
    at com.enonic.xp.admin.impl.rest.resource.application.ApplicationResource.installApplication(ApplicationResource.java:441)
    at com.enonic.xp.admin.impl.rest.resource.application.ApplicationResource.lambda$install$7(ApplicationResource.java:327)
    at com.enonic.xp.admin.impl.rest.resource.application.ApplicationResource$$Lambda$916/0x0000000101490440.call(Unknown Source)
    at com.enonic.xp.admin.impl.rest.resource.application.ApplicationResource.lock(ApplicationResource.java:603)
    at com.enonic.xp.admin.impl.rest.resource.application.ApplicationResource.install(ApplicationResource.java:327)

[rymsha]

For some reason ApplicationRegistry.setConfiguration never got called.

Spec says that it is always called on ManagedService https://osgi.org/specification/osgi.cmpn/7.0.0/service.cm.html#d0e10591
So, it is still a mystery, why it wasn't.

[rymsha]

As a quickfix I will add 10seconds timeout for config resolution. Application can then start (but won't be configured).

[sigdestad]

Hmm, isn't this already configurable though a parameter? We could use different standard values for SDK and server distro?

[rymsha]

It is unrelated. you refer to configuration files polling interval.
Issue is about a problem when Felix doesn't follow OSGi specifications https://osgi.org/specification/osgi.cmpn/7.0.0/service.cm.html#d0e10591 and how to deal with it.