"Cannot remove extraneous deployment marker file" leads to strange behavior

[lmsurpre]

I enabled the experimental testcontainers "withReuse" option and, after my tests completed, I went to check out the still-running container.

What I found is general flakiness. For example, after login, it would say the admin console page doesn't exist / couldn't be found. Then, after a refresh, it would work pretty well. Occasionally I would see an intermittent error on the UI, but it was still usage.

I dug into the logs to look for errors and I found the following output was repeated over and over again, basically drowning everything else out:

22:27:55,896 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0027: Starting deployment of "-495401560-extensions.jar" (runtime-name: "-495401560-extensions.jar")
22:27:55,971 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-3) WFLYSRV0018: Deployment "deployment.-495401560-extensions.jar" is using a private module ("org.keycloak.keycloak-services") which may be changed or removed in future versions without notice.
22:27:55,976 INFO  [org.keycloak.subsystem.server.extension.KeycloakProviderDeploymentProcessor] (MSC service thread 1-3) Deploying Keycloak provider: -495401560-extensions.jar
22:27:55,990 WARN  [org.keycloak.services] (MSC service thread 1-3) KC-SERVICES0047: audience-validator (org.alvearie.keycloak.AudienceValidatorFactory) is implementing the internal SPI authenticator. This SPI is internal and may change without notice
22:27:55,991 WARN  [org.keycloak.services] (MSC service thread 1-3) KC-SERVICES0047: auth-select-patient (org.alvearie.keycloak.PatientSelectionFormFactory) is implementing the internal SPI authenticator. This SPI is internal and may change without notice
22:27:56,063 WARN  [org.keycloak.services] (MSC service thread 1-3) KC-SERVICES0047: oidc-patient-prefix-usermodel-attribute-mapper (org.alvearie.keycloak.PatientPrefixUserAttributeMapper) is implementing the internal SPI protocol-mapper. This SPI is internal and may change without notice
22:27:56,063 WARN  [org.keycloak.services] (MSC service thread 1-3) KC-SERVICES0047: oidc-usermodel-attribute-mapper-with-token-response-support (org.alvearie.keycloak.UserAttributeMapper) is implementing the internal SPI protocol-mapper. This SPI is internal and may change without notice
22:27:56,199 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0016: Replaced deployment "-495401560-extensions.jar" with deployment "-495401560-extensions.jar"
22:27:56,200 WARN  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) WFLYDS0002: Cannot remove extraneous deployment marker file /opt/jboss/keycloak/standalone/deployments/-495401560-extensions.jar.dodeploy
22:28:01,225 INFO  [org.keycloak.subsystem.server.extension.KeycloakProviderDeploymentProcessor] (MSC service thread 1-6) Undeploying Keycloak provider: -495401560-extensions.jar
22:28:01,245 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0028: Stopped deployment -495401560-extensions.jar (runtime-name: -495401560-extensions.jar) in 23ms
22:28:01,246 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0027: Starting deployment of "-495401560-extensions.jar" (runtime-name: "-495401560-extensions.jar")
22:28:01,318 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-6) WFLYSRV0018: Deployment "deployment.-495401560-extensions.jar" is using a private module ("org.keycloak.keycloak-services") which may be changed or removed in future versions without notice.
22:28:01,323 INFO  [org.keycloak.subsystem.server.extension.KeycloakProviderDeploymentProcessor] (MSC service thread 1-6) Deploying Keycloak provider: -495401560-extensions.jar
22:28:01,336 WARN  [org.keycloak.services] (MSC service thread 1-6) KC-SERVICES0047: audience-validator (org.alvearie.keycloak.AudienceValidatorFactory) is implementing the internal SPI authenticator. This SPI is internal and may change without notice
22:28:01,336 WARN  [org.keycloak.services] (MSC service thread 1-6) KC-SERVICES0047: auth-select-patient (org.alvearie.keycloak.PatientSelectionFormFactory) is implementing the internal SPI authenticator. This SPI is internal and may change without notice
22:28:01,396 WARN  [org.keycloak.services] (MSC service thread 1-6) KC-SERVICES0047: oidc-patient-prefix-usermodel-attribute-mapper (org.alvearie.keycloak.PatientPrefixUserAttributeMapper) is implementing the internal SPI protocol-mapper. This SPI is internal and may change without notice
22:28:01,397 WARN  [org.keycloak.services] (MSC service thread 1-6) KC-SERVICES0047: oidc-usermodel-attribute-mapper-with-token-response-support (org.alvearie.keycloak.UserAttributeMapper) is implementing the internal SPI protocol-mapper. This SPI is internal and may change without notice
22:28:01,488 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0016: Replaced deployment "-495401560-extensions.jar" with deployment "-495401560-extensions.jar"
22:28:01,491 WARN  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) WFLYDS0002: Cannot remove extraneous deployment marker file /opt/jboss/keycloak/standalone/deployments/-495401560-extensions.jar.dodeploy
...

I'm not sure if the constantly reloading is a feature or a bug, but when I dug in I found that the reason it cannot delete the deployment marker file is that this file is a direct mount point.

When I fixed it locally, by mounting a directory instead of just the file, the container started to behave perfectly...no more wonkiness.

[dasniko]

Thanks @lmsurpre for reporting, I'll have a look into this.

@thomasdarimont can you pls also have a look at it, as this extension was contributed by you. Thanks.

[thomasdarimont]

@dasniko I think I have a sound fix for this issue now, would you mind having a look? :)

[dasniko]

Thanks @thomasdarimont and @lmsurpre for figuring out and having a solution which works on all tested platforms.
I'll review and merge the next hours/days.