error on first time deploy of chart ISPN000512 Cannot acquire lock /opt/infinispan/server/data/___global.lck

[thetoolsmith]

I'm running into this error when deploying this helm chart to eks and not really sure what's going on. This is the first attempt at deploying this.
Our EKS cluster is 1.26

ISPN000512 Cannot acquire lock /opt/infinispan/server/data/___global.lck

install command used:
helm install infinispan-server . -n token-cache

No values overrides being used.

Full stack trace:

2023-11-13 21:13:23,197 ERROR (main) [org.infinispan.CONFIG] ISPN000660: DefaultCacheManager start failed, stopping any running components org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
	at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:88)
	at org.infinispan.globalstate.impl.GlobalStateManagerImpl.start(GlobalStateManagerImpl.java:65)
	at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:34)
	at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:27)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:616)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:607)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:576)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:807)
	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:379)
	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:252)
	at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:779)
	at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:747)
	at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:68)
	at org.infinispan.security.Security.doPrivileged(Security.java:56)
	at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:40)
	at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:71)
	at org.infinispan.server.Server.run(Server.java:417)
	at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:173)
	at org.infinispan.server.tool.Main.run(Main.java:98)
	at org.infinispan.server.Bootstrap.main(Bootstrap.java:56)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.infinispan.server.loader.Loader.run(Loader.java:106)
	at org.infinispan.server.loader.Loader.main(Loader.java:51)
Caused by: java.io.FileNotFoundException: /opt/infinispan/server/data/___global.lck (Permission denied)
	at java.base/java.io.FileOutputStream.open0(Native Method)
	at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)
	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:235)
	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:184)
	at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:82)
	... 25 more

2023-11-13 21:13:23,198 WARN  (main) [org.infinispan.CONTAINER] ISPN000574: Global state cannot persisted because it is incomplete (usually caused by errors at startup).
2023-11-13 21:13:23,384 FATAL (main) [org.infinispan.SERVER] ISPN080028: Infinispan Server failed to start org.infinispan.manager.EmbeddedCacheManagerStartupException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state

[rjl79]

I happened to have the exact same problem this morning. It seems that the persistent volume needs to be updated to allow it be writable.

Setting deploy.makeDataDirWritable: true in the values of the chart should solve this.

[ryanemerson]

This also affects our operator infinispan/infinispan-operator#392

Unfortunately we only have the workaround suggested by @rjl79. It would be great to have a better solution for this that didn't require an explicit workaround, but so far investigating this has been a low priority. Any suggestions are very welcome 🙂

[thetoolsmith]

Thanks for the workaround @rjl79, although I cannot seem to get that to work.
I changed makeDataDirWritable: true in values.yaml and got Init:CreateContainerConfigError. I also tried this by modified my install command as such helm install infinispan-server . --set deploy.makeDataDirWritable=true -n token-cache and got Init:CreateContainerConfigError as well.

Error: container has runAsNonRoot and image will run as root (pod: "infinispan-server-0_token-service-cache-playground(1177de15-91ca-4ae6-97e0-92692a877e27)", container: data-chmod-pv)

Does this need to run as Root? I don't see anything in values.yaml that I can change related to this.

[rigazilla]

another workaround could be to explicitly set the fs group on the pv. See here

[thetoolsmith]

@rigazilla Sorry not quite following how that changes anything. Our situation is we are deploying to a namespace without privileged pod security. The initContainer data-chmod-pv seems like what is requiring elevated permissions and terminates when it's done. Leaving the infinispan container running as the jboss user. From what the template blocks says, the initContainer data-chmod-pv only runs when we set deploy.makeDataDirWriteable to true which is to fix the above issue error ISPN000512 Cannot acquire lock /opt/infinispan/server/data/___global.lck. Taking all this in seems to make me think we just need a way to set the perms and mount that /opt/infinispan/server/data other than using the initContainer. I added the fsGroup as in the link you provided, but it didn't change anything on our end when turning off privileged access in the namespace. I haven't had much exposure to helm charts, so maybe I'm missing something.

[rigazilla]

@thetoolsmith , you have security issue or the solution doesn't work?

I didn't specify but you don't need the deploy.makeDataDirWriteable: true change for this; with fsGroup kubernetes should set correctly the group on the file system for you (see here)

[thetoolsmith]

Thanks @rigazilla . I updated the config and the containers are starting up now without the need to set privileged on the namespace.

[rigazilla]

possibly we could expose the fsgroup workaround via a flag in the values.yaml file

[ryanemerson]

+1 to allowing this to be configurable. How about allowing a user to specify the entire container securityContext via deploy.container.securityContext?