Container failing to initialize when storage exists

[jamesrenaud]

Hey folks,

I'm using an NFS drive to persist data for a Nexus 3 container. When the container starts, there is an existing blobs directory and configuration in place. As the container starts, it fails with the following:

2017-07-05 15:07:41,933+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Quartz Scheduler v2.2.2 2017-07-05 15:07:42,454+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Backup' [db.backup] : state=WAITING 2017-07-05 15:07:42,543+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] : state=WAITING 2017-07-05 15:07:42,557+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Rebuild' [repository.rebuild-index] : state=WAITING 2017-07-05 15:07:42,684+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl - Failed transition: NEW -> STARTED org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: null, Unable to initialize blob store directory structure: default at org.sonatype.nexus.blobstore.file.internal.FileBlobStore.init(FileBlobStore.java:584) [na:na] at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.newBlobStore(BlobStoreManagerImpl.java:201) [na:na] at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.doStart(BlobStoreManagerImpl.java:102) [na:na] at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.doStart(RepositoryManagerImpl.java:208) [na:na] at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:153) [org.sonatype.nexus.extender:3.3.2.02] at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:93) [org.sonatype.nexus.extender:3.3.2.02] at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:188) [org.sonatype.nexus.extender:3.3.2.02] at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429) [org.apache.felix.framework-5.6.1.jar:na] at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [org.apache.felix.framework-5.6.1.jar:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] Caused by: java.nio.file.AccessDeniedException: /nexus-data/blobs/default at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) [na:1.8.0_131] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) [na:1.8.0_131] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) [na:1.8.0_131] at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) [na:1.8.0_131] at java.nio.file.Files.createDirectory(Files.java:674) [na:1.8.0_131] at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) [na:1.8.0_131] at java.nio.file.Files.createDirectories(Files.java:767) [na:1.8.0_131] at org.sonatype.nexus.common.io.DirectoryHelper.mkdir(DirectoryHelper.java:143) [na:na] at org.sonatype.nexus.blobstore.file.internal.FileBlobStore.init(FileBlobStore.java:579) [na:na] ... 17 common frames omitted

So far as I can tell it's expecting an empty or clean install default store in the blobs directory, but because I have an existing installation this fails. How is the container supposed to handle an existing persistent data volume?

[staylorx]

I have the same issue but with quieter logs: "Nexus data directory already in use: /nexus-data". It's a clear error but as @JamesrenaudMitel asks how do containers share existing volumes?

[jamesrenaud]

@staylorx I vaguely remember this issue and it may have been a permissions problem on the directory you're mounting to nexus-data. I believe I solved this by doing a CHOWN -R 200:200 /nexus-data

[joedragons]

A data directory should only be used by one NXRM instance at the time for the record. Using it against multiple NXRM instances at a time will cause trouble.

[lopf]

Same here. The use-case is a nexus container running in a Kubernetes Deployment and the node where the pod is running on abruptly dies. A replica count of 1 makes the Deployment to start a new pod but it then crash-loops because of the "lock". The PV is mounted via NFS.

[ericleu]

I am noticing the same issue in nexus3 (3.8.0) as well; I am using docker container image, not sure if that makes a diff. I found a work around to bring nexus back by removing /nexus-data/lock and /nexus-data/tmp/*. Since this is a POC, repos are still empty. So I am not sure what the actual effect is when there is live data.

[chtompki]

I've also seen this particularly with regards to having an efs system symlinked into /nexus-data/blobs. We get exceptions that the file system is already created. Must the java process create the filesystem for things to be valid?

Edit. I've found that the best plan is to chown -R 200:200 the efs file system and volume mount it into the container separately. After the container is running is when the chown need take place.