Skip to content
This repository has been archived by the owner on Aug 6, 2022. It is now read-only.

Can't create collections on mounted volumes #105

Closed
hgsaldanha opened this issue Feb 8, 2017 · 4 comments
Closed

Can't create collections on mounted volumes #105

hgsaldanha opened this issue Feb 8, 2017 · 4 comments
Labels
question

Comments

@hgsaldanha
Copy link

Hi! Thank you for this great job!
I'm running a solr/docker container using the command:
docker run --name asspa-solr -v /BACKUP/docker_volumes/asspa/solr/cores:/opt/solr/server/solr/mycores -m 1g --network asspa -d -p 8983:8983 -t mpfba/asspa-solr:1.0.0 -cloud -z 172.19.0.2 -m 768m -s /opt/solr/server/solr/mycores

The image used is based on solr:6.4.1 and has only one difference: a COPY command of a configset folder.

Command to create a new collection:
docker exec -it -u solr asspa-solr bin/solr create -c teste2 -d asspa_configs

The result :(

`2017-02-08 19:53:27.243 INFO (qtp575335780-19) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=json and sendToOCPQueue=true
2017-02-08 19:53:27.245 INFO (qtp575335780-19) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=json} status=0 QTime=1
2017-02-08 19:53:27.366 INFO (qtp575335780-17) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=list&wt=json and sendToOCPQueue=true
2017-02-08 19:53:27.366 INFO (qtp575335780-17) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=list&wt=json} status=0 QTime=0
2017-02-08 19:53:27.369 INFO (qtp575335780-13) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&maxShardsPerNode=1&collection.configName=teste2&name=teste2&action=CREATE&numShards=1&wt=json and sendToOCPQueue=true
2017-02-08 19:53:27.386 INFO (OverseerThreadFactory-6-thread-5-processing-n:172.19.0.3:8983_solr) [ ] o.a.s.c.CreateCollectionCmd Create collection teste2
2017-02-08 19:53:27.513 INFO (qtp575335780-15) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=teste2&newCollection=true&name=teste2_shard1_replica1&action=CREATE&numShards=1&collection=teste2&shard=shard1&wt=javabin&version=2
2017-02-08 19:53:27.514 ERROR (qtp575335780-15) [c:teste2 s:shard1 x:teste2_shard1_replica1] o.a.s.c.CoreContainer Error creating core [teste2_shard1_replica1]: SolrCore 'teste2_shard1_replica1' is not available due to init failure: SolrCore 'teste2_shard1_replica1' is not available due to init failure: java.nio.file.AccessDeniedException: /opt/solr/server/solr/mycores/teste2_shard1_replica1
org.apache.solr.common.SolrException: SolrCore 'teste2_shard1_replica1' is not available due to init failure: SolrCore 'teste2_shard1_replica1' is not available due to init failure: java.nio.file.AccessDeniedException: /opt/solr/server/solr/mycores/teste2_shard1_replica1
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1151)
at org.apache.solr.cloud.ZkController.publish(ZkController.java:1198)
at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1372)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:885)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:827)
at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:88)
at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:377)
at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:379)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:165)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: SolrCore 'teste2_shard1_replica1' is not available due to init failure: java.nio.file.AccessDeniedException: /opt/solr/server/solr/mycores/teste2_shard1_replica1
... 39 more
Caused by: org.apache.solr.common.SolrException: java.nio.file.AccessDeniedException: /opt/solr/server/solr/mycores/teste2_shard1_replica1
at org.apache.solr.core.SolrCore.(SolrCore.java:959)
at org.apache.solr.core.SolrCore.(SolrCore.java:823)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890)
... 35 more
Caused by: java.lang.IllegalStateException: java.nio.file.AccessDeniedException: /opt/solr/server/solr/mycores/teste2_shard1_replica1
at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:479)
at org.apache.solr.core.SolrCore.(SolrCore.java:900)
... 37 more
Caused by: java.nio.file.AccessDeniedException: /opt/solr/server/solr/mycores/teste2_shard1_replica1
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at org.apache.lucene.store.FSDirectory.(FSDirectory.java:157)
at org.apache.lucene.store.MMapDirectory.(MMapDirectory.java:159)
at org.apache.lucene.store.MMapDirectory.(MMapDirectory.java:112)
at org.apache.lucene.store.FSDirectory.open(FSDirectory.java:197)
at org.apache.solr.core.NRTCachingDirectoryFactory.create(NRTCachingDirectoryFactory.java:55)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:346)
at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:201)
at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:475)
... 38 more
2017-02-08 19:53:27.519 ERROR (qtp575335780-15) [c:teste2 s:shard1 x:teste2_shard1_replica1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'teste2_shard1_replica1': Unable to create core [teste2_shard1_replica1] Caused by: /opt/solr/server/solr/mycores/teste2_shard1_replica1
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:861)
at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:88)
at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:377)
at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:379)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:165)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unable to create core [teste2_shard1_replica1]
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:903)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:827)
... 34 more
Caused by: org.apache.solr.common.SolrException: SolrCore 'teste2_shard1_replica1' is not available due to init failure: SolrCore 'teste2_shard1_replica1' is not available due to init failure: java.nio.file.AccessDeniedException: /opt/solr/server/solr/mycores/teste2_shard1_replica1
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1151)
at org.apache.solr.cloud.ZkController.publish(ZkController.java:1198)
at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1372)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:885)
... 35 more
Caused by: org.apache.solr.common.SolrException: SolrCore 'teste2_shard1_replica1' is not available due to init failure: java.nio.file.AccessDeniedException: /opt/solr/server/solr/mycores/teste2_shard1_replica1
... 39 more
Caused by: org.apache.solr.common.SolrException: java.nio.file.AccessDeniedException: /opt/solr/server/solr/mycores/teste2_shard1_replica1
at org.apache.solr.core.SolrCore.(SolrCore.java:959)
at org.apache.solr.core.SolrCore.(SolrCore.java:823)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890)
... 35 more
Caused by: java.lang.IllegalStateException: java.nio.file.AccessDeniedException: /opt/solr/server/solr/mycores/teste2_shard1_replica1
at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:479)
at org.apache.solr.core.SolrCore.(SolrCore.java:900)
... 37 more
Caused by: java.nio.file.AccessDeniedException: /opt/solr/server/solr/mycores/teste2_shard1_replica1
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at org.apache.lucene.store.FSDirectory.(FSDirectory.java:157)
at org.apache.lucene.store.MMapDirectory.(MMapDirectory.java:159)
at org.apache.lucene.store.MMapDirectory.(MMapDirectory.java:112)
at org.apache.lucene.store.FSDirectory.open(FSDirectory.java:197)
at org.apache.solr.core.NRTCachingDirectoryFactory.create(NRTCachingDirectoryFactory.java:55)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:346)
at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:201)
at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:475)
... 38 more

2017-02-08 19:53:27.520 INFO (qtp575335780-15) [c:teste2 s:shard1 x:teste2_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=teste2&newCollection=true&name=teste2_shard1_replica1&action=CREATE&numShards=1&collection=teste2&shard=shard1&wt=javabin&version=2} status=400 QTime=6
2017-02-08 19:53:27.520 ERROR (OverseerThreadFactory-6-thread-5-processing-n:172.19.0.3:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: http://172.19.0.3:8983/solr
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.19.0.3:8983/solr: Error CREATEing SolrCore 'teste2_shard1_replica1': Unable to create core [teste2_shard1_replica1] Caused by: /opt/solr/server/solr/mycores/teste2_shard1_replica1
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:610)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-02-08 19:53:27.521 ERROR (OverseerThreadFactory-6-thread-5-processing-n:172.19.0.3:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [teste2].
2017-02-08 19:53:27.527 INFO (OverseerThreadFactory-6-thread-5-processing-n:172.19.0.3:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
2017-02-08 19:53:28.397 INFO (OverseerThreadFactory-6-thread-5-processing-n:172.19.0.3:8983_solr) [ ] o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [teste2]
2017-02-08 19:53:28.428 INFO (qtp575335780-13) [ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
2017-02-08 19:53:29.401 INFO (OverseerCollectionConfigSetProcessor-97418717314875427-172.19.0.3:8983_solr-n_0000000009) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000026 doesn't exist. Requestor may have disconnected from ZooKeeper
2017-02-08 19:53:32.521 INFO (qtp575335780-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&=1486576555855&since=0} status=0 QTime=1
2017-02-08 19:53:43.521 INFO (qtp575335780-15) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&=1486576555855&since=0} status=0 QTime=1
2017-02-08 19:53:54.523 INFO (qtp575335780-17) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&=1486576555855&since=0} status=0 QTime=1
2017-02-08 19:53:58.431 ERROR (qtp575335780-13) [ ] o.a.s.h.a.CollectionsHandler Timed out waiting for new collection's replicas to become ACTIVE with timeout=30
2017-02-08 19:53:58.432 INFO (qtp575335780-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&maxShardsPerNode=1&collection.configName=teste2&name=teste2&action=CREATE&numShards=1&wt=json} status=0 QTime=31063
2017-02-08 19:54:05.523 INFO (qtp575335780-17) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&=1486576555855&since=0} status=0 QTime=1`

The mounted volume is writeable using /bin/bash inside docker docker container.

Can anyone help me?
@makuk66
Copy link
Contributor

makuk66 commented Feb 8, 2017

If you see a java.nio.file.AccessDeniedException, then it's an ownership issue. If you're using your own Dockerfile that does a COPY, do you chown -R that to the solr user after the copy? Or do you perhaps need to explicitly repeat the USER solr in your Dockerfile?

@hgsaldanha
Copy link
Author

I added the USER solr and the folder has a 0777 access. Still the same error.

@risdenk
Copy link

risdenk commented Feb 9, 2017

The issue is you are mounting the core directory and that has the wrong permissions.

-v /BACKUP/docker_volumes/asspa/solr/cores:/opt/solr/server/solr/mycores

The UID of Solr inside the container can't write to that directory that was mounted. Check the permissions of the mounted directory.

@hgsaldanha
Copy link
Author

It was a permission issue. Solved.
Thanks a lot!

@jeffyjefflabs jeffyjefflabs mentioned this issue Dec 15, 2018
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@makuk66 @risdenk @hgsaldanha