Inconsistent state of neo4j database

[peter-ki]

Running neo4j 3.1.1 CE in the official neo4j-docker. DB has around 220GB of data. After a restart, the server crashes upon startup.

docker logs neo4j

reveals

Starting Neo4j.
2017-02-10 13:18:24.569+0000 INFO  No SSL certificate found, generating a self-signed certificate..
2017-02-10 13:18:25.249+0000 INFO  Starting...
2017-02-10 13:18:26.905+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2017-02-10 13:18:33.348+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@3681d595' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@3681d595' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@3681d595' was successfully initialized, but failed to start. Please see attached cause exception.
	at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:227)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:68)
	at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:28)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase@3681d595' was successfully initialized, but failed to start. Please see attached cause exception.
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:443)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:199)
	... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /var/lib/neo4j/data/databases/graph.db
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:193)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:124)
	at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:57)
	at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
	... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine@7a53b31e' failed to initialize. Please see attached cause exception.
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:415)
	at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:62)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:98)
	at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:517)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
	at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:100)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:189)
	... 9 more
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.lucene.util.fst.FST.<init>(FST.java:387)
	at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
	at org.apache.lucene.codecs.blocktreeords.OrdsFieldReader.<init>(OrdsFieldReader.java:83)
	at org.apache.lucene.codecs.blocktreeords.OrdsBlockTreeTermsReader.<init>(OrdsBlockTreeTermsReader.java:147)
	at org.apache.lucene.codecs.blocktreeords.BlockTreeOrdsPostingsFormat.fieldsProducer(BlockTreeOrdsPostingsFormat.java:93)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:104)
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58)
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:50)
	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731)
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:50)
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
	at org.neo4j.kernel.api.impl.schema.LuceneSchemaIndex.isOnline(LuceneSchemaIndex.java:151)
	at org.neo4j.kernel.api.impl.schema.WritableDatabaseSchemaIndex.isOnline(WritableDatabaseSchemaIndex.java:86)
	at org.neo4j.kernel.api.impl.schema.LuceneSchemaIndexProvider.indexIsOnline(LuceneSchemaIndexProvider.java:174)
	at org.neo4j.kernel.api.impl.schema.LuceneSchemaIndexProvider.getInitialState(LuceneSchemaIndexProvider.java:135)
	at org.neo4j.kernel.impl.api.index.IndexingService.init(IndexingService.java:201)
	at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.init(RecordStorageEngine.java:418)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:405)
	at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:62)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:98)
	at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:517)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
	at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:100)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:189)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:124)
	at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:57)

Any clue how to recover from this inconsistent state?

[peter-ki]

I've tracked this down, my problem is resolved. The neo4j.conf file was overwritten with a version with too little working memory. But neo4j should have handled the situation more gracefully...

[chrisvest]

Reducing our reliance on heap memory for scaling to large workloads is a long-term work item for us. it has improved a lot over the last couple of years, and we plan to improve it further. I'm going to close this.