org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException during import on Windows

[vsletten]

I'm trying to import a 3 GB node file and a 1.5 GB edge file on Windows. Using neo4j version community-1.9.M01. After crunching away on the nodes file for about 20 sec, I get the exception shown in the title. Here is the full stack trace. Note, the batch-importer works fine for me on much smaller files.

C:\batch-import-master>java -server -Xmx12G -jar target/batch-import-jar-with-dependencies.jar target/db nodes2.csv edges1.csv
Using Existing Configuration File
.........................Exception in thread "main" org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: Unable to close store C:\batch-import-master\target\db\neostore.propertystore.db.strings
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:636)
at org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:118)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:571)
at org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:231)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:571)
at org.neo4j.unsafe.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:703)
at org.neo4j.batchimport.Importer.finish(Importer.java:83)
at org.neo4j.batchimport.Importer.main(Importer.java:77)
Caused by: java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open
at sun.nio.ch.FileDispatcherImpl.truncate0(Native Method)
at sun.nio.ch.FileDispatcherImpl.truncate(Unknown Source)
at sun.nio.ch.FileChannelImpl.truncate(Unknown Source)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:606)
... 7 more

[jexp]

Thanks for the feedback, I escalated it within the Neo4j team, sorry have never run the Batch-Inserter on a Windows (don't have any). In the meantime could you run your import on a Linux instance (e.g. on ec2) ?

[jakewins]

@vsletten this is a known issue, and should go away if you turn off memory mapping (which should be disabled by default on windows). The performance impact of that may be large, however. If you find performance to be too slow, and you have tuned the memory settings of the batch importer to reasonable levels, consider doing the batch import on an OS X, Unix or Linux system.

[jexp]

Then it perhaps makes sense to set the mmio-settings to be all smaller or equal to 2G ?

[rswarup82]

I am trying to import 2.8M nodes using super fast batch importer. It seems in windows machine has some issue. I am getting below exception. While googling found same issue found before. Looks like issue is still persist.

C:\Tools\NEO4J_Project\batch_importer_20>import.bat graph.db
Usage: Importer data/dir nodes.csv relationships.csv [node_index node-index-name fulltext|exact nodes_index.csv rel_index rel-index-name fulltext|exact rels_index.csv ...
Using: Importer graph.db

Using Existing Configuration File
store_dir=C:\Tools\NEO4J_Project\batch_importer_20\graph.db
use_memory_mapped_buffers=true
neostore.propertystore.db.index.keys.mapped_memory=1M
neostore.propertystore.db.strings.mapped_memory=100M
neostore.propertystore.db.arrays.mapped_memory=215M
neostore.relationshipstore.db.mapped_memory=1000M
batch_import.keep_db=false
neostore.propertystore.db.index.mapped_memory=1M
dump_configuration=true
neostore.propertystore.db.mapped_memory=1000M
cache_type=none
batch_import.csv.delim=;
neostore.nodestore.db.mapped_memory=1000M
batch_import.graph_db=graph.db
batch_import.nodes_files=sample/semicolon_seperated_delimited/txn.csv
............................
Importing 2826680 Nodes took 18 seconds
Exception in thread "main" org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: Unable to close store C:\Tools\NEO4J_Project\batch_importer_20\graph.db\neostore
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:596)
at org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:253)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:539)
at org.neo4j.unsafe.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:992)
at org.neo4j.batchimport.Importer.finish(Importer.java:88)
at org.neo4j.batchimport.Importer.doImport(Importer.java:239)
at org.neo4j.batchimport.Importer.main(Importer.java:83)
Caused by: java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open
at sun.nio.ch.FileDispatcherImpl.truncate0(Native Method)
at sun.nio.ch.FileDispatcherImpl.truncate(Unknown Source)
at sun.nio.ch.FileChannelImpl.truncate(Unknown Source)
at org.neo4j.kernel.impl.nioneo.store.StoreFileChannel.truncate(StoreFileChannel.java:69)
at org.neo4j.kernel.impl.nioneo.store.StoreFileChannel.truncate(StoreFileChannel.java:28)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:576)
... 6 more

Should I need to try from linux machine ? Looking forward to hear from you. Thanks

[jexp]

Hi @rswarup82,

yes linux would be the easiest alternative.

You can try to change the config to use_memory_mapped_buffers=false
and increase the heap size in the batch-script to accomodate 2G + the sum of your configured mmio sizes: i.e. 5GB heap

I would change those to this:

neostore.nodestore.db.mapped_memory=250M
neostore.relationshipstore.db.mapped_memory=1000M
neostore.relationshipgroupstore.db.mapped_memory=10M
neostore.propertystore.db.mapped_memory=500M
neostore.propertystore.db.strings.mapped_memory=500M
neostore.propertystore.db.arrays.mapped_memory=215M

[rswarup82]

Hi Michael,

It's working now. Thanks for your quick help.