Exception during data load using BatchGraph #509

Closed
satb opened this Issue Dec 4, 2013 · 6 comments

Projects

None yet

2 participants

satb commented Dec 4, 2013

I have a 2 node Titan/Cassandra cluster. I loaded data on one node and am running a script to load data on the same node I have the data. Consistently and repeatedly, I keep getting this error. Can you please help me with any workarounds?

I have some 6-7 files of data that I read sequentially in the script and load the data. The script looks like this

['f1', 'f2', 'f3', 'f4', 'f5', 'f6'].each { filename ->
            println "Processing ${filename}"
            def attrf = new File("../data/${filename}.csv".toString())
            attrf.eachLine{
                def parts = it.split('\t')
                def attr = parts[0]
                def vIds = parts[1].split(",")
                def bgVId = filename + '_' + attr
                Vertex v = bg.addVertex(bgVId)
                v.setProperty(filename.substring(0, filename.length()-1), attr)
                vIds.each { vId ->
                    bg.addEdge(null, bg.getVertex(bgVId), bg.getVertex(vId), 'of')
                }
            }
            println "Done with ${filename}"
        }

The first 3 files load fine and I can see the "Done with ${filename}" message on the console. With the 4th file, it blows up with the below exception

Exception in thread "main" com.thinkaurelius.titan.core.TitanException: Could not commit transaction due to exception during persistence
    at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1012)
    at com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.commit(TitanBlueprintsGraph.java:57)
    at com.tinkerpop.blueprints.util.wrappers.batch.BatchGraph.nextElement(BatchGraph.java:225)
    at com.tinkerpop.blueprints.util.wrappers.batch.BatchGraph.addEdge(BatchGraph.java:373)
    at com.tinkerpop.blueprints.util.wrappers.batch.BatchGraph.addEdge(BatchGraph.java:367)
    at com.tinkerpop.blueprints.Graph$addEdge.call(Unknown Source)
    at com.intuit.guns.core.graph.data.ImportManager$_bulkLoadMerchants_closure5_closure8_closure9.doCall(ImportManager.groovy:194)
    at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
    at groovy.lang.Closure.call(Closure.java:411)
    at groovy.lang.Closure.call(Closure.java:427)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1326)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1298)
    at org.codehaus.groovy.runtime.dgm$148.invoke(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at com.intuit.guns.core.graph.data.ImportManager$_bulkLoadMerchants_closure5_closure8.doCall(ImportManager.groovy:193)
    at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
    at groovy.lang.Closure.call(Closure.java:411)
    at groovy.lang.Closure.call(Closure.java:427)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.callClosureForLine(DefaultGroovyMethods.java:3884)
    at org.codehaus.groovy.runtime.IOGroovyMethods.eachLine(IOGroovyMethods.java:463)
    at org.codehaus.groovy.runtime.ResourceGroovyMethods.eachLine(ResourceGroovyMethods.java:269)
    at org.codehaus.groovy.runtime.ResourceGroovyMethods.eachLine(ResourceGroovyMethods.java:235)
    at org.codehaus.groovy.runtime.dgm$769.invoke(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at com.intuit.guns.core.graph.data.ImportManager$_bulkLoadMerchants_closure5.doCall(ImportManager.groovy:186)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
    at groovy.lang.Closure.call(Closure.java:411)
    at groovy.lang.Closure.call(Closure.java:427)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1326)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1298)
    at org.codehaus.groovy.runtime.dgm$148.invoke(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at com.intuit.guns.core.graph.data.ImportManager.bulkLoadMerchants(ImportManager.groovy:183)
    at com.intuit.guns.core.graph.data.ImportManager$bulkLoadMerchants.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
    at com.intuit.guns.core.graph.data.ImportManager.main(ImportManager.groovy:205)
Caused by: com.thinkaurelius.titan.core.TitanException: Unexpected exception during backend operation
    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:67)
    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:324)
    at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1002)
    ... 64 more
Caused by: com.thinkaurelius.titan.core.TitanException: Permanent exception during backend operation
    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:65)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction.flushInternal(BufferTransaction.java:96)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction.mutate(BufferTransaction.java:84)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferedKeyColumnValueStore.mutate(BufferedKeyColumnValueStore.java:52)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.CachedKeyColumnValueStore.mutate(CachedKeyColumnValueStore.java:118)
    at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.mutate(ExpectedValueCheckingStore.java:102)
    at com.thinkaurelius.titan.diskstorage.BackendTransaction.mutateVertexIndex(BackendTransaction.java:128)
    at com.thinkaurelius.titan.graphdb.database.IndexSerializer.addProperty(IndexSerializer.java:152)
    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.persist(StandardTitanGraph.java:370)
    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.access$000(StandardTitanGraph.java:49)
    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:314)
    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:245)
    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:62)
    ... 66 more
Caused by: com.thinkaurelius.titan.diskstorage.PermanentStorageException: Permanent failure in storage backend
    at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftKeyColumnValueStore.convertException(CassandraThriftKeyColumnValueStore.java:313)
    at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager.mutateMany(CassandraThriftStoreManager.java:213)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction$1.call(BufferTransaction.java:99)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction$1.call(BufferTransaction.java:96)
    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:62)
    ... 78 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
    at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
    at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:964)
    at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:950)
    at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager.mutateMany(CassandraThriftStoreManager.java:211)
    ... 81 more
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:152)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
    ... 92 more

Owner

Try a smaller batch size for BatchGraph and set the write-conssitency-level
to ONE for Cassandra.

On Tue, Dec 3, 2013 at 4:50 PM, satb notifications@github.com wrote:

I have a 2 node Titan/Cassandra cluster. I loaded data on one node and am
running a script to load data data on the same node I have the data.
Consistently and repeatedly, I keep getting this error. Can you please help
me with any workarounds?

I have some 6-7 files of data that I read sequentially in the script and
load the data. The script looks like this

['f1', 'f2', 'f3', 'f4', 'f5', 'f6'].each { filename ->
println "Processing ${filename}"
def attrf = new File("../data/${filename}.csv".toString())
attrf.eachLine{
def parts = it.split('\t')
def attr = parts[0]
def vIds = parts[1].split(",")
def bgVId = filename + '_' + attr
Vertex v = bg.addVertex(bgVId)
v.setProperty(filename.substring(0, filename.length()-1), attr)
vIds.each { vId ->
bg.addEdge(null, bg.getVertex(bgVId), bg.getVertex(vId), 'of')
}
}
println "Done with ${filename}"
}

The first 3 files load fine and I can see the "Done with ${filename}"
message on the console. With the 4th file, it blows up with the below
exception

Exception in thread "main" com.thinkaurelius.titan.core.TitanException: Could not commit transaction due to exception during persistence
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1012)
at com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.commit(TitanBlueprintsGraph.java:57)
at com.tinkerpop.blueprints.util.wrappers.batch.BatchGraph.nextElement(BatchGraph.java:225)
at com.tinkerpop.blueprints.util.wrappers.batch.BatchGraph.addEdge(BatchGraph.java:373)
at com.tinkerpop.blueprints.util.wrappers.batch.BatchGraph.addEdge(BatchGraph.java:367)
at com.tinkerpop.blueprints.Graph$addEdge.call(Unknown Source)
at com.intuit.guns.core.graph.data.ImportManager$_bulkLoadMerchants_closure5_closure8_closure9.doCall(ImportManager.groovy:194)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
at groovy.lang.Closure.call(Closure.java:411)
at groovy.lang.Closure.call(Closure.java:427)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1326)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1298)
at org.codehaus.groovy.runtime.dgm$148.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.intuit.guns.core.graph.data.ImportManager$_bulkLoadMerchants_closure5_closure8.doCall(ImportManager.groovy:193)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
at groovy.lang.Closure.call(Closure.java:411)
at groovy.lang.Closure.call(Closure.java:427)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.callClosureForLine(DefaultGroovyMethods.java:3884)
at org.codehaus.groovy.runtime.IOGroovyMethods.eachLine(IOGroovyMethods.java:463)
at org.codehaus.groovy.runtime.ResourceGroovyMethods.eachLine(ResourceGroovyMethods.java:269)
at org.codehaus.groovy.runtime.ResourceGroovyMethods.eachLine(ResourceGroovyMethods.java:235)
at org.codehaus.groovy.runtime.dgm$769.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.intuit.guns.core.graph.data.ImportManager$_bulkLoadMerchants_closure5.doCall(ImportManager.groovy:186)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
at groovy.lang.Closure.call(Closure.java:411)
at groovy.lang.Closure.call(Closure.java:427)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1326)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1298)
at org.codehaus.groovy.runtime.dgm$148.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.intuit.guns.core.graph.data.ImportManager.bulkLoadMerchants(ImportManager.groovy:183)
at com.intuit.guns.core.graph.data.ImportManager$bulkLoadMerchants.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
at com.intuit.guns.core.graph.data.ImportManager.main(ImportManager.groovy:205)
Caused by: com.thinkaurelius.titan.core.TitanException: Unexpected exception during backend operation
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:67)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:324)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1002)
... 64 more
Caused by: com.thinkaurelius.titan.core.TitanException: Permanent exception during backend operation
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:65)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction.flushInternal(BufferTransaction.java:96)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction.mutate(BufferTransaction.java:84)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferedKeyColumnValueStore.mutate(BufferedKeyColumnValueStore.java:52)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.CachedKeyColumnValueStore.mutate(CachedKeyColumnValueStore.java:118)
at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.mutate(ExpectedValueCheckingStore.java:102)
at com.thinkaurelius.titan.diskstorage.BackendTransaction.mutateVertexIndex(BackendTransaction.java:128)
at com.thinkaurelius.titan.graphdb.database.IndexSerializer.addProperty(IndexSerializer.java:152)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.persist(StandardTitanGraph.java:370)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.access$000(StandardTitanGraph.java:49)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:314)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:245)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:62)
... 66 more
Caused by: com.thinkaurelius.titan.diskstorage.PermanentStorageException: Permanent failure in storage backend
at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftKeyColumnValueStore.convertException(CassandraThriftKeyColumnValueStore.java:313)
at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager.mutateMany(CassandraThriftStoreManager.java:213)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction$1.call(BufferTransaction.java:99)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction$1.call(BufferTransaction.java:96)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:62)
... 78 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:964)
at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:950)
at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager.mutateMany(CassandraThriftStoreManager.java:211)
... 81 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 92 more


Reply to this email directly or view it on GitHubhttps://github.com/thinkaurelius/titan/issues/509
.

Matthias Broecheler
http://www.matthiasb.com

satb commented Dec 4, 2013

@mbroecheler - Thanks. The changes didn't work...it made the problem worse. Not even the first file loads now. I get the same exception as above. I fire off the script to start BatchLoad, the script waits for about 2 minutes and then throws the exception

bash-4.1$ cat ../titan-server-0.4.1/conf/titan-cassandra-es.properties 
storage.backend=cassandrathrift
storage.hostname=127.0.0.1
cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.25

storage.index.search.backend=elasticsearch
storage.index.search.directory=../db/es
storage.index.search.client-only=false
storage.index.search.local-mode=true

storage.batch-loading=true
storage.write-consistency-level=ONE

Batch Graph defined like so:

def g = TitanFactory.open('<path_to_titan-cassandra-es.properties>')
def bg = new BatchGraph(g, VertexIDType.STRING, 100000)
satb commented Dec 4, 2013

Tried just changing the batch size to 100,000 and left the write-consistency-level change out (defaults to QUORUM I believe right?) - but that didn't change anything. Starts to load and after the 3rd file is loaded, it thinks and thinks and then throws an exception. The fourth file I believe has many more edges than the first 3. Don't know if that makes any difference.

satb commented Dec 4, 2013

If I shut down the second node (i.e. only single node now), then the batch data load happens smoothly. All files finished loading without issues. So something between the two nodes when they are part of the cluster is causing the problem. What could that be? And why would it fail only after a few files have successfully loaded? The data characteristics between the files are not all that different...probably the fourth file which failed consistently has more edges than the first three. Could it thrift frame size? I am a relative newcomer to both Titan and Cassandra so I am not quite sure I can make a good prediction

satb commented Dec 4, 2013

In case we need anything else from this machine please let me know. I have to put this VM back into the pool pretty soon.

Owner

It probably has to do with how the Cassandra cluster is configured
(balanced token ring, yaml configuration), how Titan is configured (QUORUM
writes are more expensive, id pool configuration, etc) or characteristics
of your environment (network latancy, etc).

There seems to be nothing wrong with Titan here since the read timeout
occurs in Cassandra. Please try to tune the system and your topology.

On Wed, Dec 4, 2013 at 12:43 PM, satb notifications@github.com wrote:

In case we need anything else from this machine please let me know. I have
to put this VM back into the pool pretty soon.


Reply to this email directly or view it on GitHubhttps://github.com/thinkaurelius/titan/issues/509#issuecomment-29843401
.

Matthias Broecheler
http://www.matthiasb.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment