could not re-open management log exception while running groovy script

[amyth]

Hi,

I had a Janusgraph/Cassandra (forked) implementation on my dev server. I had written a population script to populate the data that reads data from a JSON file and creates the required vertices, edges, and properties. Due to some implementation changes, I wanted to update the script to read a CSV file instead. In order to test this, I wanted to keep a backup of the current implementation so I simply made a copy of janusgraph-0.1.0-hadoop2 folder and tried to run the script. And since then I am getting the following exception.

could not re-open management log

I have tried the following:

• ./bin/janusgraph.sh restart (stop/start)
• ./bin/janusgraph.sh clean
• Delete whole project and re-create it
• Download version 0.1.1 and try running the script with it.

But nothing works, every time I run the script I get the aforementioned error.

I tried to debug the script and found that the following statement is raising the error:

this.management.awaitGraphIndexStatus(this.graph, 'uniqueUid').call()

When I try running this in the gremlin-shell everything works fine. The problem arises when I try to execute the script as follows:

./bin/gremlin.sh -e /path/to/script

Here is the link to the script that I am trying, just in case if it's required for debugging purposes.

questions

A few questions that I am looking an answer to are:

• Where does the management log reside?
• If it's the log/gremlin-server.log why am I facing this issue even after deleting everything and starting from scratch?

Any help will be much appreciated. Thanks.

[pluradj]

Please post the full stack trace.

[amyth]

Hey Jason (@pluradj), The output of the script does not print the full stack trace even with verbose option -v while executing the script. Here is the output that I get.

cd ~/Downloads/janusgraph-0.1.0-hadoop2
./bin/gremlin.sh -e ~/Documents/scripts/xonnect/utils/population_scripts/populateJanus.groovy -v

objc[2039]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/bin/java (0x10a54b4c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x10bdbc4e0). One of the two will be used. Which one is undefined.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/admin/Desktop/janusgraph-0.1.1-hadoop2/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/admin/Desktop/janusgraph-0.1.1-hadoop2/lib/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Preparing schema.
javax.script.ScriptException: org.janusgraph.core.JanusGraphException: Could not re-open management log

Here is the screenshot for the same.

Also, I did not find anything related to this in the log files. I tried running the same commands within the Gremlin console to get the full stack trace, but everything works fine when I do this within the console.

[pluradj]

You need to add a try/catch block

try {
    this.management.awaitGraphIndexStatus(this.graph, 'uniqueUid').call()
} catch (Exception e) {
    e.printStackTrace()
}

[amyth]

My bad, the earlier stack trace was another error. Here is the stack trace to the exception in question.

org.janusgraph.core.JanusGraphException: Could not re-open management log
org.janusgraph.core.JanusGraphException: Could not re-open management log
	at org.janusgraph.diskstorage.Backend.getSystemMgmtLog(Backend.java:334)
	at org.janusgraph.graphdb.database.StandardJanusGraph.openManagement(StandardJanusGraph.java:303)
	at org.janusgraph.graphdb.database.management.GraphIndexStatusWatcher.call(GraphIndexStatusWatcher.java:64)
	at org.janusgraph.graphdb.database.management.GraphIndexStatusWatcher.call(GraphIndexStatusWatcher.java:32)
	at java_util_concurrent_Callable$call.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
	at JanusGraphBuilder.createSchema(Script1.groovy:133)
	at JanusGraphBuilder.populate(Script1.groovy:148)
	at JanusGraphBuilder.main(Script1.groovy:33)
	at JanusGraphBuilder$main.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
	at Script1.run(Script1.groovy:250)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:619)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:448)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:421)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:212)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.ScriptExecutor.evaluate(ScriptExecutor.java:55)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.ScriptExecutor.main(ScriptExecutor.java:44)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Permanent failure in storage backend
	at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.ensureColumnFamilyExists(CassandraThriftStoreManager.java:539)
	at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.ensureColumnFamilyExists(CassandraThriftStoreManager.java:511)
	at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.openDatabase(CassandraThriftStoreManager.java:328)
	at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.openDatabase(CassandraThriftStoreManager.java:67)
	at org.janusgraph.diskstorage.log.kcvs.KCVSLogManager.openLog(KCVSLogManager.java:225)
	at org.janusgraph.diskstorage.Backend.getSystemMgmtLog(Backend.java:332)
	... 21 more
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
	at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.ensureKeyspaceExists(CassandraThriftStoreManager.java:484)
	at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.ensureColumnFamilyExists(CassandraThriftStoreManager.java:517)
	... 26 more
Caused by: java.lang.IllegalStateException: Pool not open
	at org.apache.commons.pool.BaseKeyedObjectPool.assertOpen(BaseKeyedObjectPool.java:160)
	at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1113)
	at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.ensureKeyspaceExists(CassandraThriftStoreManager.java:453)
	... 27 more

(will delete the previous stack trace to keep the thread clean)

[pluradj]

In your createSchema(), be aware that in addition to creating schema, you are trying to override the ids.block-size configuration property:

this.management.set('ids.block-size', 20000000)

If you want to set the ids.block-size, it would be best to put it in conf/testconf.properties when you first create the graph. That property has GLOBAL_OFFLINE mutability. Because your code is trying to change that configuration value, after the management transaction containing that update gets committed, the graph needs to be shutdown and reconnected before continuing to work with the graph. This is why Thrift could no longer connect to the keyspace in the code where the error was thrown.

I'll also note that if you're creating the indexes into an empty keyspace before any graph data is populated, the indexes will get enabled immediately and you wouldn't need to reindex, so you could remove this code:

this.management.awaitGraphIndexStatus(this.graph, 'uniqueUid').call()
this.management = this.graph.openManagement()
this.management.updateIndex(this.management.getGraphIndex('uniqueUid'), SchemaAction.REINDEX).get()
this.management.commit()

[pluradj]

@amyth I believe this issue is resolved. Please reopen or reach out on the users mailing list or Gitter.

[amyth]

Hi @pluradj, Yes the issue is resolved after removing the block-size modification statement and re-indexing statements.