neo4j hangs when calling ga.timetree stored procedure in a transaction

[ahmetkizilay]

I am trying to run graphaware's timetree stored procedures inside a transaction. It looks like Neo4j hangs after a couple of cypher calls when the stored procedure is called inside a transaction if there is another query updating the database.

In essence, I am running tests and I would like to clear the database before I run a query. However when I call some other cypher with Neo4j::Session.query inside or outside a transaction, I get the error.

See the sample project to replicate the issue.
https://github.com/ahmetkizilay/neo4jrb-timetree-error

neo4j-core (6.1.4), neo4j community edition 3.0.1, graphaware timetree 3.0.1.38.24

[ikwattro]

@ahmetkizilay Do you have any logs in the database, espcially neo4j.log and debug.log ?

[ahmetkizilay]

see the logs here @ikwattro, thanks!
https://gist.github.com/ahmetkizilay/45870280d5dc92f0f9357dac0c728864

[cheerfulstoic]

Here's the log of the HTTP requests (the 0 and 1 are from the ruby script):

0
I, [2016-06-02T11:33:29.038617 #21438]  INFO -- : post http://localhost:9930/db/data/transaction
D, [2016-06-02T11:33:29.038662 #21438] DEBUG -- request: Content-Type: "application/json"
User-Agent: "neo4j-core-gem/6.1.5 (https://github.com/neo4jrb/neo4j-core)"
Authorization: "Basic <REDACTED>"
I, [2016-06-02T11:33:29.341466 #21438]  INFO -- Status: 201
D, [2016-06-02T11:33:29.341761 #21438] DEBUG -- response: date: "Thu, 02 Jun 2016 09:33:29 GMT"
location: "http://localhost:9930/db/data/transaction/1"
content-type: "application/json"
access-control-allow-origin: "*"
content-length: "172"
server: "Jetty(9.2.9.v20150224)"
I, [2016-06-02T11:33:29.342491 #21438]  INFO -- : post http://localhost:9930/db/data/transaction/1
D, [2016-06-02T11:33:29.342585 #21438] DEBUG -- request: Content-Type: "application/json"
User-Agent: "neo4j-core-gem/6.1.5 (https://github.com/neo4jrb/neo4j-core)"
Authorization: "Basic <REDACTED>"
I, [2016-06-02T11:33:29.852110 #21438]  INFO -- Status: 200
D, [2016-06-02T11:33:29.852204 #21438] DEBUG -- response: date: "Thu, 02 Jun 2016 09:33:29 GMT"
content-type: "application/json"
access-control-allow-origin: "*"
content-length: "1415"
server: "Jetty(9.2.9.v20150224)"
I, [2016-06-02T11:33:29.852557 #21438]  INFO -- : post http://localhost:9930/db/data/transaction/1/commit
D, [2016-06-02T11:33:29.852602 #21438] DEBUG -- request: Content-Type: "application/json"
User-Agent: "neo4j-core-gem/6.1.5 (https://github.com/neo4jrb/neo4j-core)"
Accept: "application/json; charset=UTF-8"
Authorization: "Basic <REDACTED>"
I, [2016-06-02T11:33:29.958604 #21438]  INFO -- Status: 200
D, [2016-06-02T11:33:29.958675 #21438] DEBUG -- response: date: "Thu, 02 Jun 2016 09:33:29 GMT"
content-type: "application/json"
access-control-allow-origin: "*"
content-length: "26"
server: "Jetty(9.2.9.v20150224)"
1
I, [2016-06-02T11:33:29.958996 #21438]  INFO -- : post http://localhost:9930/db/data/transaction
D, [2016-06-02T11:33:29.959045 #21438] DEBUG -- request: Content-Type: "application/json"
User-Agent: "neo4j-core-gem/6.1.5 (https://github.com/neo4jrb/neo4j-core)"
Authorization: "Basic <REDACTED>"
I, [2016-06-02T11:33:29.969087 #21438]  INFO -- Status: 201
D, [2016-06-02T11:33:29.969218 #21438] DEBUG -- response: date: "Thu, 02 Jun 2016 09:33:29 GMT"
location: "http://localhost:9930/db/data/transaction/2"
content-type: "application/json"
access-control-allow-origin: "*"
content-length: "172"
server: "Jetty(9.2.9.v20150224)"
I, [2016-06-02T11:33:29.969565 #21438]  INFO -- : post http://localhost:9930/db/data/transaction/2
D, [2016-06-02T11:33:29.969613 #21438] DEBUG -- request: Content-Type: "application/json"
User-Agent: "neo4j-core-gem/6.1.5 (https://github.com/neo4jrb/neo4j-core)"
Authorization: "Basic <REDACTED>"

[cheerfulstoic]

I think that's pretty normal. @ikwattro maybe it could be something with the plugin?

[ikwattro]

i'll investigate

[ahmetkizilay]

@cheerfulstoic

I am trying to replicate the error by calling rest api methods with httpie, but I can't. REST api calls seem to work without any problems...

From faraday logs, I see that this is the routine:

1. POST /db/data/transaction with the delete all query.
2. POST /db/data/transaction/{id} with the stored procedure query.
3. POST /db/data/transaction/{id}/commit with empty body.

Here's the log for httpie queries: https://gist.github.com/ahmetkizilay/5fa52063738acce4c34546571d68f8b8
You can also pull the example repository to run the httpie calls yourself.

One thing I see with faraday is that faraday does not include content-length in the headers. I tried to add Content-Length in cypher_transaction.rb but I still get errors.

What do you think?

[cheerfulstoic]

When I was running it it seemed to hang on the second go around. Did you try it in a loop?

[ahmetkizilay]

hi @cheerfulstoic

I am trying
/bin/bash ./requests/test-should-hang.sh (multiple times) and
/bin/bash ./requests/test-multiple.sh (this one only after server restart)

it is strange that neo4j hangs with neo4jrb, but not with direct http calls.

[ahmetkizilay]

Hi @ikwattro, you might find this interesting.

(Now I am using Neo4j community edition 3.0.3)

It looks like the problem occurs after deleting the TimeTreeRoot node. When I delete all nodes except the TimeTreeRoot in a transaction, the database does not hang anymore.

Below is the sample code. you can view full code here.

Thanks a lot.

  begin
    tx = Neo4j::Transaction.new

    delete_all_except_timetreeroot = "
      MATCH (n) WHERE NOT ('TimeTreeRoot' IN labels(n))
      DETACH DELETE n
    "
    Neo4j::Session.query(delete_all_except_timetreeroot)

    simple_timetree_query= "
      CREATE (tsn:TSN)
      SET tsn = { value: 'prop1', val2: 2 }
      WITH tsn
      CALL ga.timetree.single({time: timestamp(), resolution: 'Millisecond', create: true})
      YIELD instant as instant
      CREATE (tsn)-[:HAS_EVENT]->(instant)
    "
    Neo4j::Session.query(simple_timetree_query)

  rescue Exception => exp
    puts exp.message
    tx.failure
  ensure
    tx.close
  end

[ikwattro]

Thanks for the detailed report. Would you mind opening an issue and mention me on the timetree repository itself, also paste this snippet there. Thanks a lot !