Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Allow batch created unique nodes to be referable #84

Closed
lassewesth opened this Issue · 19 comments
@lassewesth
Owner

@maxdemarzi: 'http://docs.neo4j.org/chunked/milestone/rest-api-batch-ops.html

19.14.2. Refer to items created earlier in the same batch job
The batch operation API allows you to refer to the URI returned from a created resource in subsequent job descriptions, within the same batch call.

In the following, both the add_node_to_index call and the create_relationship call fail since create_unique_node is a reference to an index entry instead of a node.

batch_result = @neo.batch
[:create_unique_node, "person", "ssn", "000-00-0001", {:first_name=>"Jane", :last_name=>"Doe", :ssn=>"000-00-0001"],
[:add_node_to_index, "person_ssn", "ssn", "000-00-0001", "{0}"],
[:create_node, {:street1=>"94437 Kemmer Crossing", :street2=>"Apt. 333", :city=>"Abshireton", :state=>"AA", :zip=>"65820"],
[:create_relationship, "has", "{0}", "{2}", {}]
'

@lassewesth
Owner

valueCount: 1
strings:

@jcbozonier

Is there a workaround for this issue? It seems like a pretty big deal breaker for me right now. Thanks!

@maxdemarzi
Collaborator

Yup, use Cypher:

Create the unique nodes as normal, then reference them in cypher to create the relationship between them.

batch_result = @neo.batch [:create_unique_node, "person", "ssn", "000-00-0003", {:first_name=>"Jane"}], 
[:create_unique_node, "person", "ssn", "000-00-0004", {:first_name=>"John"}], 
[:execute_query, "START a = node:person(ssn=\"000-00-0003\"), 
b = node:person(ssn=\"000-00-0004\") 
CREATE a -[new_rel:follows]-> b 
RETURN new_rel"]
@jexp
Collaborator

But please use parameters ! for cypher.

batch_result = @neo.batch 
    [:create_unique_node, "person", "ssn", "000-00-0003", {:first_name=>"Jane"}], 
    [:create_unique_node, "person", "ssn", "000-00-0004", {:first_name=>"John"}], 
    [:execute_query, "START a = node:person(ssn={ssn1}), b = node:person(ssn={ssn2}) CREATE a -[new_rel:follows]->b RETURN new_rel", 
        {:ssn1=>"000-00-0003",:ssn2=>"000-00-0004"}
    ]
@jcbozonier
@henry74

Wow, glad I found this. I was about to file a bug with neography regarding creation of nodes through batch. Any ETA on when you think they may be fixed? While I'm glad there is a workaround - this is a pretty serious bug which has a large impact on REST API performance against a Neo4j database.

For others looking to use this workaround - here's a couple of links which helped me figure out some of the details of passing parameters and saving relationship properties:

http://docs.neo4j.org/chunked/snapshot/cypher-parameters.html
http://docs.neo4j.org/chunked/snapshot/query-create.html#create-create-a-relationship-and-set-properties

Using neography, you can pass your relationship properties as a cypher parameters in the ruby hash format.

@ashleydw

This also fails for me when I try to create a unique index:

[:add_node_to_index, 'nodes_index', 'type', 'repo', "{0}", true]

The first index is created, but following ones are ignored with no error

@denisj

I'm using Neo4J 1.9.1. I make a batch request on the rest API like that

[{ 
  :body => 
  {
    :properties => 
      { 
        "screen_name" => "temptalia",
        "type"=>"ACCOUNT"
      }, 
    :key=>"screen_name", 
    :value=>"temptalia"
   }, 
   :to => "/index/node/unique_nodes?unique", 
   :method => "POST", 
   :id => 0
}, 
{
  :body => true, 
  :to => "/node/{0}/properties/seed", 
  :method => PUT", 
  :id => 1
}]

I create a unique node and I want to set to it a property by referencing the previous batch result.

But that doesn't work. After some debugging in Neo4J, I found that the generated path was

/node/http://localhost:7474/db/data/index/node/unique_nodes/screen_name/temptalia/111952/properties/seed

but I'm expected something like that (according to the rest API doc)

http://localhost:7474/db/data/node/111952/properties/seed
@gcolbert

@denisj : the {0} reference is just a copy of the "to" parameter from the id=0 request. Since you have used "/node/{0}/properties/seed", this is logical that the generated path starts with /node/http://. That means that you have to start your "to" parameter with {0}.

Where I agree is that it would be more logical to be able to create unique nodes with the /node?unique endpoint directly (and not /index/node). To me, the use of a Lucene index is an implementation detail.

@denisj

Thank you for your answer. I forgot to say that I'm using Neography, which constructs these requests.

@Anonyfox

@denisj What's your solution? I have the same problem, using batch creation with Neography.

@gcolbert

Is this bug specific to Neography or is this the same thing if we use /db/data/batch directly? If this is a general problem (as I think it is), can we have a working example using /db/data/batch?

@justinoverton

It is not specific to Neography. It happens if you use batch directly.

Using Create Unique

POST: http://localhost:7474/db/data/batch 
[
    {
        "method": "POST",
        "to": "/index/node/people?uniqueness=create_or_fail",
        "id": 0,
        "body": {
            "key": "name",
            "value": "Tobias",
            "properties": {
                "name": "Tobias",
                "sequence": 1
            }
        }
    },
    {
        "method": "POST",
        "to": "/node",
        "id": 1,
        "body": {
            "foo": "bar"
        }
    },
    {
        "method": "POST",
        "to": "{0}/relationships",
        "id": 2,
        "body": {
            "to": "{1}",
            "type": "TESTREL"
        }
    }
]

Output:

{
    "message": "",
    "exception": "BatchOperationFailedException",
    "fullname": "org.neo4j.server.rest.domain.BatchOperationFailedException",
    "stacktrace": ["org.neo4j.server.rest.batch.NonStreamingBatchOperations.invoke(NonStreamingBatchOperations.java:63)", "org.neo4j.server.rest.batch.BatchOperations.performRequest(BatchOperations.java:188)", "org.neo4j.server.rest.batch.BatchOperations.parseAndPerform(BatchOperations.java:159)", "org.neo4j.server.rest.batch.NonStreamingBatchOperations.performBatchJobs(NonStreamingBatchOperations.java:48)", "org.neo4j.server.rest.web.BatchOperationService.batchProcess(BatchOperationService.java:117)", "org.neo4j.server.rest.web.BatchOperationService.performBatchOperations(BatchOperationService.java:72)", "java.lang.reflect.Method.invoke(Method.java:601)", "org.neo4j.server.rest.security.SecurityFilter.doFilter(SecurityFilter.java:112)"]
}

Without Create Unique

POST: http://localhost:7474/db/data/batch
[
    {
        "method": "POST",
        "to": "/node",
        "id": 0,
        "body": {
                "name": "Tobias",
                "sequence": 1
        }
    },
    {
        "method": "POST",
        "to": "/node",
        "id": 1,
        "body": {
            "foo": "bar"
        }
    },
    {
        "method": "POST",
        "to": "{0}/relationships",
        "id": 2,
        "body": {
            "to": "{1}",
            "type": "TESTREL"
        }
    }
]

Output:

[{
    "id": 0,
    "location": "http://localhost:7474/db/data/node/18322",
    "body": {
        "extensions": {},
        "paged_traverse": "http://localhost:7474/db/data/node/18322/paged/traverse/{returnType}{?pageSize,leaseTime}",
        "outgoing_relationships": "http://localhost:7474/db/data/node/18322/relationships/out",
        "all_typed_relationships": "http://localhost:7474/db/data/node/18322/relationships/all/{-list|&|types}",
        "traverse": "http://localhost:7474/db/data/node/18322/traverse/{returnType}",
        "property": "http://localhost:7474/db/data/node/18322/properties/{key}",
        "all_relationships": "http://localhost:7474/db/data/node/18322/relationships/all",
        "self": "http://localhost:7474/db/data/node/18322",
        "outgoing_typed_relationships": "http://localhost:7474/db/data/node/18322/relationships/out/{-list|&|types}",
        "properties": "http://localhost:7474/db/data/node/18322/properties",
        "incoming_relationships": "http://localhost:7474/db/data/node/18322/relationships/in",
        "incoming_typed_relationships": "http://localhost:7474/db/data/node/18322/relationships/in/{-list|&|types}",
        "create_relationship": "http://localhost:7474/db/data/node/18322/relationships",
        "data": {
            "sequence": 1,
            "name": "Tobias"
        }
    },
    "from": "/node"
}, {
    "id": 1,
    "location": "http://localhost:7474/db/data/node/18323",
    "body": {
        "extensions": {},
        "paged_traverse": "http://localhost:7474/db/data/node/18323/paged/traverse/{returnType}{?pageSize,leaseTime}",
        "outgoing_relationships": "http://localhost:7474/db/data/node/18323/relationships/out",
        "all_typed_relationships": "http://localhost:7474/db/data/node/18323/relationships/all/{-list|&|types}",
        "traverse": "http://localhost:7474/db/data/node/18323/traverse/{returnType}",
        "property": "http://localhost:7474/db/data/node/18323/properties/{key}",
        "all_relationships": "http://localhost:7474/db/data/node/18323/relationships/all",
        "self": "http://localhost:7474/db/data/node/18323",
        "outgoing_typed_relationships": "http://localhost:7474/db/data/node/18323/relationships/out/{-list|&|types}",
        "properties": "http://localhost:7474/db/data/node/18323/properties",
        "incoming_relationships": "http://localhost:7474/db/data/node/18323/relationships/in",
        "incoming_typed_relationships": "http://localhost:7474/db/data/node/18323/relationships/in/{-list|&|types}",
        "create_relationship": "http://localhost:7474/db/data/node/18323/relationships",
        "data": {
            "foo": "bar"
        }
    },
    "from": "/node"
}, {
    "id": 2,
    "location": "http://localhost:7474/db/data/relationship/43814",
    "body": {
        "extensions": {},
        "start": "http://localhost:7474/db/data/node/18322",
        "property": "http://localhost:7474/db/data/relationship/43814/properties/{key}",
        "self": "http://localhost:7474/db/data/relationship/43814",
        "properties": "http://localhost:7474/db/data/relationship/43814/properties",
        "type": "TESTREL",
        "end": "http://localhost:7474/db/data/node/18323",
        "data": {}
    },
    "from": "http://localhost:7474/db/data/node/18322/relationships"
}]
@gcolbert

@justinoverton : I see above (see @maxdemarzi and @jexp answers) that there is a workaround for people using Neography (workaround is to use Cypher). I'm wondering if there is such a workaround for people not using Neography.

@justinoverton

@gcolbert : You can still use cypher as a workaround using the RestAPI directly. That's what I ended up doing. Though it would be nice if the REST API supported it directly.

@gcolbert

@justinoverton : you mean that you end up using /db/data/cypher? But it doesn't allow batch mode, does it?
Looking at the documentation :
http://docs.neo4j.org/chunked/stable/rest-api-cypher.html <-- nothing about batch insertions here
http://docs.neo4j.org/chunked/stable/rest-api-batch-ops.html <-- nothing about cypher here

@justinoverton

@gcolbert I'll update your question on Stack Overflow as well.... here is what you want. You want the Cypher as part of the batch.

[
    {"method":"POST","to":"/index/node/concept?uniqueness=get_or_create","id":0,"body":{"key":"nom", "value":"organisation", "properties": {"nom":"organisation"}}},
    {"method":"POST","to":"/index/node/concept?uniqueness=get_or_create","id":1,"body":{"key":"nom", "value":"établissement", "properties": {"nom":"établissement"}}},
    {
        "method": "POST",
        "to": "/cypher",
        "id": 1,
        "body": {
          "query" : "START a=node:concept(nom={aVal}), b=node:concept(nom={bVal}) CREATE b-[r:est]->a RETURN a, b, r",
          "params" : {
            "aVal" : "établissement",
            "bVal" : "organisation"
          }
        }
    }
]

Output:

[{
    "id": 0,
    "location": "http://localhost:7474/db/data/index/node/concept/nom/organisation/18688",
    "body": {
        "extensions": {},
        "paged_traverse": "http://localhost:7474/db/data/node/18688/paged/traverse/{returnType}{?pageSize,leaseTime}",
        "outgoing_relationships": "http://localhost:7474/db/data/node/18688/relationships/out",
        "all_typed_relationships": "http://localhost:7474/db/data/node/18688/relationships/all/{-list|&|types}",
        "traverse": "http://localhost:7474/db/data/node/18688/traverse/{returnType}",
        "property": "http://localhost:7474/db/data/node/18688/properties/{key}",
        "all_relationships": "http://localhost:7474/db/data/node/18688/relationships/all",
        "self": "http://localhost:7474/db/data/node/18688",
        "outgoing_typed_relationships": "http://localhost:7474/db/data/node/18688/relationships/out/{-list|&|types}",
        "properties": "http://localhost:7474/db/data/node/18688/properties",
        "incoming_relationships": "http://localhost:7474/db/data/node/18688/relationships/in",
        "incoming_typed_relationships": "http://localhost:7474/db/data/node/18688/relationships/in/{-list|&|types}",
        "create_relationship": "http://localhost:7474/db/data/node/18688/relationships",
        "data": {
            "nom": "organisation"
        },
        "indexed": "http://localhost:7474/db/data/index/node/concept/nom/organisation/18688"
    },
    "from": "/index/node/concept?uniqueness=get_or_create"
}, {
    "id": 1,
    "location": "http://localhost:7474/db/data/index/node/concept/nom/%EF%BF%BDtablissement/18689",
    "body": {
        "extensions": {},
        "paged_traverse": "http://localhost:7474/db/data/node/18689/paged/traverse/{returnType}{?pageSize,leaseTime}",
        "outgoing_relationships": "http://localhost:7474/db/data/node/18689/relationships/out",
        "all_typed_relationships": "http://localhost:7474/db/data/node/18689/relationships/all/{-list|&|types}",
        "traverse": "http://localhost:7474/db/data/node/18689/traverse/{returnType}",
        "property": "http://localhost:7474/db/data/node/18689/properties/{key}",
        "all_relationships": "http://localhost:7474/db/data/node/18689/relationships/all",
        "self": "http://localhost:7474/db/data/node/18689",
        "outgoing_typed_relationships": "http://localhost:7474/db/data/node/18689/relationships/out/{-list|&|types}",
        "properties": "http://localhost:7474/db/data/node/18689/properties",
        "incoming_relationships": "http://localhost:7474/db/data/node/18689/relationships/in",
        "incoming_typed_relationships": "http://localhost:7474/db/data/node/18689/relationships/in/{-list|&|types}",
        "create_relationship": "http://localhost:7474/db/data/node/18689/relationships",
        "data": {
            "nom": "�tablissement"
        },
        "indexed": "http://localhost:7474/db/data/index/node/concept/nom/%EF%BF%BDtablissement/18689"
    },
    "from": "/index/node/concept?uniqueness=get_or_create"
}, {
    "id": 1,
    "body": {
        "columns": ["a", "b", "r"],
        "data": [
            [{
                "paged_traverse": "http://localhost:7474/db/data/node/18689/paged/traverse/{returnType}{?pageSize,leaseTime}",
                "outgoing_relationships": "http://localhost:7474/db/data/node/18689/relationships/out",
                "data": {
                    "nom": "�tablissement"
                },
                "traverse": "http://localhost:7474/db/data/node/18689/traverse/{returnType}",
                "all_typed_relationships": "http://localhost:7474/db/data/node/18689/relationships/all/{-list|&|types}",
                "self": "http://localhost:7474/db/data/node/18689",
                "all_relationships": "http://localhost:7474/db/data/node/18689/relationships/all",
                "property": "http://localhost:7474/db/data/node/18689/properties/{key}",
                "properties": "http://localhost:7474/db/data/node/18689/properties",
                "outgoing_typed_relationships": "http://localhost:7474/db/data/node/18689/relationships/out/{-list|&|types}",
                "incoming_relationships": "http://localhost:7474/db/data/node/18689/relationships/in",
                "incoming_typed_relationships": "http://localhost:7474/db/data/node/18689/relationships/in/{-list|&|types}",
                "extensions": {},
                "create_relationship": "http://localhost:7474/db/data/node/18689/relationships"
            }, {
                "paged_traverse": "http://localhost:7474/db/data/node/18688/paged/traverse/{returnType}{?pageSize,leaseTime}",
                "outgoing_relationships": "http://localhost:7474/db/data/node/18688/relationships/out",
                "data": {
                    "nom": "organisation"
                },
                "traverse": "http://localhost:7474/db/data/node/18688/traverse/{returnType}",
                "all_typed_relationships": "http://localhost:7474/db/data/node/18688/relationships/all/{-list|&|types}",
                "self": "http://localhost:7474/db/data/node/18688",
                "all_relationships": "http://localhost:7474/db/data/node/18688/relationships/all",
                "property": "http://localhost:7474/db/data/node/18688/properties/{key}",
                "properties": "http://localhost:7474/db/data/node/18688/properties",
                "outgoing_typed_relationships": "http://localhost:7474/db/data/node/18688/relationships/out/{-list|&|types}",
                "incoming_relationships": "http://localhost:7474/db/data/node/18688/relationships/in",
                "incoming_typed_relationships": "http://localhost:7474/db/data/node/18688/relationships/in/{-list|&|types}",
                "extensions": {},
                "create_relationship": "http://localhost:7474/db/data/node/18688/relationships"
            }, {
                "start": "http://localhost:7474/db/data/node/18688",
                "data": {},
                "self": "http://localhost:7474/db/data/relationship/44187",
                "property": "http://localhost:7474/db/data/relationship/44187/properties/{key}",
                "properties": "http://localhost:7474/db/data/relationship/44187/properties",
                "type": "est",
                "extensions": {},
                "end": "http://localhost:7474/db/data/node/18689"
            }]
        ]
    },
    "from": "/cypher"
}]
@gcolbert

Ah, thank you @justinoverton :+1: I guess the documentation should give such an example.

@jakewins
Collaborator

Note that the last example uses get_or_create uniqueness while the original example used create_or_fail, which is why this did not work. get_or_create was introduced to cater to precisely this use case.

For people coming later to this ticket: The preferred method today is to use cypher exclusively, along with the MERGE keyword and the transactional endpoint.

This approach is still useful, however, if you are using legacy indexes.

@jakewins jakewins closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.