Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Neo4j - Java Heap Space Error - TraversalDescription.traverse(Node); #32

Closed
blkdog opened this Issue · 21 comments

2 participants

@blkdog

If maxDepth is 3 code runs fine, if maxDepth = 4 this code hangs @ the call to td.traverse(start) (~line 38) & then I get:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515) at java.lang.StringBuilder.append(StringBuilder.java:189) at com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:172) at com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:157) at com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.readFromAsString(AbstractMessageReaderWriterProvider.java:114) at com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:73) at com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:58) at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:552) at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:505) at org.neo4j.rest.graphdb.RequestResult.extractFrom(RequestResult.java:87) at org.neo4j.rest.graphdb.ExecutingRestRequest.post(ExecutingRestRequest.java:138) at org.neo4j.rest.graphdb.ExecutingRestAPI.traverse(ExecutingRestAPI.java:486) at org.neo4j.rest.graphdb.RestAPIFacade.traverse(RestAPIFacade.java:202) at org.neo4j.rest.graphdb.traversal.RestTraversal.traverse(RestTraversal.java:162) at com.tr.cmg.alloy.dao.psd.RelationalPathSearchDAO.executeRelationalPathSearch(RelationalPathSearchDAO.java:38) at com.tr.cmg.alloy.dao.psd.RelationalPathSearchDAO.main(RelationalPathSearchDAO.java:115)

code ...

package com.tr.cmg.alloy.dao.psd;

import java.util.ArrayList;

import org.neo4j.graphdb.Direction;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.Path;
import org.neo4j.graphdb.Relationship;
import org.neo4j.graphdb.traversal.TraversalDescription;
import org.neo4j.rest.graphdb.index.RestIndex;
import org.neo4j.rest.graphdb.traversal.RestTraversalDescription;

import com.tr.cmg.alloy.aspects.audit.DaoTierAuditAspect;
import com.tr.cmg.alloy.dao.psd.relationship.RelationType;
import com.tr.cmg.alloy.dao.support.neo4j.Neo4jDaoSupport;
import com.tr.cmg.alloy.domain.psd.NodeType;
import com.tr.cmg.alloy.services.psd.rps.RelationalPathSearch;
import com.tr.cmg.alloy.services.psd.rps.RelationalPathSearchResult;

public class RelationalPathSearchDAO extends Neo4jDaoSupport{

public RelationalPathSearchResult executeRelationalPathSearch(
RelationalPathSearch rps) {

RestIndex<Node> personIndex = API.getIndex(NodeType.PERSON.name());

Node start = personIndex.get("KEY", rps.getPerson1PermId()).getSingle();
Node end = personIndex.get("KEY", rps.getPerson2PermId()).getSingle();

Iterable<Node> nodes = null;
Iterable<Relationship> relationships = null;

if (start != null && end != null) {

    TraversalDescription td = createTraversalDescription(rps);
    nodes = null;
    relationships = null;
    for (Path position : td.traverse(start)) {

        // Log audit record info
        DaoTierAuditAspect
                .storeSqlStatement("Relational Path Neo4j Query: "
                        + rps.getPerson1PermId() + " -> "
                        + rps.getPerson2PermId());
        DaoTierAuditAspect.storeSqlParameters("");

        System.out.println(position);

        if (position.endNode().equals(end)) {
            nodes = position.nodes();
            relationships = position.relationships();
            break;
        }
    }

    if(nodes == null) {
        ArrayList<Node> x = new ArrayList<Node>();
        x.add(start);
        x.add(end);
        nodes = x;
    }
}

return new RelationalPathSearchResult(nodes, relationships);

}

private TraversalDescription createTraversalDescription(RelationalPathSearch rps) {

RestTraversalDescription rtd = (RestTraversalDescription) API.createTraversalDescription();
rtd.maxDepth(4);

return  rtd
        .depthFirst()
        .relationships(RelationType.IsDirector, Direction.BOTH)
        .relationships(RelationType.HasGraduatedFrom, Direction.BOTH)
        .relationships(RelationType.IsOfficer, Direction.BOTH)
        .relationships(RelationType.IsUnknownOfficerDirectorAt, Direction.BOTH)
        .relationships(RelationType.IsInsiderAt, Direction.BOTH)
        .relationships(RelationType.IsEquityAnalystAt, Direction.BOTH)
        .relationships(RelationType.IsMemberOrChairmanOf, Direction.BOTH)
        .relationships(RelationType.IsChiefInvestmentOfficerAt, Direction.BOTH)
        .relationships(RelationType.IsDirectorOfResearchAt, Direction.BOTH)
        .relationships(RelationType.IsPortfolioManagerAt, Direction.BOTH)
        .relationships(RelationType.IsTraderAt, Direction.BOTH)
        .relationships(RelationType.IsEconomistAt, Direction.BOTH)
        .relationships(RelationType.IsSalesProfessionalAt, Direction.BOTH)
        .relationships(RelationType.IsStrategistAt, Direction.BOTH)
        .relationships(RelationType.IsExecutiveOfficerAt, Direction.BOTH)
        .relationships(RelationType.IsShariahSupervisoryAt, Direction.BOTH)
        .relationships(RelationType.IsNonExecutiveAt, Direction.BOTH)
        .relationships(RelationType.IsVEExecutive, Direction.BOTH)
        .relationships(RelationType.IsProvidingResearchOn, Direction.BOTH)
        .relationships(RelationType.IsAnalystAt, Direction.BOTH)
        .relationships(RelationType.IsReuterMessengerUserAt, Direction.BOTH)
        .relationships(RelationType.IsProvidingAssetClassCoverageOn, Direction.BOTH)
        .relationships(RelationType.IsSpeakingLanguageOf, Direction.BOTH)
        .relationships(RelationType.IsProvidingGeographyCoverageOf, Direction.BOTH)
        .relationships(RelationType.IsUnknownSellSideBuySideAt, Direction.BOTH)
        .relationships(RelationType.IsUnknownInsiderAt, Direction.BOTH)
        .relationships(RelationType.IsUnknownResearchAnalystAt, Direction.BOTH)
        .relationships(RelationType.IsUnknownVEExecutiveAt, Direction.BOTH)
        .relationships(RelationType.IsUnknownReutersMessengerUserAt, Direction.BOTH)
        .relationships(RelationType.IsUnknownStreetEventsUserAt, Direction.BOTH);

}

public static void main(String[] args) {

RelationalPathSearchDAO dao = new RelationalPathSearchDAO();

RelationalPathSearch rps = new RelationalPathSearch();
rps.setPerson1PermId("34414591164");
rps.setPerson2PermId("34414710307");

RelationalPathSearchResult result = dao.executeRelationalPathSearch(rps);
result.toJSON();

}

@jexp
Owner

Which version are you using?
How much java heap space do you use for your client?
How many nodes / paths are returned at depth 3 and how many do you expect at depth 4 ?

@blkdog
@blkdog
@jexp
Owner

Robert, using a traversal like this is like pulling the whole db into memory and across the wire and running the graph operation in your client java code. I try to look into the issue itself but I think the json document returned is just too big to fit into memory.

should probably just use a cypher query like this:

start person1=node:PERSON(KEY={id1}),person2=node:PERSON(KEY={id2}),
match path=(person1)-[:hasGraduatedFrom]->(college)<-[:hasGraduatedFrom]-(person2)
return rels(path)

parameters.put("id1", rps.getPerson1PermId());
parameters.put("id2", rps.getPerson2PermId());

restApi.query(query,parameters);

or use RestCypherQueryEngine to run your query.

HTH

Michael

@jexp
Owner

Could you do me a favor and try the following:

RestTraversalDescription td = createTraversalDescription(rps);
Map map = td.getPostData();
System.out.println(org.neo4j.rest.graphdb.JsonHelper.createJsonFrom(map)); // put it into the file data.json

RestNode start = ...
System.out.println(start.getUri()+"/traverse/fullpath") // URI 

curl -X POST -d @data.json -H accept:application/json -H content-type:application/json URI

and see how much data is returned?

You can also execute the request in java using HttpConnection but it is more effort to do.

@blkdog
@jexp
Owner

But it is AM, 2 AM here :)

@blkdog
@jexp
Owner

Sorry, it is RestTraversal which just implements RestTraversalDescription.

@blkdog
@blkdog
@jexp
Owner

Your attachment didn't make it. Perhaps you can send it to me directly, check the Neo4j google group.

Also how much memory does your server have?

And can you run these two:

this one streams the results

curl -X POST -d @data.json -H X-Stream:true -H accept:application/json -H content-type:application/json http://milona-16.int.westgroup.com:7474/db/data/node/141995/traverse/fullpath

and this which generates a smaller result:

curl -X POST -d @data.json -H X-Stream:true -H accept:application/json -H content-type:application/json http://milona-16.int.westgroup.com:7474/db/data/node/141995/traverse/node

@blkdog
@jexp
Owner

Yes that would still be great. Or you put the data.json into the issue as text. Then I can run them myself :)

@jexp
Owner

Yes that would be great ! Just to see / show how large a datafile it will become :)

@blkdog
@blkdog
@jexp
Owner
@blkdog
@jexp
Owner
@blkdog
@jexp jexp closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.