Neo4j export #91

darabos · 2020-09-30T22:00:56Z

This is part 1: exporting attributes for existing nodes.

There's an option to set node.keys and let them build the query. But if I use that, the label is a must. If I write the same query manually, I can leave it off. (http://5.9.211.195:8000/neo4j-spark-docs/1.0.0/writing.html#bookmark-write-node)

Open tasks:

Make sure this works if the keys are not defined everywhere.
Attribute export for edges.
Edge export for existing nodes. (I don't think this is important.)
Export whole graph as new stuff.
Documentation.
Tests. (Maybe when the final Neo4j Spark Connector is released.)

xandrew-lynx

👍

So if key is not actually a key, then what happens? We will set the parameter for all nodes that match on they "keys", right? I'm not necessarily saying this is bad, just making sure I understand what's going on.

xandrew-lynx · 2020-10-01T07:52:19Z

app/com/lynxanalytics/biggraph/graph_operations/ExportToNeo4j.scala

+    implicit val sd = rc.sparkDomain
+    val df = inputs.t.df
+    val keyMatch = keys.map(k => s"`$k`: event.`$k`").mkString(", ")
+    val query = s"MERGE (n$labels {$keyMatch}) SET n += event"


Is event a cypher reserved word here? It refers to the actual record streaming in?

Is event a cypher reserved word here? It refers to the actual record streaming in?

Yes, it's the actual record. The connector adds UNWIND $events AS event before the query we specify. I'm not sure what is reserved, but that's a part outside of our control at least.

darabos · 2020-10-01T08:05:26Z

So if key is not actually a key, then what happens? We will set the parameter for all nodes that match on they "keys", right? I'm not necessarily saying this is bad, just making sure I understand what's going on.

I think so. I'll be sure to note this in the docs. I think it's just how Neo4j updates work. (Much like SQL updates.)

darabos · 2020-10-01T08:51:06Z

So if key is not actually a key, then what happens? We will set the parameter for all nodes that match on they "keys", right? I'm not necessarily saying this is bad, just making sure I understand what's going on.

The Neo4j import creates an <id> field, which would make an obvious key. But since the ID is not a property, it does not work. I've checked how it can be made to work — it cannot. As https://stackoverflow.com/questions/27729489/neo4j-merge-statement-with-id and the linked answer you must have your hand-crafted keys in Neo4j.

darabos · 2020-10-01T13:26:41Z

Isn't that a joy to see? 😄 I got this with gender and comment selected as node label and relationship type. If you omit one or both:

PTAL

darabos · 2020-10-01T13:33:05Z

app/com/lynxanalytics/biggraph/graph_operations/ExportToNeo4j.scala

+      """)
+    } else {
+      neo.send(vs, s"""
+        CALL apoc.create.node(split(event.`$nodeLabelsColumn`, ','), event) YIELD node


See https://neo4j.com/labs/apoc/4.1/overview/apoc.create/apoc.create.node/.
This is the way to create a node with a dynamically picked label. Requires apoc to be installed.

darabos · 2020-10-01T13:33:26Z

app/com/lynxanalytics/biggraph/graph_operations/ExportToNeo4j.scala

+    } else {
+      neo.send(es, s"""
+      MATCH (src {`$VID`: event.`$SRCID`}), (dst {`$VID`: event.`$DSTID`})
+      CALL apoc.create.relationship(src, event.`$relationshipTypeColumn`, event, dst) YIELD rel


See https://neo4j.com/labs/apoc/4.1/overview/apoc.create/apoc.create.relationship/.
Same deal.

darabos · 2020-10-01T13:34:39Z

app/com/lynxanalytics/biggraph/graph_operations/ExportToNeo4j.scala

+    // Prefix the internal IDs with this operation's GUID so different exports don't collide.
+    val guid = F.lit(this.gUID.toString + "-")


Does this make sense?

Yeah, I think this is perfect! But what about adding GUID as vertex and edge attribute as well? That would allow users to clean up results of somehow screwed up exports with relative ease. Maybe we can add export date as well?

Yeah, I think this is perfect! But what about adding GUID as vertex and edge attribute as well? That would allow users to clean up results of somehow screwed up exports with relative ease. Maybe we can add export date as well?

Actually the timestamp is better than the GUID. Maybe you've recreated your LynxKite instance. Maybe two LynxKite instances use the same Neo4j instance. If everyone exports the example graph, you can end up with two exports with the same GUID quick enough.

I'll switch to just using the timestamp. Easier to interpret too.

xandrew-lynx

This looks great, thanks a lot!

I only added some nits.

But I wouldn't delay the testing until after the release. If things go well, this will be a very high profile feature. I understand that this is substantial work because we somehow have to get a testing neo4j instance up, so this will probably remove our chances for a release today, but still. Or is this going to be simpler after the final connector is released somehow?

app/com/lynxanalytics/biggraph/frontend_operations/ExportOperations.scala

xandrew-lynx · 2020-10-02T06:30:41Z

app/com/lynxanalytics/biggraph/frontend_operations/ExportOperations.scala

+      import Operation.Implicits._
+      params ++= List(
+        Param("labels", "Relationship labels", defaultValue = ""),
+        Choice("keys", "Attribute to use as key",


Suggested change

Choice("keys", "Attribute to use as key",

Choice("keys", "Attributes to use as key",

app/com/lynxanalytics/biggraph/frontend_operations/ExportOperations.scala

xandrew-lynx · 2020-10-02T06:39:25Z

app/com/lynxanalytics/biggraph/graph_operations/ExportToNeo4j.scala

+    url: String, username: String, password: String, labels: String, keys: Seq[String],
+    version: Long, nodesOrRelationships: String)
+  extends SparkOperation[ExportAttributesToNeo4j.Input, ExportAttributesToNeo4j.Output] {
+  val neo = Neo4jConnection(url, username, password)


When I first saw this (and of course didn't realize I just reviewed the definition of this class :( ) I thought why would you put a connection kind of class up here, why not create the connection only when needed, in execute. But of course, there is no connection being made here, it just collects some metadata which will be used in a helper method of this object. So then what about naming it differently? E.g. Neo4JHelper?

When I first saw this (and of course didn't realize I just reviewed the definition of this class :( ) I thought why would you put a connection kind of class up here, why not create the connection only when needed, in execute. But of course, there is no connection being made here, it just collects some metadata which will be used in a helper method of this object. So then what about naming it differently? E.g. Neo4JHelper?

I changed it to Neo4jConnectionParameters.

xandrew-lynx · 2020-10-02T06:47:18Z

app/com/lynxanalytics/biggraph/graph_operations/ExportToNeo4j.scala

+    // Prefix the internal IDs with this operation's GUID so different exports don't collide.
+    val guid = F.lit(this.gUID.toString + "-")


Yeah, I think this is perfect! But what about adding GUID as vertex and edge attribute as well? That would allow users to clean up results of somehow screwed up exports with relative ease. Maybe we can add export date as well?

xandrew-lynx · 2020-10-02T06:53:13Z

app/com/lynxanalytics/biggraph/frontend_operations/ExportOperations.scala

+        Param("username", "Neo4j username", defaultValue = "neo4j"),
+        Param("password", "Neo4j password", defaultValue = "neo4j"),
+        NonNegInt("version", "Snapshot revision ID", default = 1),
+        Choice("node_labels", "Attribute with node labels",


Mention that the value of this attribute can be a comma separated list of label values if user wants multiple labels. Would be more elegant to allow vectors here, but probably now is not the time.

Mention that the value of this attribute can be a comma separated list of label values if user wants multiple labels. Would be more elegant to allow vectors here, but probably now is not the time.

I expect you usually just use one label. Described the commas in the help.

xandrew-lynx · 2020-10-02T07:00:37Z

On the screenshot: indeed feels cool! :) And if you just export twice, you will simply see two copies, right?

darabos · 2020-10-02T07:39:04Z

But I wouldn't delay the testing until after the release. If things go well, this will be a very high profile feature. I understand that this is substantial work because we somehow have to get a testing neo4j instance up, so this will probably remove our chances for a release today, but still. Or is this going to be simpler after the final connector is released somehow?

There's a whole Neo4j test library which we were using for the old import. Instead of starting a full Neo4j instance I think it sets something up in-process. Now we can't use it because it's an old version. I'll look around to see if there is a replacement.

xandrew-lynx · 2020-10-02T08:21:09Z

On the testing: yes, please look around. Of course, if the conclusion is that now it's hard/ugly but in the not-too-far future we a see good solution on the horizon then sure, we can wait of it.

darabos

Thanks for the comments! You can review the docs now and I'll try to do the tests.

app/com/lynxanalytics/biggraph/frontend_operations/ExportOperations.scala

darabos · 2020-10-02T09:26:14Z

app/com/lynxanalytics/biggraph/frontend_operations/ExportOperations.scala

+        Param("username", "Neo4j username", defaultValue = "neo4j"),
+        Param("password", "Neo4j password", defaultValue = "neo4j"),
+        NonNegInt("version", "Snapshot revision ID", default = 1),
+        Choice("node_labels", "Attribute with node labels",


Mention that the value of this attribute can be a comma separated list of label values if user wants multiple labels. Would be more elegant to allow vectors here, but probably now is not the time.

I expect you usually just use one label. Described the commas in the help.

darabos · 2020-10-02T09:27:28Z

app/com/lynxanalytics/biggraph/graph_operations/ExportToNeo4j.scala

+    url: String, username: String, password: String, labels: String, keys: Seq[String],
+    version: Long, nodesOrRelationships: String)
+  extends SparkOperation[ExportAttributesToNeo4j.Input, ExportAttributesToNeo4j.Output] {
+  val neo = Neo4jConnection(url, username, password)


When I first saw this (and of course didn't realize I just reviewed the definition of this class :( ) I thought why would you put a connection kind of class up here, why not create the connection only when needed, in execute. But of course, there is no connection being made here, it just collects some metadata which will be used in a helper method of this object. So then what about naming it differently? E.g. Neo4JHelper?

I changed it to Neo4jConnectionParameters.

xandrew-lynx · 2020-10-02T12:04:07Z

web/app/help/operations/export-edge-attributes-to-neo4j.asciidoc

+The relationships in Neo4j are identified by a key property (or properties).
+You must have a corresponding edge attribute in LynxKite by the same name.
+This will be used to find the right relationship to update in Neo4j.
+


I think we need to explicitly spell out what happens if multiple edges match a given key value combination
a) in Neo4j
b) in LynxKite

I think we need to explicitly spell out what happens if multiple edges match a given key value combination
a) in Neo4j
b) in LynxKite

Done. I've also added the full Cypher query to the docs. I imagine this may be of interest to advanced Neo4j users and it's not too scary.

xandrew-lynx · 2020-10-02T12:06:25Z

web/app/help/operations/export-graph-to-neo4j.asciidoc

+[p-node_labels]#Node labels#::
+A string vertex attribute that is a comma-separated list of labels to apply to the newly
+created nodes. Optional. You must have https://neo4j.com/developer/neo4j-apoc/[Neo4j APOC]
+installed on the Neo4j instance to use this.


If omitted, the nodes will have no label, right? Is this acceptable in Neo4J then?

If omitted, the nodes will have no label, right? Is this acceptable in Neo4J then?

Surprisingly, yes.

xandrew-lynx · 2020-10-02T12:07:56Z

web/app/help/operations/export-graph-to-neo4j.asciidoc

+[p-relationship_type]#Attribute with relationship type#::
+A string edge attribute that specifies the relationship type for each newly created relationship.
+Optional. You must have https://neo4j.com/developer/neo4j-apoc/[Neo4j APOC]
+installed on the Neo4j instance to use this.


Interesting. So edges can have only one relationship type, but vertices can have multiple labels? Sorry for bothering you with basic neo4j questions..

Interesting. So edges can have only one relationship type, but vertices can have multiple labels? Sorry for bothering you with basic neo4j questions..

Yes. Edges must always have a type, and they can only have a single type. Nodes can have multiple labels or no labels. It's pretty clear that you didn't design this system. 😄

xandrew-lynx

Nice! These tests ended up being very pretty and readable!

darabos · 2020-10-02T15:45:46Z

Nice! These tests ended up being very pretty and readable!

They are actually surprisingly fast too. Around 10-30 seconds. 👍 https://www.testcontainers.org/

First stab at Neo4j export.

b9f2081

xandrew-lynx reviewed Oct 1, 2020

View reviewed changes

Base automatically changed from darabos-neo4j to master October 1, 2020 07:59

darabos added 2 commits October 1, 2020 13:22

Skip null keys and hide special attributes.

8b6714c

Full graph export.

c0d9558

darabos commented Oct 1, 2020

View reviewed changes

xandrew-lynx reviewed Oct 2, 2020

View reviewed changes

darabos added 2 commits October 2, 2020 10:04

Rename Neo4jConnection to Neo4jConnectionParameters.

80dc844

Use timestamp instead of GUID.

7abcbd7

darabos added 2 commits October 2, 2020 10:21

Add return type.

234f057

Docs for Neo4j export. "Attribute(s)"

9f7225d

darabos commented Oct 2, 2020

View reviewed changes

xandrew-lynx approved these changes Oct 2, 2020

View reviewed changes

Tests for Neo4j import and export.

e9cf1c5

darabos changed the title ~~[WIP] Neo4j export~~ Neo4j export Oct 2, 2020

xandrew-lynx approved these changes Oct 2, 2020

View reviewed changes

darabos added 2 commits October 2, 2020 17:21

Include the Cypher queries in the help.

faabd7e

Test for APOC-based codepath.

6f0df20

darabos merged commit c2d3b69 into master Oct 2, 2020

darabos deleted the darabos-neo4j-export branch October 2, 2020 17:21

darabos mentioned this pull request Oct 14, 2020

Add new import/export operations to Python API #99

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Neo4j export #91

Neo4j export #91

darabos commented Sep 30, 2020 •

edited

xandrew-lynx left a comment

xandrew-lynx Oct 1, 2020

darabos Oct 1, 2020

darabos commented Oct 1, 2020

darabos commented Oct 1, 2020

darabos commented Oct 1, 2020

darabos Oct 1, 2020

darabos Oct 1, 2020

darabos Oct 1, 2020

xandrew-lynx Oct 2, 2020

darabos Oct 2, 2020

xandrew-lynx Oct 2, 2020

xandrew-lynx left a comment

xandrew-lynx Oct 2, 2020

xandrew-lynx Oct 2, 2020

darabos Oct 2, 2020

xandrew-lynx Oct 2, 2020

xandrew-lynx Oct 2, 2020

darabos Oct 2, 2020

xandrew-lynx commented Oct 2, 2020

darabos commented Oct 2, 2020

xandrew-lynx commented Oct 2, 2020

darabos left a comment

darabos Oct 2, 2020

darabos Oct 2, 2020

xandrew-lynx Oct 2, 2020

darabos Oct 2, 2020

xandrew-lynx Oct 2, 2020

darabos Oct 2, 2020

xandrew-lynx Oct 2, 2020

darabos Oct 2, 2020

xandrew-lynx left a comment

darabos commented Oct 2, 2020

		// Prefix the internal IDs with this operation's GUID so different exports don't collide.
		val guid = F.lit(this.gUID.toString + "-")

	Choice("keys", "Attribute to use as key",
	Choice("keys", "Attributes to use as key",

Neo4j export #91

Neo4j export #91

Conversation

darabos commented Sep 30, 2020 • edited

xandrew-lynx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darabos commented Oct 1, 2020

darabos commented Oct 1, 2020

darabos commented Oct 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xandrew-lynx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xandrew-lynx commented Oct 2, 2020

darabos commented Oct 2, 2020

xandrew-lynx commented Oct 2, 2020

darabos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xandrew-lynx left a comment

Choose a reason for hiding this comment

darabos commented Oct 2, 2020

darabos commented Sep 30, 2020 •

edited