Add project to Spark Packages

[conker84]

Add the project to spark-pakages in order to easly integrate it in spark applications: (i.e.)

> $SPARK_HOME/bin/spark-shell --packages neo4j-contrib:neo4j-spark-connector:0.1.0

[zhiboz]

As neo4j-spark-connector hit spark-packages.org lately, just wonder roughly when we would be able to run

spark-shell --packages neo4j-contrib:neo4j-spark-connector:0.1.0

[conker84]

@zhiboz i completly agree with you. This is what i meant with my issue

[jexp]

Ah, now I understand :)

Unfortunately it's not so easy to release it to maven central as graphframes is not on maven central. :(

I would have to setup bintray deployment or figure out another way.

[jexp]

Also I wanted to wait with the first release until I got some initial feedback from ppl. on how well it works for them.

[conker84]

@jexp as a first step you can think to jitpack (if it works with spark packages), :) what do you think?

[jexp]

released to: https://m2.neo4j.org/content/repositories/releases/org/neo4j/spark/

please note the 2 versions for the 2 scala versions

https://github.com/neo4j-contrib/neo4j-spark-connector/releases/tag/1.0.0-RC1

[jexp]

this should work now, could you try it?

bin/spark-shell \
--packages org.neo4j.spark:neo4j-spark-connector_2.10:1.0.0-RC1,graphframes:graphframes:0.1.0-spark1.6 \
--repositories https://m2.neo4j.org/content/repositories/releases,http://dl.bintray.com/spark-packages/maven

[zhiboz]

It appears the library can be loaded properly now, thanks!

When I tried to follow along the example in Neo4jGraphFrame, against a local neo4j v3.0.0 instance with auth off, I encounted the following error:

org.neo4j.driver.v1.exceptions.ClientException: Invalid input '{': expected an identifier character, whitespace, '?', '!', '.', node labels, '[', "=~", IN, STARTS, ENDS, CONTAINS, IS, '^', '*', '/', '%', '+', '-', '=', "<>", "!=", '<', '>', "<=", ">=", AND, XOR, OR, AS, ',', ORDER, SKIP, LIMIT, LOAD CSV, START, MATCH, UNWIND, MERGE, CREATE, SET, DELETE, REMOVE, FOREACH, WITH, CALL, RETURN, UNION, ';' or end of input (line 1, column 102 (offset: 101))
"MATCH (n:Person)-[r:KNOWS]->(m:Person) RETURN id(n) as src, id(m) as dst, r.${edge._2} as prop"
                                                                                               ^
    at org.neo4j.driver.internal.connector.socket.SocketResponseHandler.handleFailureMessage(SocketResponseHandler.java:68)
    at org.neo4j.driver.internal.messaging.PackStreamMessageFormatV1$Reader.unpackFailureMessage(PackStreamMessageFormatV1.java:456)
    at org.neo4j.driver.internal.messaging.PackStreamMessageFormatV1$Reader.read(PackStreamMessageFormatV1.java:417)
    at org.neo4j.driver.internal.connector.socket.SocketClient.receiveOne(SocketClient.java:128)
    at org.neo4j.driver.internal.connector.socket.SocketConnection.receiveOne(SocketConnection.java:135)
    at org.neo4j.driver.internal.connector.ConcurrencyGuardingConnection.receiveOne(ConcurrencyGuardingConnection.java:150)
    at org.neo4j.driver.internal.pool.PooledConnection.receiveOne(PooledConnection.java:142)
    at org.neo4j.driver.internal.InternalStatementResult.tryFetchNext(InternalStatementResult.java:303)
    at org.neo4j.driver.internal.InternalStatementResult.hasNext(InternalStatementResult.java:181)
    at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:88)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:86)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Anything I missed? Thanks!

[jexp]

Sorry, the setup was changed from "none" to null to skip a property. Will fix the readme.

[jexp]

After much fiddling I also got it released to spark-packages, so you should be able to run this now:

bin/spark-shell \
--packages neo4j-contrib:neo4j-spark-connector:1.0.0-RC1,graphframes:graphframes:0.1.0-spark1.6