Support pyspark

[melin]

Pyspark is best supported. Algorithmic people are familiar with Python and easy to use

[harryprince]

same need +1

[wey-gu]

@melin @harryprince
I had spent some time today to figure out that pyspark is supported out of box, I will write more in doc/blog later.

/spark/bin/pyspark --driver-class-path nebula-spark-connector-3.0.0.jar --jars nebula-spark-connector-3.0.0.jar

df = spark.read.format(
    "com.vesoft.nebula.connector.NebulaDataSource").option(
        "type", "vertex").option(
        "spaceName", "basketballplayer").option(
        "label", "player").option(
        "returnCols", "name,age").option(
        "metaAddress", "metad0:9559").option(
        "partitionNumber", 1).load()

>>> df.show(n=2)
+---------+--------------+---+
|_vertexId|          name|age|
+---------+--------------+---+
|player105|   Danny Green| 31|
|player109|Tiago Splitter| 34|
+---------+--------------+---+
only showing top 2 rows

[wey-gu]

• Article here: https://www.siwei.io/en/spark-on-nebula-graph/
• Doc/ Examples

[harryprince]

does the schema information could be detected automatically like we use Hive with meta info?
specific schema via option seems not a better way, when the column list is too long.

[wey-gu]

does the schema information could be detected automatically like we use Hive with meta info?
specific schema via option seems not a better way, when the column list is too long.

I think you could use the nebula-python client to fetch meta/schema easier(it should be working to do so via spark-c, too with this py4j under the hood, I didn't try that yet), while, please be noted returnCols isn't mandatory, if it's omitted, all prop will be fetched by default.

[wey-gu]

Now both write and read examples were provided #55