Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix version validation #88

Merged
merged 4 commits into from
Mar 7, 2023
Merged

fix version validation #88

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
297e18e
fix the version validation info
Nicole00 Feb 28, 2023
4b1689f
update version match
Nicole00 Feb 28, 2023
eaed6b0
fix assert info
Nicole00 Mar 7, 2023
dcafac4
update Nebula to NebulaGraph
Nicole00 Mar 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 36 additions & 14 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,29 @@ Nebula Spark Connector support spark 2.2 and 2.4.

## How to Use

If you use Maven to manage your project, add the following dependency to your pom.xml:
If you use Maven to manage your project, add one of the following dependency to your pom.xml:

```
<!-- connector for spark 2.4 -->
<dependency>
<groupId>com.vesoft</groupId>
<artifactId>nebula-spark-connector</artifactId>
<version>3.0-SNAPSHOT</version>
</dependency>

<!-- connector for spark 2.2 -->
<dependency>
<groupId>com.vesoft</groupId>
<artifactId>nebula-spark-connector_2.2</artifactId>
<version>3.0-SNAPSHOT</version>
</dependency>

<!-- connector for spark 3.0 -->
<dependency>
<groupId>com.vesoft</groupId>
<artifactId>nebula-spark-connector_3.0</artifactId>
<version>3.0-SNAPSHOT</version>
</dependency>
```

Write DataFrame `INSERT` into Nebula Graph as Vertices:
Expand Down Expand Up @@ -309,19 +325,25 @@ df = spark.read.format(

## Version match
Nicole00 marked this conversation as resolved.
Show resolved Hide resolved

There are the version correspondence between Nebula Spark Connector and Nebula:

| Nebula Spark Connector Version | Nebula Version |
|:------------------------------:|:--------------:|
| 2.0.0 | 2.0.0, 2.0.1 |
| 2.0.1 | 2.0.0, 2.0.1 |
| 2.1.0 | 2.0.0, 2.0.1 |
| 2.5.0 | 2.5.0, 2.5.1 |
| 2.5.1 | 2.5.0, 2.5.1 |
| 2.6.0 | 2.6.0, 2.6.1 |
| 2.6.1 | 2.6.0, 2.6.1 |
| 3.0.0 | 3.0.x, 3.1.x |
| 3.0-SNAPSHOT | nightly |
There are the version correspondence between Nebula Spark Connector and Nebula、Spark:

| Nebula Spark Connector Version | Nebula Version | Spark Version |
|:-----------------------------------------:|:--------------:|:-------------:|
|nebula-spark-connector-2.0.0.jar | 2.0.0, 2.0.1 | 2.4.* |
|nebula-spark-connector-2.0.1.jar | 2.0.0, 2.0.1 | 2.4.* |
|nebula-spark-connector-2.1.0.jar | 2.0.0, 2.0.1 | 2.4.* |
|nebula-spark-connector-2.5.0.jar | 2.5.0, 2.5.1 | 2.4.* |
|nebula-spark-connector-2.5.1.jar | 2.5.0, 2.5.1 | 2.4.* |
|nebula-spark-connector-2.6.0.jar | 2.6.0, 2.6.1 | 2.4.* |
|nebula-spark-connector-2.6.1.jar | 2.6.0, 2.6.1 | 2.4.* |
|nebula-spark-connector-3.0.0.jar | 3.x | 2.4.* |
|nebula-spark-connector-3.3.0.jar | 3.x | 2.4.* |
|nebula-spark-connector_2.2-3.3.0.jar | 3.x | 2.2.* |
|nebula-spark-connector-3.4.0.jar | 3.x | 2.4.* |
|nebula-spark-connector_2.2-3.4.0.jar | 3.x | 2.2.* |
|nebula-spark-connector-3.0-SNAPSHOT.jar | nightly | 2.4.* |
|nebula-spark-connector_2.2-3.0-SNAPSHOT.jar| nightly | 2.2.* |
|nebula-spark-connector_3.0-3.0-SNAPSHOT.jar| nightly | 3.* |

## Performance
We use LDBC dataset to test nebula-spark-connector's performance, here's the result.
Expand Down
50 changes: 36 additions & 14 deletions README_CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,28 @@ Nebula Spark Connector 支持 Spark 2.2 和 2.4.
* Nebula Spark Connector 2.5.0 增加了 DELETE 写入模式，相关说明参考[Delete Vertex](https://docs.nebula-graph.com.cn/2.5.1/3.ngql-guide/12.vertex-statements/4.delete-vertex/)

## 使用说明
如果你使用Maven管理项目，请在pom.xml文件中增加依赖:
如果你使用Maven管理项目，请在pom.xml文件中增加下列某一项依赖:
```
<!-- connector for spark 2.4 -->
<dependency>
<groupId>com.vesoft</groupId>
<artifactId>nebula-spark-connector</artifactId>
<version>3.0-SNAPSHOT</version>
</dependency>

<!-- connector for spark 2.2 -->
<dependency>
<groupId>com.vesoft</groupId>
<artifactId>nebula-spark-connector_2.2</artifactId>
<version>3.0-SNAPSHOT</version>
</dependency>

<!-- connector for spark 3.0 -->
<dependency>
<groupId>com.vesoft</groupId>
<artifactId>nebula-spark-connector_3.0</artifactId>
<version>3.0-SNAPSHOT</version>
</dependency>
```

将 DataFrame 作为点 `INSERT` 写入 Nebula Graph :
Expand Down Expand Up @@ -305,19 +320,26 @@ df = spark.read.format(
```

## 版本匹配
Nebula Spark Connector 和 Nebula 的版本对应关系如下:

| Nebula Spark Connector Version | Nebula Version |
|:------------------------------:|:--------------:|
| 2.0.0 | 2.0.0, 2.0.1 |
| 2.0.1 | 2.0.0, 2.0.1 |
| 2.1.0 | 2.0.0, 2.0.1 |
| 2.5.0 | 2.5.0, 2.5.1 |
| 2.5.1 | 2.5.0, 2.5.1 |
| 2.6.0 | 2.6.0, 2.6.1 |
| 2.6.1 | 2.6.0, 2.6.1 |
| 3.0.0 | 3.0.0 |
| 3.0-SNAPSHOT | nightly |
Nebula Spark Connector 和 Nebula 、Spark 的版本对应关系如下:
Nicole00 marked this conversation as resolved.
Show resolved Hide resolved

| Nebula Spark Connector Version | Nebula Version | Spark Version |
|:-----------------------------------------:|:--------------:|:-------------:|
|nebula-spark-connector-2.0.0.jar | 2.0.0, 2.0.1 | 2.4.* |
|nebula-spark-connector-2.0.1.jar | 2.0.0, 2.0.1 | 2.4.* |
|nebula-spark-connector-2.1.0.jar | 2.0.0, 2.0.1 | 2.4.* |
|nebula-spark-connector-2.5.0.jar | 2.5.0, 2.5.1 | 2.4.* |
|nebula-spark-connector-2.5.1.jar | 2.5.0, 2.5.1 | 2.4.* |
|nebula-spark-connector-2.6.0.jar | 2.6.0, 2.6.1 | 2.4.* |
|nebula-spark-connector-2.6.1.jar | 2.6.0, 2.6.1 | 2.4.* |
|nebula-spark-connector-3.0.0.jar | 3.x | 2.4.* |
|nebula-spark-connector-3.3.0.jar | 3.x | 2.4.* |
|nebula-spark-connector_2.2-3.3.0.jar | 3.x | 2.2.* |
|nebula-spark-connector-3.4.0.jar | 3.x | 2.4.* |
|nebula-spark-connector_2.2-3.4.0.jar | 3.x | 2.2.* |
|nebula-spark-connector-3.0-SNAPSHOT.jar | nightly | 2.4.* |
|nebula-spark-connector_2.2-3.0-SNAPSHOT.jar| nightly | 2.2.* |
|nebula-spark-connector_3.0-3.0-SNAPSHOT.jar| nightly | 3.* |


## 性能
我们使用LDBC数据集进行Nebula-Spark-Connector的性能测试，测试结果如下：
Expand Down
3 changes: 2 additions & 1 deletion nebula-spark-common/src/main/scala/com/vesoft/nebula/connector/NebulaConfig.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -784,7 +784,8 @@ object ReadNebulaConfig {
assert(space != null && !space.isEmpty, s"config space is empty.")
assert(label != null && !label.isEmpty, s"config label is empty.")
assert(limit > 0, s"config limit must be positive, your limit is $limit")
assert(partitionNum > 0, s"config partitionNum must be positive, your partitionNum is $limit")
assert(partitionNum > 0,
s"config partitionNum must be positive, your partitionNum is $partitionNum")
if (noColumn && returnCols.nonEmpty) {
LOG.warn(
s"noColumn is true, returnCols will be invalidate "
Expand Down
4 changes: 2 additions & 2 deletions nebula-spark-common/src/main/scala/com/vesoft/nebula/connector/utils/SparkValidate.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ object SparkValidate {
val sparkVersion = SparkSession.getActiveSession.map(_.version).getOrElse("UNKNOWN")
if (sparkVersion != "UNKNOWN" && !supportedVersions.exists(sparkVersion.matches)) {
throw new RuntimeException(
s"""Your current spark version ${sparkVersion} is not supported by the current NebulaGraph Exchange.
| please visit https://github.com/vesoft-inc/nebula-exchange#version-match to know which Exchange you need.
s"""Your current spark version ${sparkVersion} is not supported by the current NebulaGraph Spark Connector.
| please visit https://github.com/vesoft-inc/nebula-spark-connector#version-match to know which Connector you need.
| """.stripMargin)
}
}
Expand Down