Skip to content

Getting TiSpark

shiyuhang0 edited this page Jun 23, 2022 · 5 revisions

What is TiSpark

TiSpark is a third-party jar package for Spark that provides the ability to read/write TiKV

Getting TiSpark Jar

Choose the version of TiSpark

TiSpark version TiDB、TiKV、PD version Spark version Scala version
2.4.x-scala_2.11 5.x, 4.x 2.3.x, 2.4.x 2.11
2.4.x-scala_2.12 5.x, 4.x 2.4.x 2.12
2.5.x 5.x, 4.x 3.0.x, 3.1.x 2.12
3.0.x 5.x, 4.x 3.0.x, 3.1.x, 3.2.x 2.12
  • TiSpark 2.4.3, 2.5.1, 3.0.1 is the latest stable version, which is highly recommended.
  • Spark < 2.3 is not supported anymore. if your spark version is below 2.3, follow this document to use another version of TiSpark

Get TiSpark jar

Currently, java8 is the only choice to build TiSpark, run mvn -version to check.

git clone https://github.com/pingcap/tispark.git

Run the following command under the TiSpark root directory:

// add -Dmaven.test.skip=true to skip the tests
mvn clean install -Dmaven.test.skip=true
// or you can add properties to specify spark version
mvn clean install -Dmaven.test.skip=true -Pspark3.2.1

TiSpark jar's Artifact ID

The Artifact ID of TiSpark is a bit different in different TiSpark version

TiSpark version Artifact ID
2.4.x-${scala_version}, 2.5.0 tispark-assembly
2.5.1 tispark-assembly-${spark_version}
3.0.x tispark-assembly-${spark_version}_${scala_version}