Feature Support Detail
Main Features
| Feature Support | TiSpark 2.4.x | TiSpark 2.5.x | TiSpark 3.0.x | TiSpark master |
|---|---|---|---|---|
| SQL select without tidb_catalog | ✔ | ✔ | ||
| SQL select with tidb_catalog | ✔ | ✔ | ✔ | |
| DataFrame append | ✔ | ✔ | ✔ | ✔ |
| DataFrame reads | ✔ | ✔ | ✔ | ✔ |
| SQL show databases | ✔ | ✔ | ✔ | ✔ |
| SQL show tables | ✔ | ✔ | ✔ | ✔ |
| SQL auth | ✔ | ✔ | ✔ | |
| SQL delete from with tidb_catalog | ✔ | ✔ | ||
| TLS | ✔ | ✔ | ||
| DataFrame auth | ✔ |
you can read with both SQL select and DataFrame reads
TiSpark can push down some operators to TiKV
- predicate
- aggregate
The following aggregate can be pushed down
- Min
- Max
- Count
- Sum
- Avg
| push down types | comment | |
|---|---|---|
| sum | smallint,bigint,decimal,mediumint,real,tinyint,int,double,year | Spark JDBC does not support sum(year) |
| count | only set can not be pushed down | Count(col1,col2,...) will not be pushed down |
| avg | smallint,bigint,decimal,mediumint,real,tinyint,int,double,year | Spark JDBC does not support avg(year) |
- Sum(float) can't be pushed down because Spark will cast float to double, but
castcan't be pushed down in TiSpark. - Count(set) can't be pushed down because TiKV does not support set type.
- Count(col1,col2,...) can't be pushed down because TiDB/TiKV does not support Count(col1,col2,...).
- Avg can be pushed down when both the sum and count can be pushed down.
TiSpark currently supports Range Partition and Hash Partition. Users can select data from the Range Partition table and the Hash Partition table through TiSpark.
In most cases, TiSpark use a full table scan on partition tables. Only in certain cases, TiSpark applies partition pruning. For more details, see here.
tidb-5.0 supports Expression Index.
TiSpark currently supports retrieving data from table with Expression Index, but the Expression Index will not be used by the planner of TiSpark.
TiSpark can read from TiFlash with the configuration spark.tispark.isolation_read_engines
You can only write to TiDB with DataFrame append now
- write to the partition table