-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feat](metatable) support table$partitions for hive table #40774
Conversation
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
12fa5c5
to
dd98481
Compare
run buildall |
1 similar comment
run buildall |
TPC-H: Total hot run time: 43230 ms
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 195876 ms
|
ClickBench: Total hot run time: 30.8 s
|
run buildall |
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 42306 ms
|
TPC-DS: Total hot run time: 198605 ms
|
ClickBench: Total hot run time: 30.83 s
|
0d8fcd8
to
392d5aa
Compare
mysql> select * from multi_catalog.orc_partitioned_columns$partitions; here should return NULL for t_int |
done |
run buildall |
TPC-H: Total hot run time: 41588 ms
|
TPC-DS: Total hot run time: 199952 ms
|
TeamCity be ut coverage result: |
ClickBench: Total hot run time: 33.42 s
|
run buildall |
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 41731 ms
|
TPC-DS: Total hot run time: 195487 ms
|
ClickBench: Total hot run time: 33.77 s
|
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Support new grammar: `table_name$partitions` User can query partition info by using: ``` select * from table_name$partitions ``` `table_name$partitions` is a special meta table corresponding to the table. Its schema is the partition columns of the table, and column type is always "String". And the value is the partition column's value You can use `DESC table_name$partitions` to get schema: ``` mysql> desc partition_values_all_types$partitions; +-------+----------------+------+------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+----------------+------+------+---------+-------+ | p1 | boolean | Yes | true | NULL | NONE | | p2 | tinyint | Yes | true | NULL | NONE | | p3 | smallint | Yes | true | NULL | NONE | | p4 | int | Yes | true | NULL | NONE | | p5 | bigint | Yes | true | NULL | NONE | | p6 | date | Yes | true | NULL | NONE | | p7 | datetime(6) | Yes | true | NULL | NONE | | p8 | varchar(65533) | Yes | true | NULL | NONE | | p9 | decimal(9,2) | Yes | true | NULL | NONE | | p10 | decimal(18,10) | Yes | true | NULL | NONE | | p11 | decimal(38,9) | Yes | true | NULL | NONE | +-------+----------------+------+------+---------+-------+ ``` Where `px` are partition columns of table `partition_values_all_types`; ``` mysql> select * from partition_values_all_types$partitions order by p1,p2,p3; +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ | p1 | p2 | p3 | p4 | p5 | p6 | p7 | p8 | p9 | p10 | p11 | +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | | 0 | -128 | -32768 | -2147483648 | -9223372036854775808 | 1900-01-01 | 1899-01-01 23:59:59.000000 | NULL | -9999999.99 | -99999999.9999999999 | -99999999999999999999999999999.999999999 | | 0 | 127 | 32767 | 2147483647 | 9223372036854775807 | 9999-12-31 | 0001-01-01 00:00:01.321000 | boston | 9999999.99 | 99999999.9999999999 | 99999999999999999999999999999.999999999 | +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ ``` Currently, this grammar can only be used for partition table's in Hive Catalog. Table in other catalogs or non partition table can not use this grammar. Internally, it is implemented as a table valued function: `partition_values` ``` mysql> select * from partition_values("catalog" = "hive", "database" = "multi_catalog", "table" = "partition_values_all_types"); +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ | p1 | p2 | p3 | p4 | p5 | p6 | p7 | p8 | p9 | p10 | p11 | +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ | 0 | 127 | 32767 | 2147483647 | 9223372036854775807 | 9999-12-31 | 0001-01-01 00:00:01.321000 | boston | 9999999.99 | 99999999.9999999999 | 99999999999999999999999999999.999999999 | | 0 | -128 | -32768 | -2147483648 | -9223372036854775808 | 1900-01-01 | 1899-01-01 23:59:59.000000 | NULL | -9999999.99 | -99999999.9999999999 | -99999999999999999999999999999.999999999 | | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ ```
Support new grammar: `table_name$partitions` User can query partition info by using: ``` select * from table_name$partitions ``` `table_name$partitions` is a special meta table corresponding to the table. Its schema is the partition columns of the table, and column type is always "String". And the value is the partition column's value You can use `DESC table_name$partitions` to get schema: ``` mysql> desc partition_values_all_types$partitions; +-------+----------------+------+------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+----------------+------+------+---------+-------+ | p1 | boolean | Yes | true | NULL | NONE | | p2 | tinyint | Yes | true | NULL | NONE | | p3 | smallint | Yes | true | NULL | NONE | | p4 | int | Yes | true | NULL | NONE | | p5 | bigint | Yes | true | NULL | NONE | | p6 | date | Yes | true | NULL | NONE | | p7 | datetime(6) | Yes | true | NULL | NONE | | p8 | varchar(65533) | Yes | true | NULL | NONE | | p9 | decimal(9,2) | Yes | true | NULL | NONE | | p10 | decimal(18,10) | Yes | true | NULL | NONE | | p11 | decimal(38,9) | Yes | true | NULL | NONE | +-------+----------------+------+------+---------+-------+ ``` Where `px` are partition columns of table `partition_values_all_types`; ``` mysql> select * from partition_values_all_types$partitions order by p1,p2,p3; +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ | p1 | p2 | p3 | p4 | p5 | p6 | p7 | p8 | p9 | p10 | p11 | +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | | 0 | -128 | -32768 | -2147483648 | -9223372036854775808 | 1900-01-01 | 1899-01-01 23:59:59.000000 | NULL | -9999999.99 | -99999999.9999999999 | -99999999999999999999999999999.999999999 | | 0 | 127 | 32767 | 2147483647 | 9223372036854775807 | 9999-12-31 | 0001-01-01 00:00:01.321000 | boston | 9999999.99 | 99999999.9999999999 | 99999999999999999999999999999.999999999 | +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ ``` Currently, this grammar can only be used for partition table's in Hive Catalog. Table in other catalogs or non partition table can not use this grammar. Internally, it is implemented as a table valued function: `partition_values` ``` mysql> select * from partition_values("catalog" = "hive", "database" = "multi_catalog", "table" = "partition_values_all_types"); +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ | p1 | p2 | p3 | p4 | p5 | p6 | p7 | p8 | p9 | p10 | p11 | +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ | 0 | 127 | 32767 | 2147483647 | 9223372036854775807 | 9999-12-31 | 0001-01-01 00:00:01.321000 | boston | 9999999.99 | 99999999.9999999999 | 99999999999999999999999999999.999999999 | | 0 | -128 | -32768 | -2147483648 | -9223372036854775808 | 1900-01-01 | 1899-01-01 23:59:59.000000 | NULL | -9999999.99 | -99999999.9999999999 | -99999999999999999999999999999.999999999 | | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | +------+------+--------+-------------+----------------------+------------+----------------------------+--------+-------------+----------------------+------------------------------------------+ ```
Support new grammar:
table_name$partitions
User can query partition info by using:
table_name$partitions
is a special meta table corresponding to the table.Its schema is the partition columns of the table, and column type is always "String".
And the value is the partition column's value
You can use
DESC table_name$partitions
to get schema:Where
px
are partition columns of tablepartition_values_all_types
;Currently, this grammar can only be used for partition table's in Hive Catalog.
Table in other catalogs or non partition table can not use this grammar.
Internally, it is implemented as a table valued function:
partition_values