Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] support iceberg query with branch/tag/version/timestamp #47997

Merged
merged 4 commits into from
Jul 15, 2024

Conversation

stephen-shelby
Copy link
Contributor

@stephen-shelby stephen-shelby commented Jul 8, 2024

Why I'm doing:

What I'm doing:

syntax:

queryPeriod
    : AS OF expression
    | FOR? periodType BETWEEN expression AND expression
    | FOR? periodType FROM expression TO expression
    | FOR? periodType ALL
    | FOR? periodType AS OF end=expression
    ;

periodType
    : SYSTEM_TIME
    | TIMESTAMP
    | VERSION
    ;
select * from table version as of 123456678    -- with snapshotId
select * from table version as of "branch"        -- with branch
select * from table version as of "tag"              -- with tag
select * from table timestamp as of "2023-01-01 12:34:45"   -- The latest snapshot before that time
select * from table timestamp as of "2023-01-01"                   -- The latest snapshot before that time
select * from table timestamp as of now()   -- with date function
select * from table for system_time as of current_date(); -- with date function

In common, any version information corresponds to a snapshot id, so for iceberg, we parse all version information like branch/tag/timestamp as snasphotId then use it.

Fixes #issue
#47846

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

@stephen-shelby stephen-shelby force-pushed the timetravel branch 2 times, most recently from cead2b9 to fe4ef57 Compare July 9, 2024 03:45
@stephen-shelby stephen-shelby changed the title [Feature] support iceberg query with branch/tag/version/timestamp [UT] support iceberg query with branch/tag/version/timestamp Jul 9, 2024
@stephen-shelby stephen-shelby force-pushed the timetravel branch 2 times, most recently from 54ac201 to 690a934 Compare July 9, 2024 09:15
Youngwb
Youngwb previously approved these changes Jul 9, 2024
@stephen-shelby stephen-shelby changed the title [UT] support iceberg query with branch/tag/version/timestamp [Feature] support iceberg query with branch/tag/version/timestamp Jul 10, 2024
@stephen-shelby stephen-shelby force-pushed the timetravel branch 3 times, most recently from 9cff642 to 48b7f00 Compare July 10, 2024 10:32
Youngwb
Youngwb previously approved these changes Jul 10, 2024
dirtysalt
dirtysalt previously approved these changes Jul 10, 2024
HangyuanLiu
HangyuanLiu previously approved these changes Jul 11, 2024
Signed-off-by: stephen <stephen5217@163.com>
Signed-off-by: stephen <stephen5217@163.com>
Signed-off-by: stephen <stephen5217@163.com>
@stephen-shelby stephen-shelby force-pushed the timetravel branch 2 times, most recently from f0f3c06 to 588604b Compare July 15, 2024 06:25
Signed-off-by: stephen <stephen5217@163.com>
Copy link

sonarcloud bot commented Jul 15, 2024

Copy link

[FE Incremental Coverage Report]

pass : 94 / 115 (81.74%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/catalog/Table.java 0 1 00.00% [744]
🔵 com/starrocks/catalog/MysqlTable.java 0 1 00.00% [241]
🔵 com/starrocks/connector/unified/UnifiedMetadata.java 0 1 00.00% [129]
🔵 com/starrocks/sql/analyzer/QueryAnalyzer.java 1 3 33.33% [382, 383]
🔵 com/starrocks/sql/optimizer/transformer/RelationTransformer.java 21 30 70.00% [636, 685, 686, 687, 691, 692, 693, 695, 696]
🔵 com/starrocks/connector/iceberg/IcebergMetadata.java 39 45 86.67% [377, 384, 397, 407, 412, 448]
🔵 com/starrocks/sql/ast/QueryPeriod.java 13 14 92.86% [57]
🔵 com/starrocks/catalog/IcebergTable.java 1 1 100.00% []
🔵 com/starrocks/server/MetadataMgr.java 1 1 100.00% []
🔵 com/starrocks/sql/ast/TableRelation.java 5 5 100.00% []
🔵 com/starrocks/connector/ConnectorTableVersion.java 6 6 100.00% []
🔵 com/starrocks/connector/CatalogConnectorMetadata.java 1 1 100.00% []
🔵 com/starrocks/connector/metadata/iceberg/LogicalIcebergMetadataTable.java 1 1 100.00% []
🔵 com/starrocks/sql/optimizer/operator/logical/LogicalIcebergScanOperator.java 1 1 100.00% []
🔵 com/starrocks/connector/PointerType.java 4 4 100.00% []

Copy link

[BE Incremental Coverage Report]

pass : 0 / 0 (0%)

@stephen-shelby stephen-shelby merged commit 90f779c into StarRocks:main Jul 15, 2024
65 checks passed
zhangheihei pushed a commit to zhangheihei/starrocks that referenced this pull request Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support iceberg timetravel/branch/tag
8 participants