[feature](timestamp_ns) Support casts, functions and external tables - #66761
Open
jacktengg wants to merge 5 commits into
Open
[feature](timestamp_ns) Support casts, functions and external tables#66761jacktengg wants to merge 5 commits into
jacktengg wants to merge 5 commits into
Conversation
jacktengg
requested review from
924060929,
CalvinKirs,
airborne12,
csun5285,
eldenmoon,
englefly,
gavinchou,
liaoxin01,
luwei16,
morrySnow,
mymeiyi,
seawinde,
starocean999 and
yiguolei
as code owners
August 14, 2026 05:44
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
jacktengg
force-pushed
the
timestamp_ns-split-commit1-cast-and-functions
branch
from
August 14, 2026 17:03
a75a76c to
f7b433d
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 17603 ms |
Contributor
TPC-DS: Total hot run time: 84745 ms |
Contributor
ClickBench: Total hot run time: 14.56 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
jacktengg
force-pushed
the
timestamp_ns-split-commit1-cast-and-functions
branch
from
August 15, 2026 16:14
f7b433d to
415fb3c
Compare
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
Contributor
|
Codex automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
TPC-H: Total hot run time: 17816 ms |
Contributor
TPC-DS: Total hot run time: 85488 ms |
Contributor
ClickBench: Total hot run time: 14.63 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
Contributor
|
Codex automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
TPC-H: Total hot run time: 17692 ms |
Contributor
TPC-DS: Total hot run time: 85184 ms |
Contributor
ClickBench: Total hot run time: 16.4 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
jacktengg
changed the title
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve? Issue Number: None Related PR: apache#66333 Problem Summary: Doris DATETIME and DATETIMEV2 support at most six fractional-second digits. Extending their existing packed civil-time representation to nanoseconds would mix incompatible storage representations and semantics. Introduce TIMESTAMP_NS as a separate, fixed-precision timestamp type: - TIMESTAMP_NS always has nine fractional-second digits and does not accept a precision parameter. - Values are stored as signed Int64 nanoseconds relative to the Unix epoch. - The supported range is [1677-09-21 00:12:43.145224192, 2262-04-11 23:47:16.854775807]. - TIMESTAMP_NS has no time-zone attribute. UTC is used as the stable mapping between its civil-time fields and physical epoch-nanosecond representation. - DATETIME(p) and DATETIMEV2(p) retain their existing behavior and continue to support only precision 0 through 6. Add the new primitive type to Thrift, protobuf, FE type metadata, Nereids, BE data types, and cloud schema metadata. Add dedicated TIMESTAMP_NS literal, value, column, and SerDe implementations. Support the basic end-to-end SQL and storage paths, including: - DDL, literals, default values, partitions, buckets, and schema metadata - Insert, stream load, partial update, and decoded timestamp input - Segment storage, key coding, zone map, bloom filter, and inverted index - Comparison, predicates, hashing, grouping, ordering, joins, runtime filters, and basic aggregate functions - FE literal validation, constant folding, type coercion, and predicate simplification - MySQL text and prepared-statement binary results, using string-compatible output to preserve all nine fractional digits Arrow Flight SQL output, ORC OUTFILE, and Java UDF signatures explicitly reject TIMESTAMP_NS because those integrations are not supported yet. The broader explicit cast matrix and scalar date/time function signatures will be implemented in follow-up changes. ### Release note Add the fixed-precision TIMESTAMP_NS SQL type for nanosecond timestamps. TIMESTAMP_NS stores signed Int64 Unix-epoch nanoseconds, supports the range from 1677-09-21 00:12:43.145224192 through 2262-04-11 23:47:16.854775807, and always displays nine fractional digits. Existing DATETIME and DATETIMEV2 types remain limited to precision 0 through 6. ### Check List (For Author) - Test: Unit Test and Regression Test - Added BE unit tests for TIMESTAMP_NS values, SerDe, storage indexes, hashing, predicates, aggregates, and partitioning - Added FE unit tests for types, literals, boundaries, partitions, coercion, folding, MySQL output, and unsupported integrations - Added regression tests for DDL, predicates, storage, stream load, partial/default values, and MySQL prepared-statement output - Behavior changed: Yes; adds the TIMESTAMP_NS SQL and storage type without changing DATETIME or DATETIMEV2 precision semantics - Does this need documentation: Yes; TIMESTAMP_NS syntax, range, storage semantics, and current integration limitations require documentation
Issue Number: None
Related PR: None
Problem Summary: TIMESTAMP_NS basic type support did not include the cast matrix or the date-time function surface available to DATETIMEV2. Queries that converted TIMESTAMP_NS values, combined them with other temporal types, or used date arithmetic, extraction, formatting, aggregate, window, and conditional functions could fail during analysis or backend function resolution. This change adds explicit FE signatures and coercion rules together with BE vectorized implementations. Nanoseconds are preserved when the result remains TIMESTAMP_NS, conversions to lower-precision types use existing rounding semantics, and operations that leave the signed epoch-nanosecond domain report an error instead of wrapping.
Support casts and date-time functions for TIMESTAMP_NS.
- Test: Unit Test and Regression test
- FE unit tests for cast validation, literal conversion, type coercion, and function signatures
- BE unit tests for casts, scalar functions, conditional expressions, and aggregate/window factories
- datatype_p0/timestamp_ns regression suites
- Behavior changed: Yes (TIMESTAMP_NS supports casts and date-time functions)
- Does this need documentation: No
Issue Number: None
Related PR: None
Problem Summary: TIMESTAMP_NS external table integration was mixed into the cast and date-time function commit, which made connector and file-format changes difficult to review and test independently. This commit isolates the Iceberg, Paimon, ADBC, JNI scanner, Parquet, and ORC mappings, readers, writers, predicates, unit tests, and regression expectations without changing the combined source tree.
None
- Test: Manual test
- Verify the two replacement commits reproduce the exact tree of a3b5d8dd6f8dcd43f9ca7960aa1a23c4a04876c8
- Behavior changed: No (commit history was split without changing the combined source tree)
- Does this need documentation: No
Issue Number: None
Related PR: None
Problem Summary: Paimon stores high-precision TIMESTAMP values as unannotated Parquet INT96. FileScannerV2 inferred that encoding as DATETIMEV2(6), so TIMESTAMP(7..9) lost trailing nanoseconds before conversion to TIMESTAMP_NS. Carry the Paimon precision semantic through the schema history and V2 projection metadata, and decode matching Parquet timestamps directly as TIMESTAMP_NS without changing ORC or lower-precision timestamp semantics.
Fix nanosecond precision loss when native scanning Paimon Parquet TIMESTAMP(7..9) columns.
- Test: Regression test and Unit Test
- Regression test: external_table_p0/paimon/paimon_timestamp_types
- Unit Test: PaimonScanPlanProviderTest and targeted BE format_v2 tests
- Behavior changed: Yes. Native Paimon Parquet scans preserve nanoseconds for TIMESTAMP(7..9).
- Does this need documentation: No
jacktengg
force-pushed
the
timestamp_ns-split-commit1-cast-and-functions
branch
from
August 17, 2026 09:50
96e9fb6 to
f96babd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
TIMESTAMP_NS basic type support did not include the cast matrix or the date-time function surface available to DATETIMEV2. Queries that converted TIMESTAMP_NS values, combined them with other temporal types, or used date arithmetic, extraction, formatting, aggregate, window, and conditional functions could fail during analysis or backend function resolution. This change adds explicit FE signatures and coercion rules together with BE vectorized implementations. Nanoseconds are preserved when the result remains TIMESTAMP_NS, conversions to lower-precision types use existing rounding semantics, and operations that leave the signed epoch-nanosecond domain report an error instead of wrapping.
TIMESTAMP_NS external table integration of Iceberg, Paimon, ADBC, JNI scanner, Parquet, and ORC mappings, etc.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)