Skip to content

[feature](timestamp_ns) Add basic TIMESTAMP_NS type support - #66702

Open
jacktengg wants to merge 1 commit into
apache:dev-timestamp-nsfrom
jacktengg:timestamp_ns-split-commit0-basic-support
Open

[feature](timestamp_ns) Add basic TIMESTAMP_NS type support#66702
jacktengg wants to merge 1 commit into
apache:dev-timestamp-nsfrom
jacktengg:timestamp_ns-split-commit0-basic-support

Conversation

@jacktengg

@jacktengg jacktengg commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Problem Summary:

Doris DATETIME and DATETIMEV2 support at most six fractional-second
digits. Extending their existing packed civil-time representation to
nanoseconds would mix incompatible storage representations and semantics.

Introduce TIMESTAMP_NS as a separate, fixed-precision timestamp type:

  • TIMESTAMP_NS always has nine fractional-second digits and does not accept a precision parameter.
  • Values are stored as signed Int64 nanoseconds relative to the Unix epoch.
  • The supported range is [1677-09-21 00:12:43.145224192, 2262-04-11 23:47:16.854775807].
  • TIMESTAMP_NS has no time-zone attribute. UTC is used as the stable mapping between its civil-time fields and physical epoch-nanosecond representation.
  • DATETIME(p) and DATETIMEV2(p) retain their existing behavior and continue to support only precision 0 through 6.

Add the new primitive type to Thrift, protobuf, FE type metadata, Nereids,
BE data types, and cloud schema metadata. Add dedicated TIMESTAMP_NS literal,
value, column, and SerDe implementations.

Support the basic end-to-end SQL and storage paths, including:

  • DDL, literals, default values, partitions, buckets, and schema metadata
  • Insert, stream load, partial update, and decoded timestamp input
  • Segment storage, key coding, zone map, bloom filter, and inverted index
  • Comparison, predicates, hashing, grouping, ordering, joins, runtime filters, and basic aggregate functions
  • FE literal validation, constant folding, type coercion, and predicate simplification
  • MySQL text and prepared-statement binary results, using string-compatible output to preserve all nine fractional digits

Arrow Flight SQL output, ORC OUTFILE, and Java UDF signatures explicitly
reject TIMESTAMP_NS because those integrations are not supported yet.
The broader explicit cast matrix and scalar date/time function signatures
will be implemented in follow-up changes.

Release note

Add the fixed-precision TIMESTAMP_NS SQL type for nanosecond timestamps.
TIMESTAMP_NS stores signed Int64 Unix-epoch nanoseconds, supports the range
from 1677-09-21 00:12:43.145224192 through
2262-04-11 23:47:16.854775807, and always displays nine fractional digits.

Existing DATETIME and DATETIMEV2 types remain limited to precision 0 through 6.

Check List (For Author)

  • Test: Unit Test and Regression Test
    • Added BE unit tests for TIMESTAMP_NS values, SerDe, storage indexes, hashing, predicates, aggregates, and partitioning
    • Added FE unit tests for types, literals, boundaries, partitions, coercion, folding, MySQL output, and unsupported integrations - Added regression tests for DDL, predicates, storage, stream load, partial/default values, and MySQL prepared-statement output
  • Behavior changed: Yes; adds the TIMESTAMP_NS SQL and storage type without changing DATETIME or DATETIMEV2 precision semantics
  • Does this need documentation: Yes; TIMESTAMP_NS syntax, range, storage semantics, and current integration limitations require documentation

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

  ### What problem does this PR solve?

  Issue Number: None

  Related PR: apache#66333

  Problem Summary:

  Doris DATETIME and DATETIMEV2 support at most six fractional-second
  digits. Extending their existing packed civil-time representation to
  nanoseconds would mix incompatible storage representations and semantics.

  Introduce TIMESTAMP_NS as a separate, fixed-precision timestamp type:

  - TIMESTAMP_NS always has nine fractional-second digits and does not accept
    a precision parameter.
  - Values are stored as signed Int64 nanoseconds relative to the Unix epoch.
  - The supported range is
    [1677-09-21 00:12:43.145224192,
     2262-04-11 23:47:16.854775807].
  - TIMESTAMP_NS has no time-zone attribute. UTC is used as the stable mapping
    between its civil-time fields and physical epoch-nanosecond representation.
  - DATETIME(p) and DATETIMEV2(p) retain their existing behavior and continue
    to support only precision 0 through 6.

  Add the new primitive type to Thrift, protobuf, FE type metadata, Nereids,
  BE data types, and cloud schema metadata. Add dedicated TIMESTAMP_NS literal,
  value, column, and SerDe implementations.

  Support the basic end-to-end SQL and storage paths, including:

  - DDL, literals, default values, partitions, buckets, and schema metadata
  - Insert, stream load, partial update, and decoded timestamp input
  - Segment storage, key coding, zone map, bloom filter, and inverted index
  - Comparison, predicates, hashing, grouping, ordering, joins, runtime
    filters, and basic aggregate functions
  - FE literal validation, constant folding, type coercion, and predicate
    simplification
  - MySQL text and prepared-statement binary results, using string-compatible
    output to preserve all nine fractional digits

 Arrow Flight SQL output, ORC OUTFILE, and Java UDF signatures explicitly
  reject TIMESTAMP_NS because those integrations are not supported yet.
  The broader explicit cast matrix and scalar date/time function signatures
  will be implemented in follow-up changes.

  ### Release note

  Add the fixed-precision TIMESTAMP_NS SQL type for nanosecond timestamps.
  TIMESTAMP_NS stores signed Int64 Unix-epoch nanoseconds, supports the range
  from 1677-09-21 00:12:43.145224192 through
  2262-04-11 23:47:16.854775807, and always displays nine fractional digits.

  Existing DATETIME and DATETIMEV2 types remain limited to precision 0 through 6.

  ### Check List (For Author)

  - Test: Unit Test and Regression Test
      - Added BE unit tests for TIMESTAMP_NS values, SerDe, storage indexes,
        hashing, predicates, aggregates, and partitioning
      - Added FE unit tests for types, literals, boundaries, partitions,
        coercion, folding, MySQL output, and unsupported integrations
      - Added regression tests for DDL, predicates, storage, stream load,
        partial/default values, and MySQL prepared-statement output
  - Behavior changed: Yes; adds the TIMESTAMP_NS SQL and storage type without
    changing DATETIME or DATETIMEV2 precision semantics
  - Does this need documentation: Yes; TIMESTAMP_NS syntax, range, storage
    semantics, and current integration limitations require documentation
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@jacktengg

Copy link
Copy Markdown
Contributor Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants