Skip to content

[fix](be) Preserve time field extraction timestamp bounds - #67715

Open
HappenLee wants to merge 1 commit into
apache:masterfrom
HappenLee:fix-time-field-from-unixtime-boundary
Open

[fix](be) Preserve time field extraction timestamp bounds#67715
HappenLee wants to merge 1 commit into
apache:masterfrom
HappenLee:fix-time-field-from-unixtime-boundary

Conversation

@HappenLee

@HappenLee HappenLee commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

Nereids rewrites HOUR(CAST(FROM_UNIXTIME(ts) AS DATETIMEV2)) to
hour_from_unixtime(ts), but the optimized function rejects valid timestamps
near the upper datetime boundary. For example, ts = 253402243200 produces
9999-12-31 08:00:00 in UTC and 9999-12-31 16:00:00 in Asia/Shanghai;
without the rewrite, HOUR returns 8 and 16 respectively, while the rewrite
fails with INVALID_ARGUMENT.

The BE helper uses a fixed conservative upper bound of 253402243199.
The FE constant evaluators use an even smaller legacy bound. This change
computes the Unix timestamp corresponding to local 9999-12-31 23:59:59
for the session time zone and aligns both implementations with that bound.

BE caches the bound in fragment-local function state during open() and
loads it once per block. The row loop retains its range check and field
extraction arithmetic; there are no additional per-row date conversions
or helper calls. The shared validation covers HOUR, MINUTE, SECOND, and
MICROSECOND. Invalid inputs still fail, with an updated range error.

Validation:

  • FE DateTimeExtractAndTransformTest: 14 tests passed after rebasing onto current master, including UTC,
    Asia/Shanghai, negative/positive offsets, and boundary/invalid inputs.
  • ./build.sh --fe -j 48, including Checkstyle, passed on the original patch.
    The rebased patch also passed the FE unit test build above.
  • Modified C++ files passed clang-format 16, clang-tidy, and build hygiene.
  • Added BE unit tests for timezone bounds and function execution. The BE
    test build is blocked in unmodified common/cpp/aws_common.cpp because
    the local shared AWS SDK lacks GeneralHTTPCredentialsProvider.h;
    these tests have not run.
  • Added regression tests comparing the rewrite with the original expression,
    checking constant folding and rejecting the first invalid second. Both
    changed Groovy suites passed syntax compilation; cluster execution has
    not been performed.
  • An isolated ordinary-input extraction-loop microbenchmark showed no
    material regression (20 measured samples, alternating old/new runs,
    fixed CPU affinity, thread CPU time). It excludes function initialization
    and the complete execution framework and does not establish end-to-end
    query performance.

Release note

Fix optimized time field extraction rejecting valid FROM_UNIXTIME inputs
near the upper datetime boundary.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (format/static checks and isolated microbenchmark described above)
    • No need to test or manual test.
  • Behavior changed:

    • No.
    • Yes. Valid boundary timestamps now succeed using the session time zone's upper bound.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee
HappenLee force-pushed the fix-time-field-from-unixtime-boundary branch from b3b8c72 to d2b7e75 Compare September 9, 2026 07:34
### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary: Simplifying HOUR(CAST(FROM_UNIXTIME(ts) AS DATETIMEV2))
into hour_from_unixtime(ts) rejects otherwise valid timestamps near year
9999. The optimized BE functions use a conservative fixed upper bound,
while their FE constant evaluators use an even smaller legacy bound.
For example, timestamp 253402243200 should produce hour 8 in UTC and 16
in Asia/Shanghai but the optimized BE function rejects it.

Compute the last valid timestamp for the session time zone once in the
BE function's fragment-local state and use it in the existing row range
check. Keep date conversion outside the row loop and preserve the field
extraction arithmetic. Align FE constant evaluation with the same local
datetime boundary and update the range error messages.

### Release note

Fix optimized time field extraction rejecting valid FROM_UNIXTIME inputs
near the upper datetime boundary.

### Check List (For Author)

- Test: Unit Test / Manual test
    - FE DateTimeExtractAndTransformTest: 14 passed after rebasing onto Apache master.
    - FE build.sh --fe and Checkstyle passed on the original patch.
    - BE clang-tidy, clang-format 16 and build hygiene passed.
    - Added BE boundary/execution unit tests; execution blocked by the
      shared AWS SDK missing GeneralHTTPCredentialsProvider.h.
    - Added regression rewrite-equivalence and boundary tests; Groovy
      syntax compilation passed, cluster execution not performed.
    - Isolated normal-input extraction-loop microbenchmark: no material
      regression observed; this does not replace query-level benchmarking.
- Behavior changed: Yes; valid boundary timestamps now succeed in the
  optimized functions using the session time zone's upper bound.
- Does this need documentation: No
@HappenLee
HappenLee force-pushed the fix-time-field-from-unixtime-boundary branch from d2b7e75 to 5a79e34 Compare September 9, 2026 07:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants