[fix](be) Preserve time field extraction timestamp bounds - #67715
Open
HappenLee wants to merge 1 commit into
Open
[fix](be) Preserve time field extraction timestamp bounds#67715HappenLee wants to merge 1 commit into
HappenLee wants to merge 1 commit into
Conversation
HappenLee
requested review from
924060929,
englefly,
morrySnow and
starocean999
as code owners
September 9, 2026 07:33
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
HappenLee
force-pushed
the
fix-time-field-from-unixtime-boundary
branch
from
September 9, 2026 07:34
b3b8c72 to
d2b7e75
Compare
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: Simplifying HOUR(CAST(FROM_UNIXTIME(ts) AS DATETIMEV2))
into hour_from_unixtime(ts) rejects otherwise valid timestamps near year
9999. The optimized BE functions use a conservative fixed upper bound,
while their FE constant evaluators use an even smaller legacy bound.
For example, timestamp 253402243200 should produce hour 8 in UTC and 16
in Asia/Shanghai but the optimized BE function rejects it.
Compute the last valid timestamp for the session time zone once in the
BE function's fragment-local state and use it in the existing row range
check. Keep date conversion outside the row loop and preserve the field
extraction arithmetic. Align FE constant evaluation with the same local
datetime boundary and update the range error messages.
### Release note
Fix optimized time field extraction rejecting valid FROM_UNIXTIME inputs
near the upper datetime boundary.
### Check List (For Author)
- Test: Unit Test / Manual test
- FE DateTimeExtractAndTransformTest: 14 passed after rebasing onto Apache master.
- FE build.sh --fe and Checkstyle passed on the original patch.
- BE clang-tidy, clang-format 16 and build hygiene passed.
- Added BE boundary/execution unit tests; execution blocked by the
shared AWS SDK missing GeneralHTTPCredentialsProvider.h.
- Added regression rewrite-equivalence and boundary tests; Groovy
syntax compilation passed, cluster execution not performed.
- Isolated normal-input extraction-loop microbenchmark: no material
regression observed; this does not replace query-level benchmarking.
- Behavior changed: Yes; valid boundary timestamps now succeed in the
optimized functions using the session time zone's upper bound.
- Does this need documentation: No
HappenLee
force-pushed
the
fix-time-field-from-unixtime-boundary
branch
from
September 9, 2026 07:36
d2b7e75 to
5a79e34
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Nereids rewrites
HOUR(CAST(FROM_UNIXTIME(ts) AS DATETIMEV2))tohour_from_unixtime(ts), but the optimized function rejects valid timestampsnear the upper datetime boundary. For example,
ts = 253402243200produces9999-12-31 08:00:00in UTC and9999-12-31 16:00:00in Asia/Shanghai;without the rewrite, HOUR returns 8 and 16 respectively, while the rewrite
fails with
INVALID_ARGUMENT.The BE helper uses a fixed conservative upper bound of
253402243199.The FE constant evaluators use an even smaller legacy bound. This change
computes the Unix timestamp corresponding to local
9999-12-31 23:59:59for the session time zone and aligns both implementations with that bound.
BE caches the bound in fragment-local function state during
open()andloads it once per block. The row loop retains its range check and field
extraction arithmetic; there are no additional per-row date conversions
or helper calls. The shared validation covers HOUR, MINUTE, SECOND, and
MICROSECOND. Invalid inputs still fail, with an updated range error.
Validation:
DateTimeExtractAndTransformTest: 14 tests passed after rebasing onto current master, including UTC,Asia/Shanghai, negative/positive offsets, and boundary/invalid inputs.
./build.sh --fe -j 48, including Checkstyle, passed on the original patch.The rebased patch also passed the FE unit test build above.
test build is blocked in unmodified
common/cpp/aws_common.cppbecausethe local shared AWS SDK lacks
GeneralHTTPCredentialsProvider.h;these tests have not run.
checking constant folding and rejecting the first invalid second. Both
changed Groovy suites passed syntax compilation; cluster execution has
not been performed.
material regression (20 measured samples, alternating old/new runs,
fixed CPU affinity, thread CPU time). It excludes function initialization
and the complete execution framework and does not establish end-to-end
query performance.
Release note
Fix optimized time field extraction rejecting valid FROM_UNIXTIME inputs
near the upper datetime boundary.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)