fix: INTERVAL n WEEK resolves to seven days instead of one hour#19298
fix: INTERVAL n WEEK resolves to seven days instead of one hour#19298barry3406 wants to merge 2 commits intoapache:masterfrom
Conversation
Calcite 1.37.0's SqlIntervalQualifier.evaluateIntervalLiteralAsWeek packages a WEEK literal as a year-month shaped int array ([sign, 0, week]) but reports the type as INTERVAL_DAY, so the downstream SqlParserUtil.intervalToMillis loop interprets the week value as the hour component. The result is that `MILLIS_TO_TIMESTAMP(0) + INTERVAL '1' WEEK` resolves to 1970-01-01T01:00:00 instead of 1970-01-08, as reported in apache#18665. The bug was fixed upstream in CALCITE-6391 / Calcite 1.38, but Druid is still on 1.37, so this commit works around it inside Druid's planner. A new SqlIntervalWeekRewriteShuttle walks the parsed SqlNode tree in DruidPlanner.validate and rewrites both forms of week intervals before they reach the buggy code path: - `INTERVAL '1' WEEK` (a SqlIntervalLiteral) is replaced with an equivalent SqlIntervalLiteral whose numeric value has been multiplied by seven and whose qualifier is now DAY. - `INTERVAL 1 WEEK` (a SqlCall using SqlStdOperatorTable.INTERVAL) is rewritten so the numeric operand becomes `MULTIPLY(n, 7)` and the qualifier becomes DAY. WEEK(SUNDAY)..WEEK(SATURDAY) and ISOWEEK qualifiers carry a non-null timeFrameName and are left untouched, so this only affects the bare WEEK form that is broken in Calcite 1.37. Tests: - SqlIntervalWeekRewriteShuttleTest exercises the shuttle directly against constructed SqlIntervalLiteral and SqlCall nodes for WEEK, DAY and MONTH, and against parsed queries that use both quoted and unquoted week intervals. - CalciteQueryTest#testIntervalWeekResolution is the regression test from apache#18665. Without the fix it produces interval narrowing on 2000-01-01T01:00:00 / 2000-01-01T02:00:00; with the fix it correctly narrows to 2000-01-08 and 2000-01-15.
There was a problem hiding this comment.
Pull request overview
Fixes a Calcite 1.37.0 interval bug where INTERVAL n WEEK is interpreted as n hours by rewriting plain WEEK intervals to equivalent DAY intervals (multiplying the value by 7) before Calcite validation in Druid’s planner.
Changes:
- Add
SqlIntervalWeekRewriteShuttleto rewrite bareWEEKinterval literals/calls intoDAYintervals. - Invoke the shuttle during
DruidPlanner.validate()prior to parameter rewriting/validation. - Add unit tests for the shuttle and an end-to-end regression test for issue #18665.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| sql/src/main/java/org/apache/druid/sql/calcite/planner/SqlIntervalWeekRewriteShuttle.java | Introduces the AST rewrite workaround for Calcite’s broken plain WEEK interval handling. |
| sql/src/main/java/org/apache/druid/sql/calcite/planner/DruidPlanner.java | Runs the new shuttle during validation so Calcite never sees the buggy WEEK interval representation. |
| sql/src/test/java/org/apache/druid/sql/calcite/planner/SqlIntervalWeekRewriteShuttleTest.java | Adds direct unit coverage for quoted/unquoted week interval rewrites and parser-level checks. |
| sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java | Adds an end-to-end regression test demonstrating correct interval narrowing for WEEK. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| final SqlBasicCall numericCall = (SqlBasicCall) numeric; | ||
| assertSame(SqlStdOperatorTable.MULTIPLY, numericCall.getOperator()); | ||
| assertEquals(2, numericCall.operandCount()); | ||
| assertEquals("7", ((SqlLiteral) numericCall.operand(1)).toValue()); | ||
|
|
There was a problem hiding this comment.
SqlLiteral.toValue() for createExactNumeric returns a numeric type (typically BigDecimal), not a String. This assertion will fail at runtime; compare against BigDecimal.valueOf(7) (or use getValueAs(BigDecimal.class) / toValue().toString()).
|
Did you run into some problem while upgrading Calcite? It would be better to fix the bug that way, rather than through a workaround. Also, the bug was fixed in https://issues.apache.org/jira/browse/CALCITE-6581. I know because I wrote the patch 🙂. At the time I figured we would get the fix the next time we upgraded Calcite. |
|
Good point — I didn't attempt the Calcite upgrade because it felt too big for a bug-fix PR (1.37 → 1.38 tends to pull in a lot of changes). This shuttle is removable once the upgrade lands. Also thanks for the correction, the actual fix is CALCITE-6581 not CALCITE-6391 — I'll update the references. If you'd rather just bump Calcite instead I'm fine closing this. |
|
Certainly I'd prefer upgrading Calcite if it's straightforward. That's why I was wondering if you tried and ran into any issues. It can a be a lot of work (see #14510, #16504) but I was hoping it wouldn't be too bad this time. If upgrading Calcite isn't straightforward, we could merge your fix here temporarily, then remove it once we do upgrade Calcite. Btw, if we are going to do the temporary fix, please include |
INTERVAL n QUARTER is broken in Calcite 1.37 in the same way as WEEK: SqlIntervalQualifier.evaluateIntervalLiteralAsQuarter packages the value into a [sign, 0, quarter] year-month array without multiplying by three, so SqlParserUtil.intervalToMonths reads `quarter` months instead of `3 * quarter` months. Extend SqlIntervalWeekRewriteShuttle to rewrite QUARTER -> MONTH * 3 alongside the existing WEEK -> DAY * 7 rewrite, covering both the quoted (SqlIntervalLiteral) and unquoted (SqlCall) parsed forms. Also update the CALCITE ticket reference from CALCITE-6391 to the correct CALCITE-6581, which is the upstream fix gianm wrote for both units.
|
Added QUARTER support in 09b6250 — mirrors the WEEK rewrite ( |
|
I raised a PR for the Calcite update here: #19370. That should also fix this issue. The PR adds a test |
|
The changes LGTM, no correctness issues found. |
Fixes #18665.
Description
INTERVAL '1' WEEK(and the unquotedINTERVAL 1 WEEK) was resolving to one hour instead of seven days. The repro from #18665 makes the symptom obvious:was returning
1970-01-01T01:00:00,1970-01-01T02:00:00(one and two hours) instead of the expected1970-01-08,1970-01-15.Per @gianm's review,
INTERVAL n QUARTERis broken in Calcite 1.37.0 in the same way, so this PR also covers QUARTER.Root cause
The bugs live in Calcite 1.37.0:
SqlIntervalQualifier.evaluateIntervalLiteralAsWeekpackages a WEEK literal into a year-month shaped int array ([sign, 0, week]), even thoughSqlIntervalQualifier.typeName()reports a WEEK qualifier asINTERVAL_DAY. The downstreamSqlParserUtil.intervalToMillisloop walksret[1..]as[day, hour, minute, second, ms], so the week count lands in the hour slot.INTERVAL '1' WEEKtherefore becomes1 * 3_600_000 ms = 1 hour.SqlIntervalQualifier.evaluateIntervalLiteralAsQuarterpackages a QUARTER literal into the same year-month shaped array ([sign, 0, quarter]) without multiplying by three.SqlParserUtil.intervalToMonthsreads12 * 0 + 1 * quarter = quartermonths instead of3 * quartermonths.INTERVAL '1' QUARTERtherefore becomes one month.Both bugs were fixed upstream as CALCITE-6581 in Calcite 1.38. Druid is still on Calcite 1.37, so we work around them inside the planner instead of upgrading Calcite in this PR.
Fixed the bug ...
Added
SqlIntervalWeekRewriteShuttle, anSqlShuttlethat walks the parsedSqlNodetree before validation and rewrites the affected parsed forms into equivalent intervals in a unit Calcite 1.37 handles correctly:INTERVAL '1' WEEKparses to aSqlIntervalLiteralwith a WEEK qualifier. The shuttle replaces it with aSqlIntervalLiteralwhose value has been multiplied by seven and whose qualifier is now DAY.INTERVAL 1 WEEKparses toSqlStdOperatorTable.INTERVAL.createCall(pos, n, qualifier). The shuttle rewrites the call so the numeric operand becomesMULTIPLY(n, 7)and the qualifier becomes DAY.INTERVAL '1' QUARTERandINTERVAL 1 QUARTERare rewritten the same way, using MONTH and a factor of three.WEEK(SUNDAY)..WEEK(SATURDAY)andISOWEEKqualifiers carry a non-nulltimeFrameNameand are handled differently in Calcite, so the shuttle leaves them alone and only touches the bareWEEK/QUARTERforms that are broken in 1.37.The shuttle is invoked in
DruidPlanner.validateright beforerewriteParameters, alongside the existingSqlParameterizerShuttlepass. It runs once per validate call and is a no-op for any tree that does not contain a WEEK or QUARTER interval.This shuttle is a temporary workaround and should be removed when Druid upgrades to Calcite 1.38+.
Tests
SqlIntervalWeekRewriteShuttleTestexercises the shuttle directly against constructedSqlIntervalLiteralandSqlCallnodes for WEEK, QUARTER, DAY and MONTH (positive, negative, multi-unit), and against parsed queries that use both quoted and unquoted week and quarter intervals.CalciteQueryTest#testIntervalWeekResolutionis the end-to-end regression test from INTERVAL 1 WEEK in SQL is wrong interval #18665. Without the fix it produces an interval narrowing on2000-01-01T01:00:00.001/2000-01-01T02:00:00.001(the bug); with the fix it correctly narrows to2000-01-08and2000-01-15.CalciteQueryTest#testIntervalQuarterResolutionis the companion regression for QUARTER:INTERVAL '1' QUARTERnarrows to2000-04-01(three months later) andINTERVAL 2 QUARTERto2000-07-01.I verified the bug is reachable by temporarily disabling the shuttle and watching the regression tests fail with exactly the actual-vs-expected from the issue. Re-enabling the shuttle makes them pass.
I also re-ran the surrounding test classes to make sure the rewrite did not regress any existing INTERVAL behavior:
CalciteQueryTest(441 tests)CalciteSelectQueryTest(67 tests)CalciteJoinQueryTest(594 tests)CalciteSubqueryTest(54 tests)CalciteParameterQueryTest(20 tests)CalciteInsertDmlTest/CalciteReplaceDmlTest(107 tests)ExpressionsTest(54 tests)DruidSqlParserUtilsTest(57 tests)SqlQuidemTest(10 tests)All green.
mvn -pl sql -am verify -DskipTests=true(checkstyle, forbiddenapis, animal-sniffer) andmvn -pl sql -P rat verify -DskipTests=true(license headers) also pass.Release note
Fixed an issue where
INTERVAL n WEEKin SQL queries resolved tonhours instead ofn * 7days, andINTERVAL n QUARTERresolved tonmonths instead ofn * 3months.MILLIS_TO_TIMESTAMP(0) + INTERVAL '1' WEEKnow correctly returns1970-01-08, andINTERVAL '1' QUARTERadds three months, matching the SQL standard and other databases.Key changed/added classes in this PR
SqlIntervalWeekRewriteShuttle(new, covers WEEK and QUARTER)DruidPlanner(calls the shuttle invalidate)SqlIntervalWeekRewriteShuttleTest(new)CalciteQueryTest(regression teststestIntervalWeekResolutionandtestIntervalQuarterResolution)This PR has: