feat(timeseriesql): InfluxQL compat overhaul - parser rewrite, query fixes, DDL hardening - #99
Merged
Merged
Conversation
…y fixes, DDL hardening Major areas: Parser/lexer rewrite (parser.rs, lexer.rs): - New masking scanner (scan_chars) for correct string/regex/paren handling in SELECT clause splitting, keyword matching, and comma splitting. Fixes panics on non-ASCII chars, regex-with-semicolons, and quoted identifiers containing keywords. - OR/AND precedence fixed (OR binds looser than AND). - Left-associative arithmetic (a - b - c == (a - b) - c). - Bare DISTINCT keyword parsed as function call. - Duplicate clause detection, modulo operator, bitwise rejection. - ::field/:tag cast validation; unsupported casts error loudly. - take_ident preserves source spelling for keyword-named identifiers. Query translator (to_clickhouse.rs): - fill() grid: TO anchor extends one step past the last bucket; fill(N) emits INTERPOLATE for generated rows; fill(null) is default for GROUP BY time() without explicit fill(); SELECT INTO never fills. - ORDER BY time DESC + fill wraps ascending fill in subquery. - tz() flows into bucket expressions, GROUP BY, ORDER BY, and fill anchors for DST-correct daily bucketing. - Per-series LIMIT/OFFSET via LIMIT n BY (tags). - Raw selects with GROUP BY tag project time + tag without SQL GROUP BY. - Per-point transforms (difference, moving_average, elapsed, etc.) without GROUP BY time project raw time, order, and filter leading NULLs. - moving_average gates on full window (count >= N). - elapsed uses toNullable(time) for correct NULL defaulting. - Tag compared to numeric literal emits constant-false (1 = 0). - count(distinct(v)) â�� uniqExact; distinct(v) â�� arrayJoin(groupUniqArray). - median â�� quantileExactInclusive(0.5); percentile â�� quantileExactLow; stddev â�� stddevSamp; mode â�� arrayElement(topKWeighted(...)). - Materialized views use ANY LEFT JOIN (prevents dropping fact rows). - Precise __time â�� time renaming (rename_time_bucket_alias). - Time bounds intersection: lower keeps max, upper keeps min. DDL parser hardening (ddl_parser.rs): - CREATE USER parses both WITH PASSWORD / WITH ALL PRIVILEGES orderings. - Negative durations rejected; zero normalizes to infinite sentinel. - SHOW ... ON/FROM in either order (parse_show_on_from). - SHOW MEASUREMENTS WITH MEASUREMENT rejected until implemented. - Trailing tokens rejected in parse_ddl_statement. - CREATE DATABASE duration validation, REPLICATION range check. - CREATE MATERIALIZED VIEW AS SELECT survives trailing-token check. Digest normalization (digest.rs): - Identifiers keep original case (InfluxQL is case-sensitive). - Binary expressions parenthesized for structural distinction. - ON clause, LIMIT/OFFSET, tz() included in normalization. Point coalesce (point_coalesce.rs): - group_and_coalesce_by_measurement replaces coalesce_points_within_measurements. Zero-merge case borrows via Cow instead of cloning. Duration safety (ast.rs): - to_nanos uses saturating_mul; to_clickhouse_interval week saturating. Tests: ~60 new unit tests across parser, to_clickhouse, ddl_parser, ast, digest, lexer. Integration tests for median, stddev, mode, distinct, count_distinct, fill grid, LIMIT BY, raw selects, per-point transforms, tag-numeric, and tz/DST-correct daily bucketing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Rewrites the InfluxQL query parser and translator for broad InfluxQL compatibility, hardens the DDL parser against silently-dropped clauses, and fixes correctness issues across the SQL generation layer.
Parser/lexer rewrite - New masking scanner (
scan_chars) replaces the ad-hoc state tracking insplit_clauses,split_top_level_commas,find_top_level_operator, andsplit_statements. This fixes panics on non-ASCII characters, semicolons inside regex literals, quoted identifiers containing keywords, and regex-vs-division ambiguity. Also adds OR/AND precedence (OR binds looser), left-associative arithmetic, bareDISTINCTkeyword, duplicate clause detection, modulo, bitwise rejection,::field/::tagcast validation, and source-spelling preservation for keyword-named identifiers.Query translator fixes -
fill()grid correctness (TO anchor extends one step,fill(N)emits INTERPOLATE,fill(null)is default, SELECT INTO never fills), ORDER BY DESC+fill subquery wrapping,tz()flowing into bucket/group-by/fill-anchor expressions for DST-correct daily bucketing, per-series LIMIT/OFFSET viaLIMIT n BY, raw selects with GROUP BY tag projecting without SQL GROUP BY, per-point transforms (difference/moving_average/elapsed) without GROUP BY time,moving_averagefull-window gating,elapsedtoNullable defaulting, tag-vs-numeric constant-false,count(distinct)/distinctcorrect translation,median/percentile/stddev/modematching InfluxQL semantics, materialized views using LEFT JOIN, and precise__time�timerenaming.DDL parser hardening -
CREATE USERparses bothWITH PASSWORD/WITH ALL PRIVILEGESorderings, negative durations rejected (zero normalizes to infinite), SHOW clauses accept ON/FROM in either order,SHOW MEASUREMENTS WITH MEASUREMENTrejected loudly, trailing tokens rejected, CREATE DATABASE duration validation, REPLICATION range check, and CREATE MATERIALIZED VIEW AS SELECT surviving the trailing-token check.Digest normalization - Identifiers keep original case (InfluxQL is case-sensitive), binary expressions are parenthesized for structural distinction, and ON/LIMIT/OFFSET/tz() are included.
Point coalesce -
group_and_coalesce_by_measurementreplacescoalesce_points_within_measurements; zero-merge case borrows viaCowinstead of cloning.Duration safety -
to_nanosusessaturating_mul.Related issues
Checklist
cargo fmt --all --checkpassescargo clippy --all-targets -- -D warningspassescargo test --libpassescargo test --test '*'passesdocs/user-guide/configuration.mdandconfig.toml.examplehyperbytedb_*)Release notes
Changed - Query parser and translator rewritten for broad InfluxQL compatibility: correct operator precedence, fill() grid semantics, tz()-aware bucketing, per-series LIMIT, per-point transforms, and aggregate function parity (median, percentile, stddev, mode, distinct, count(distinct)). DDL parser now rejects silently-dropped clauses (negative durations, SHOW MEASUREMENTS WITH MEASUREMENT, trailing tokens, CREATE USER ordering). Duration arithmetic saturates instead of overflowing.
Test plan
~60 new unit tests in
parser.rs,to_clickhouse.rs,ddl_parser.rs,ast.rs,digest.rs, andlexer.rscovering masking scanner edge cases, operator precedence, fill grid correctness, tz/DST bucketing, LIMIT BY, raw selects, per-point transforms, tag-numeric comparisons, non-ASCII handling, and DDL parsing. New integration tests incombination_tests.rsfor median, stddev, mode, distinct, count_distinct, fill grid (leading/trailing gaps, DESC, defaults, offset), LIMIT per-series, raw select with GROUP BY tag, per-point transforms without GROUP BY time, tag-numeric returning empty, and DST-correct daily bucketing. Existing tests updated to match corrected translations.