Skip to content

feat(timeseriesql): InfluxQL compat overhaul - parser rewrite, query fixes, DDL hardening - #99

Merged
austin-barrington merged 1 commit into
mainfrom
feat/bug_bashing
Jul 10, 2026
Merged

feat(timeseriesql): InfluxQL compat overhaul - parser rewrite, query fixes, DDL hardening#99
austin-barrington merged 1 commit into
mainfrom
feat/bug_bashing

Conversation

@austin-barrington

@austin-barrington austin-barrington commented Jul 10, 2026

Copy link
Copy Markdown
Member

Description

Rewrites the InfluxQL query parser and translator for broad InfluxQL compatibility, hardens the DDL parser against silently-dropped clauses, and fixes correctness issues across the SQL generation layer.

Parser/lexer rewrite - New masking scanner (scan_chars) replaces the ad-hoc state tracking in split_clauses, split_top_level_commas, find_top_level_operator, and split_statements. This fixes panics on non-ASCII characters, semicolons inside regex literals, quoted identifiers containing keywords, and regex-vs-division ambiguity. Also adds OR/AND precedence (OR binds looser), left-associative arithmetic, bare DISTINCT keyword, duplicate clause detection, modulo, bitwise rejection, ::field/::tag cast validation, and source-spelling preservation for keyword-named identifiers.

Query translator fixes - fill() grid correctness (TO anchor extends one step, fill(N) emits INTERPOLATE, fill(null) is default, SELECT INTO never fills), ORDER BY DESC+fill subquery wrapping, tz() flowing into bucket/group-by/fill-anchor expressions for DST-correct daily bucketing, per-series LIMIT/OFFSET via LIMIT n BY, raw selects with GROUP BY tag projecting without SQL GROUP BY, per-point transforms (difference/moving_average/elapsed) without GROUP BY time, moving_average full-window gating, elapsed toNullable defaulting, tag-vs-numeric constant-false, count(distinct)/distinct correct translation, median/percentile/stddev/mode matching InfluxQL semantics, materialized views using LEFT JOIN, and precise __time�time renaming.

DDL parser hardening - CREATE USER parses both WITH PASSWORD/WITH ALL PRIVILEGES orderings, negative durations rejected (zero normalizes to infinite), SHOW clauses accept ON/FROM in either order, SHOW MEASUREMENTS WITH MEASUREMENT rejected loudly, trailing tokens rejected, CREATE DATABASE duration validation, REPLICATION range check, and CREATE MATERIALIZED VIEW AS SELECT surviving the trailing-token check.

Digest normalization - Identifiers keep original case (InfluxQL is case-sensitive), binary expressions are parenthesized for structural distinction, and ON/LIMIT/OFFSET/tz() are included.

Point coalesce - group_and_coalesce_by_measurement replaces coalesce_points_within_measurements; zero-merge case borrows via Cow instead of cloning.

Duration safety - to_nanos uses saturating_mul.

Related issues

Checklist

  • cargo fmt --all --check passes
  • cargo clippy --all-targets -- -D warnings passes
  • cargo test --lib passes
  • cargo test --test '*' passes
  • New features have documentation (user-guide or developer-guide)
  • New config keys have entries in docs/user-guide/configuration.md and config.toml.example
  • New metrics follow the naming convention (hyperbytedb_*)
  • Breaking changes are noted in the PR description

Release notes

Changed - Query parser and translator rewritten for broad InfluxQL compatibility: correct operator precedence, fill() grid semantics, tz()-aware bucketing, per-series LIMIT, per-point transforms, and aggregate function parity (median, percentile, stddev, mode, distinct, count(distinct)). DDL parser now rejects silently-dropped clauses (negative durations, SHOW MEASUREMENTS WITH MEASUREMENT, trailing tokens, CREATE USER ordering). Duration arithmetic saturates instead of overflowing.

Test plan

~60 new unit tests in parser.rs, to_clickhouse.rs, ddl_parser.rs, ast.rs, digest.rs, and lexer.rs covering masking scanner edge cases, operator precedence, fill grid correctness, tz/DST bucketing, LIMIT BY, raw selects, per-point transforms, tag-numeric comparisons, non-ASCII handling, and DDL parsing. New integration tests in combination_tests.rs for median, stddev, mode, distinct, count_distinct, fill grid (leading/trailing gaps, DESC, defaults, offset), LIMIT per-series, raw select with GROUP BY tag, per-point transforms without GROUP BY time, tag-numeric returning empty, and DST-correct daily bucketing. Existing tests updated to match corrected translations.

…y fixes, DDL hardening

Major areas:

Parser/lexer rewrite (parser.rs, lexer.rs):
- New masking scanner (scan_chars) for correct string/regex/paren handling
  in SELECT clause splitting, keyword matching, and comma splitting.
  Fixes panics on non-ASCII chars, regex-with-semicolons, and quoted
  identifiers containing keywords.
- OR/AND precedence fixed (OR binds looser than AND).
- Left-associative arithmetic (a - b - c == (a - b) - c).
- Bare DISTINCT keyword parsed as function call.
- Duplicate clause detection, modulo operator, bitwise rejection.
- ::field/:tag cast validation; unsupported casts error loudly.
- take_ident preserves source spelling for keyword-named identifiers.

Query translator (to_clickhouse.rs):
- fill() grid: TO anchor extends one step past the last bucket; fill(N)
  emits INTERPOLATE for generated rows; fill(null) is default for
  GROUP BY time() without explicit fill(); SELECT INTO never fills.
- ORDER BY time DESC + fill wraps ascending fill in subquery.
- tz() flows into bucket expressions, GROUP BY, ORDER BY, and fill
  anchors for DST-correct daily bucketing.
- Per-series LIMIT/OFFSET via LIMIT n BY (tags).
- Raw selects with GROUP BY tag project time + tag without SQL GROUP BY.
- Per-point transforms (difference, moving_average, elapsed, etc.)
  without GROUP BY time project raw time, order, and filter leading NULLs.
- moving_average gates on full window (count >= N).
- elapsed uses toNullable(time) for correct NULL defaulting.
- Tag compared to numeric literal emits constant-false (1 = 0).
- count(distinct(v)) � uniqExact; distinct(v) � arrayJoin(groupUniqArray).
- median � quantileExactInclusive(0.5); percentile � quantileExactLow;
  stddev � stddevSamp; mode � arrayElement(topKWeighted(...)).
- Materialized views use ANY LEFT JOIN (prevents dropping fact rows).
- Precise __time � time renaming (rename_time_bucket_alias).
- Time bounds intersection: lower keeps max, upper keeps min.

DDL parser hardening (ddl_parser.rs):
- CREATE USER parses both WITH PASSWORD / WITH ALL PRIVILEGES orderings.
- Negative durations rejected; zero normalizes to infinite sentinel.
- SHOW ... ON/FROM in either order (parse_show_on_from).
- SHOW MEASUREMENTS WITH MEASUREMENT rejected until implemented.
- Trailing tokens rejected in parse_ddl_statement.
- CREATE DATABASE duration validation, REPLICATION range check.
- CREATE MATERIALIZED VIEW AS SELECT survives trailing-token check.

Digest normalization (digest.rs):
- Identifiers keep original case (InfluxQL is case-sensitive).
- Binary expressions parenthesized for structural distinction.
- ON clause, LIMIT/OFFSET, tz() included in normalization.

Point coalesce (point_coalesce.rs):
- group_and_coalesce_by_measurement replaces coalesce_points_within_measurements.
  Zero-merge case borrows via Cow instead of cloning.

Duration safety (ast.rs):
- to_nanos uses saturating_mul; to_clickhouse_interval week saturating.

Tests: ~60 new unit tests across parser, to_clickhouse, ddl_parser, ast,
digest, lexer. Integration tests for median, stddev, mode, distinct,
count_distinct, fill grid, LIMIT BY, raw selects, per-point transforms,
tag-numeric, and tz/DST-correct daily bucketing.
@austin-barrington
austin-barrington merged commit 1d09117 into main Jul 10, 2026
4 checks passed
@austin-barrington
austin-barrington deleted the feat/bug_bashing branch July 10, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant