Skip to content

v1.1.0

Choose a tag to compare

View all tags
@orian orian released this 28 Sep 10:29
· 4 commits to main since this release
v1.1.0
40af0b4

The parser now follows ClickHouse 26.8 much more closely.

  • Silent SQL losses fixed: it no longer drops or changes clauses when printing. For example, DETACH used to print as DROP, SELECT 1, limit(1) became LIMIT (1), and INTERVAL … PRECEDING lost PRECEDING.
  • Much more valid SQL accepted: EXPLAIN in all forms, INTERSECT, PASTE/NATURAL JOIN, CREATE/DROP INDEX, {name:Identifier} parameters, table-function expressions, and every SYSTEM SYNC/FLUSH/START|STOP form.
  • SQL that ClickHouse rejects is rejected too.
  • Formatting: PrintVisitor is about twice as fast.
  • Traversal: Walk and the visitor reach every node.

⚠️ This minor release contains breaking API changes. Go 1.27 is required, ASTVisitor has new methods, some fields changed type or can now be nil, and positions and AST shapes changed. Read the Breaking section below (also in COMPATIBILITY.md) before upgrading.

Breaking

  • Go 1.27 is required (go.mod said go 1.21.0).
  • ASTVisitor has new methods: VisitStreamClause (#102), VisitCreateIndex
    and VisitDropIndex (#135). Visitors embedding DefaultASTVisitor or
    PrintVisitor inherit them; direct implementations must add them.
    StreamClause.Accept now dispatches to VisitStreamClause.
  • Changed types: ShowStmt.Format is *FormatClause, from the embedded
    OutputClauses (name in Format.Format.Name; was *StringLiteral), and
    prints unquoted, FORMAT JSON rather than FORMAT 'JSON' (#87, #49).
  • Fields that can now be nil, or hold other node types:
    • SubQuery.Select is nil for (EXPLAIN …); see SubQuery.Explain (#133);
    • TableIndex.Granularity is nil when GRANULARITY is omitted (#134);
    • ExplainStmt.Statement can be any statement, not only *SelectQuery, and
      is nil for EXPLAIN CURRENT TRANSACTION; ExplainStmt.Type is empty when
      the kind is omitted (#86);
    • SystemCtrlExpr.Cluster (the table) is nil when none is given (#108);
    • TableArgListExpr.Args can hold any expression (#129).
  • Positions (#92):
    • Pos()/End() of a quoted StringLiteral/Ident include the quotes; the
      LiteralPos/LiteralEnd/NamePos/NameEnd fields still exclude them;
    • End() of nodes closed by ), ] or } points after the delimiter
      (ParamExprList, FunctionExpr, ArrayParamList, MapLiteral,
      TableSchemaClause, TableArgListExpr, ComplexType, QueryParam, …);
    • a statement's [Pos(), End()) covers exactly its text (it used to stop
      early or include the ;);
    • corrected fields: CompressionCodec.RightParenPos (the )), column
      COMMENT LiteralPos (the string), TypedPlaceholder.RightBracePos (the
      }), Fill.FillPos (FILL), IsNullExpr/IsNotNullExpr.IsPos (IS;
      Pos() is the operand), ShowStmt/GrantPrivilegeStmt.StatementEnd;
    • SystemSyncExpr.End(), SystemFlushExpr.End(), ExplainStmt.End() and
      TableIndex.End() cover the whole node (#62, #114, #86, #134).
  • AST shapes:
    • ATTACH/DETACH keep their verb (IsAttach/IsDetach; DropStmt.Type()
      returns DETACH TABLE); DETACH TABLE t used to print as DROP TABLE t
      (#43, #52, #68);
    • a parenthesised set-operation operand is a group node: a SelectQuery with
      HasParen and the inner query in Group (#79, #93);
    • trailing FORMAT/query-level SETTINGS of a statement live in the embedded
      OutputClauses; after FORMAT on a SELECT in SelectQuery.OutputSettings
      (#87);
    • all parts of one TimeSeries target form a single TimeSeriesTargetClause
      (#42);
    • FROM t STREAM is TableExpr.Stream, not the alias STREAM (#81);
    • JoinExpr.Modifiers keeps GLOBAL, and can hold NATURAL/PASTE;
      a LOCAL JOIN b aliases a as LOCAL (#74, #89);
    • lower-case array join is an ARRAY JOIN (JoinExpr.Left is a
      *ColumnExprList), not a join on a table named by its first expression
      (#50);
    • SystemCtrlExpr.Type is the whole target, e.g. "REPLICATION QUEUES"
      (#108).

Changed behavior

  • Traversal: Walk (and Find, FindAll, WalkWithBreak, Transform)
    and DefaultASTVisitor now reach every child node and agree with each other
    (#105, #106). They used to skip, among others, EXCEPT operands, INSERT … VALUES rows, TTL actions, named-collection parameters, DISTINCT ON
    columns, projection ORDER BY columns and RENAME pairs. Map keys are
    walked in place, so rewriting one changes the AST. IntervalFrom and
    WindowFrameParam no longer visit their child twice (#98).
  • PrintVisitor prints every node like its String() (#98) and streams
    into one builder (#99). BeautifyVisitor emits every statement type (#51).
  • Printed SQL fixed (the old output was invalid or meant something else):
    • WINDOW after HAVING (#88); IF [NOT] EXISTS before the name in DROP INDEX|PROJECTION and ADD INDEX (#64);
    • PRECEDING/FOLLOWING after an INTERVAL frame bound (#109);
    • SYSTEM SYNC REPLICA (#62), SYSTEM FLUSH DISTRIBUTED (#114), SYSTEM STOP|START DISTRIBUTED SENDS without … SENDS SENDS (#108);
    • SELECT 1, limit(1) keeps the column instead of becoming LIMIT (1)
      (#138); PASTE/NATURAL JOIN are joins, not aliases (#89);
    • (3,) keeps its comma (#70), f(p)(DISTINCT x) its DISTINCT (#71),
      INSERT INTO FUNCTION its FUNCTION (#72), PARTITION ID its ID (#76),
      WITH TOTALS without GROUP BY (#77), WITH FILL (#47);
    • USING is always parenthesised and comma joins print , (#74); -1::T
      keeps the sign on the literal (#83); TOP n WITH TIES (#90); no double
      spaces in CREATE TABLE t AS …/CREATE DATABASE … ENGINE (#45, #68);
    • CREATE TABLE keeps its query-level SETTINGS, and repeated CREATE USER
      HOST/SETTINGS clauses are merged (#37).
  • Parsing: INTERSECT is a keyword, not an implicit alias (#107);
    SELECT 1 EXCEPT (SELECT 2) is a set operation, not a column transformer
    (#107); in an expression { always starts a query parameter, and a
    {'k': v} map literal is valid only in INSERT … VALUES rows and as a
    SETTINGS value (#50).

Newly rejected input

  • Select list and joins: a missing select list (SELECT, SELECT FROM t;
    a lone SELECT used to parse to zero statements) (#82); a comma before a
    clause other than FROM (SELECT a, ORDER BY a) (#138); a JOIN without
    ON/USING except CROSS, NATURAL, PASTE, ARRAY (#50); PASTE/
    NATURAL as an implicit alias, or with ON/USING, a strictness, CROSS
    or ARRAY (#89); GLOBAL ARRAY JOIN and LEFT GLOBAL JOIN (#74).
  • Expressions: column transformers (EXCEPT, APPLY, REPLACE) after
    anything but *, t.*, COLUMNS(…) or * LIKE 'p' (#50); WITH CUBE/ROLLUP
    after GROUP BY CUBE(…)/ROLLUP(…) (#50); a map literal in an expression
    (#50); a trailing comma except in a one-element tuple (f(a,), [1,])
    (#70).
  • Repeated or misplaced clauses: repeated engine clauses in CREATE TABLE,
    repeated CREATE USER clauses, GROUP BY modifiers out of order (#37);
    repeated TimeSeries target parts and RECENT without SAMPLES (#42); an
    alias, FINAL or SAMPLE after STREAM (#81); repeated WITH TIES and
    LIMIT … BY … WITH TIES (#90); FORMAT/SETTINGS after statements that
    take none (GRANT, SYSTEM, USE, SET, DELETE, USER/ROLE, CREATE FUNCTION/NAMED
    COLLECTION; used to be dropped silently), a quoted SHOW … FORMAT 'x', and
    INSERT … VALUES (…) FORMAT x (#87).
  • DDL: an empty column list () (#45); ATTACH OR REPLACE,
    ATTACH/DETACH of FUNCTION/ROLE/USER/NAMED COLLECTION, DROP … PERMANENTLY, CREATE DICTIONARY d without a definition (#68); OR REPLACE
    with IF NOT EXISTS in CREATE TABLE/VIEW (#50); several pairs in RENAME DATABASE (#50); a PARTITION value other than a literal, query parameter,
    tuple or CAST of one (DROP PARTITION p, PARTITION toDate(…)) (#50); a
    malformed UUID '…' (ClickHouse: CANNOT_PARSE_UUID) (#50).
  • SYSTEM: START|STOP DISTRIBUTED MERGES|FETCHES|TTL MERGES (used to print
    as … MERGES MERGES) (#108).
  • Access control (#50): several roles in ALTER ROLE … RENAME TO; ON CLUSTER before the last role name; WITH ADMIN OPTION on a privilege grant;
    a column list on a wildcard target; the target *.table; GRANT ADMIN OPTION ON ….

Additive

  • New types: OutputClauses (#87), StreamClause (#81), CreateIndex,
    DropIndex (#135).
  • New fields: CreateTable.Settings, WithTimeoutClause.WithTimeoutEnd
    (#37); TimeSeriesTargetClause.InnerUUID, EngineShorthand (#42);
    IsAttach/IsDetach/Permanently, DropDatabase.Modifier (#68);
    ColumnExprList.HasTrailingComma (#70); AlterTableDelete.InPartition
    (#76); SelectQuery.HasParen, Group, OutputSettings, Union,
    Intersect, IntersectModifier, ExceptModifier (#79, #87, #93, #107,
    #125); TableExpr.Stream (#81); LimitClause.WithTies, WithTiesEnd (#90);
    OrderExpr.Nulls, Collate (#47); SystemSyncExpr.SyncEnd, Target,
    OnCluster, Database, IfExists, Mode, From, CacheName (#62);
    SystemCtrlExpr.OnCluster (#108); SystemFlushExpr.AsyncInsertQueue,
    OnCluster, Tables, Settings (#114); ExplainStmt.Settings,
    ExplainEnd (#86); Ident.Param for {name:Identifier} (#131);
    SubQuery.Explain (#133); span fields CastExpr.RightParenPos,
    SubQuery.RightParenPos, UsingClause.UsingEnd, OrderExpr.OrderEnd,
    NamedCollectionParam.ParamEnd, IsNullExpr/IsNotNullExpr.NullEnd (#92).
  • New methods: VisitStreamClause, VisitCreateIndex, VisitDropIndex on
    DefaultASTVisitor and PrintVisitor; InsertStmt.IsTableFunction() (#72).
  • Newly accepted input:
    • SELECT: INTERSECT [DISTINCT|ALL], EXCEPT DISTINCT|ALL, bare UNION
      (#107, #125); (SELECT …) UNION … and a statement starting with (
      (#93); EXPLAIN without a kind, with settings, of any statement, and as a
      subquery FROM (EXPLAIN …) (#86, #133); any expression as a
      table-function argument (#129); {name:Identifier} as a table, database or
      other name (#131); PASTE JOIN, NATURAL … JOIN, GLOBAL <kind> JOIN
      (#74, #89); FROM t STREAM [BOUNDED] [UNORDERED] (#81); LIMIT n WITH TIES (#90); NULLS FIRST|LAST and COLLATE in ORDER BY (#47); a
      keyword-named function after a comma (SELECT 1, format(…)) (#138); map
      literals in INSERT … VALUES (#50).
    • DDL: CREATE [UNIQUE] INDEX and DROP INDEX (#135); an index without
      GRANULARITY (#134); CREATE TABLE … SETTINGS … SETTINGS … (#37);
      TimeSeries RECENT SAMPLES, INNER UUID, INNER ENGINE (#42); DETACH … PERMANENTLY, ATTACH DICTIONARY d, DROP|DETACH DATABASE … SYNC (#68);
      DROP [DETACHED] PARTITION ID, DELETE IN PARTITION (#76); FORMAT … SETTINGS on non-SELECT statements (#87); a final single-token clause at
      end of input, e.g. ALTER TABLE t FREEZE (#46).
    • SYSTEM: every SYNC form (#62), START|STOP targets with ON CLUSTER
      and table (#108), FLUSH LOGS [log, …], FLUSH DISTRIBUTED … SETTINGS,
      FLUSH ASYNC INSERT QUEUE (#114).

What's Changed

  • Parser: keep query-level CREATE TABLE SETTINGS and reject repeated clauses by @orian in #37
  • Parser: support TimeSeries RECENT SAMPLES and INNER UUID targets by @orian in #42
  • Parser: reject empty column list and fix double space before AS schema by @orian in #45
  • BeautifyVisitor: emit every statement type instead of dropping it by @orian in #51
  • Parser: add missing space after CASE operand before WHEN by @orian in #69
  • Parser: keep ATTACH and DETACH instead of printing CREATE and DROP by @orian in #68
  • Parser: keep the trailing comma of a one-element tuple by @orian in #70
  • Parser: print DISTINCT in a parametric aggregate's argument list by @orian in #71
  • Parser: keep FUNCTION in INSERT INTO [TABLE] FUNCTION by @orian in #72
  • Parser: keep GLOBAL on joins and parenthesise USING by @orian in #74
  • Parser: keep ID in PARTITION ID and parse it in DROP and DELETE by @orian in #76
  • Parser: print WITH TOTALS when a SELECT has no GROUP BY by @orian in #77
  • Parser: keep parentheses around a UNION/EXCEPT operand by @orian in #79
  • Parser: parse FROM t STREAM instead of reading STREAM as an alias by @orian in #81
  • Parser: print a sign directly against a numeric literal by @orian in #83
  • Parser: keep query output FORMAT/SETTINGS instead of dropping FORMAT by @orian in #87
  • Parser: support LIMIT n WITH TIES and keep TOP n in TOP n WITH TIES by @orian in #90
  • Parser: fix statement Pos/End spans and check them for every fixture by @orian in #92
  • Docs: add COMPATIBILITY.md for public API changes by @orian in #94
  • Parser: model parenthesised set-operation operands as group nodes by @orian in #96
  • Parser: support NULLS FIRST/LAST and COLLATE in ORDER BY elements by @orian in #97
  • PrintVisitor: print every subtree like String() and stop crashing by @orian in #100
  • PrintVisitor: stream expression nodes instead of calling String() (#99, part 1) by @orian in #101
  • PrintVisitor: stream SELECT-area nodes instead of calling String() (#99, part 2) by @orian in #103
  • PrintVisitor: stream DDL nodes instead of calling String() (#99, part 3) by @orian in #104
  • PrintVisitor: keep PRECEDING/FOLLOWING after an INTERVAL frame bound (#109, P1) by @orian in #110
  • Parser: keep REPLICA in SYSTEM SYNC and support the other SYNC forms (#62) by @orian in #113
  • PrintVisitor: stream ALTER nodes instead of calling String() (#99, part 4) by @orian in #112
  • Parser: fix SYSTEM START|STOP targets; stop reading SENDS as a table (#108, P1) by @orian in #116
  • Walk: visit every child the visitor reaches (#105, P1) by @orian in #117
  • Visitor: recurse into every child Walk reaches; add VisitStreamClause (#106, #102) by @orian in #118
  • Print WINDOW after HAVING, where ClickHouse expects it (#88) by @orian in #119
  • Print IF [NOT] EXISTS before the name in ALTER DROP/ADD INDEX|PROJECTION (#64) by @orian in #121
  • Parser: keep DISTRIBUTED in SYSTEM FLUSH and support the other FLUSH forms (#114) by @orian in #122
  • Parser: accept EXPLAIN without a kind, all kinds, settings and any statement (#86) by @orian in #124
  • Parser: support INTERSECT, EXCEPT DISTINCT|ALL and bare UNION (#107, #125) by @orian in #127
  • PrintVisitor: stream the remaining statements (#99, part 5, final) by @orian in #128
  • Parser: fix end-of-input token skipping, empty select lists, and keyword calls after a comma (#46, #82, #138 P1) by @orian in #139
  • Add a gofmt pre-commit hook and a gofmt check in CI by @orian in #140
  • Make the test suite faster: no coverage in make test; require Go 1.27 by @orian in #142
  • Parser: accept any expression as a table-function argument (#129) by @orian in #141
  • Parser: accept {name:Identifier} query parameters as names (#131) by @orian in #144
  • Parser: make GRANULARITY optional in index definitions (#134) by @orian in #145
  • Parser: accept EXPLAIN as a subquery, FROM (EXPLAIN ...) (#133) by @orian in #146
  • Add String/PrintVisitor benchmarks and a cross-version benchmark script by @orian in #148
  • Parser: support PASTE JOIN and NATURAL JOIN (#89) by @orian in #147
  • Parser: support CREATE INDEX and DROP INDEX statements (#135) by @orian in #149
  • Parser: reject SELECT forms ClickHouse rejects; accept maps in VALUES (#50, part 1) by @orian in #150
  • Parser: reject DDL forms ClickHouse rejects (#50, part 2) by @orian in #151
  • Parser: reject role and GRANT forms ClickHouse rejects (#50, part 3) by @orian in #153
  • Release v1.1.0 by @orian in #154

Full Changelog: v1.0.4...v1.1.0

Contributors

orian
Assets 2
Loading