From 06d888c6ef0ec4aa9935552a4d94da039cce5729 Mon Sep 17 00:00:00 2001 From: Xiang Fu Date: Fri, 17 Apr 2026 19:58:09 -0700 Subject: [PATCH 01/32] Add SQL DDL support for CREATE/DROP TABLE and SHOW CREATE TABLE (Slices 1+2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduces a new `pinot-sql-ddl` module and a `/sql/ddl` controller endpoint that compile and execute Pinot DDL statements: - CREATE TABLE: compiles DDL → Schema + TableConfig, runs the full validation stack (TableConfigUtils, TaskConfigUtils, TableConfigValidatorRegistry), and persists via PinotHelixResourceManager. Rejects sibling-table creation when the DDL column list conflicts with an existing stored schema. - DROP TABLE: drops OFFLINE, REALTIME, or both, with IF EXISTS support. - SHOW TABLES: lists all raw table names in the resolved database scope. - SHOW CREATE TABLE: emits canonical round-trip DDL via CanonicalDdlEmitter. Key components: - DdlCompiler: Calcite/JavaCC parser → ResolvedTableDefinition → Schema + TableConfig via PropertyMapping (promoted scalars, stream.*, task.*, JSON blobs, custom-config fall-through). - PropertyExtractor: inverse of PropertyMapping; emits a deterministic lexicographically-sorted property map from a TableConfig for canonical DDL. - CanonicalDdlEmitter: renders canonical CREATE TABLE SQL; preserves db-qualified table names and emits all sortedColumn values as CSV. - SchemaEmitter: emits column declarations including DIMENSION ARRAY (MV) syntax for multi-value fields. - Auth-before-existence checks throughout to prevent metadata leakage. - 53 unit tests covering compiler, round-trip, and emitter paths. Co-Authored-By: Claude Sonnet 4.6 --- pinot-common/src/main/codegen/config.fmpp | 25 + .../src/main/codegen/includes/parserImpls.ftl | 187 ++++++ .../pinot/sql/parsers/CalciteSqlParser.java | 15 + .../parser/SqlPinotColumnDeclaration.java | 168 ++++++ .../parsers/parser/SqlPinotCreateTable.java | 134 +++++ .../sql/parsers/parser/SqlPinotDropTable.java | 100 ++++ .../sql/parsers/parser/SqlPinotProperty.java | 87 +++ .../parser/SqlPinotShowCreateTable.java | 88 +++ .../parsers/parser/SqlPinotShowTables.java | 77 +++ .../pinot/sql/parsers/PinotDdlParserTest.java | 357 ++++++++++++ pinot-controller/pom.xml | 4 + .../resources/PinotDdlRestletResource.java | 546 ++++++++++++++++++ .../resources/ddl/DdlExecutionRequest.java | 37 ++ .../resources/ddl/DdlExecutionResponse.java | 196 +++++++ .../api/PinotDdlRestletResourceTest.java | 337 +++++++++++ pinot-sql-ddl/pom.xml | 65 +++ .../sql/ddl/compile/CompiledCreateTable.java | 52 ++ .../pinot/sql/ddl/compile/CompiledDdl.java | 56 ++ .../sql/ddl/compile/CompiledDropTable.java | 57 ++ .../ddl/compile/CompiledShowCreateTable.java | 57 ++ .../sql/ddl/compile/CompiledShowTables.java | 30 + .../pinot/sql/ddl/compile/DataTypeMapper.java | 78 +++ .../ddl/compile/DdlCompilationException.java | 34 ++ .../pinot/sql/ddl/compile/DdlCompiler.java | 385 ++++++++++++ .../pinot/sql/ddl/compile/DdlOperation.java | 27 + .../sql/ddl/compile/PropertyMapping.java | 473 +++++++++++++++ .../pinot/sql/ddl/resolved/ColumnRole.java | 32 + .../resolved/ResolvedColumnDefinition.java | 88 +++ .../ddl/resolved/ResolvedTableDefinition.java | 83 +++ .../sql/ddl/reverse/CanonicalDdlEmitter.java | 132 +++++ .../sql/ddl/reverse/PropertyExtractor.java | 298 ++++++++++ .../pinot/sql/ddl/reverse/SchemaEmitter.java | 142 +++++ .../pinot/sql/ddl/reverse/SqlIdentifiers.java | 84 +++ .../sql/ddl/compile/DdlCompilerTest.java | 359 ++++++++++++ .../ddl/reverse/CanonicalDdlEmitterTest.java | 309 ++++++++++ .../sql/ddl/roundtrip/RoundTripTest.java | 355 ++++++++++++ pom.xml | 6 + 37 files changed, 5560 insertions(+) create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotColumnDeclaration.java create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotDropTable.java create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotProperty.java create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowCreateTable.java create mode 100644 pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java create mode 100644 pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionRequest.java create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java create mode 100644 pinot-sql-ddl/pom.xml create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledCreateTable.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDdl.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDropTable.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowTables.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompilationException.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlOperation.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ColumnRole.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedColumnDefinition.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedTableDefinition.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SqlIdentifiers.java create mode 100644 pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java create mode 100644 pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java create mode 100644 pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java diff --git a/pinot-common/src/main/codegen/config.fmpp b/pinot-common/src/main/codegen/config.fmpp index 29f10c291981..5fc67ca23d03 100644 --- a/pinot-common/src/main/codegen/config.fmpp +++ b/pinot-common/src/main/codegen/config.fmpp @@ -53,6 +53,18 @@ data: { "BIG_DECIMAL" "STRING" "BYTES" + # Pinot DDL (CREATE/DROP/SHOW TABLE) keywords + "DIMENSION" + "METRIC" + "GRANULARITY" + "OFFLINE" + "REALTIME" + "PROPERTIES" + "TABLES" + "TABLE_TYPE" + # IF is not a SQL reserved word but Calcite core does not declare it as a token; we need it + # for DDL "IF [NOT] EXISTS" clauses. + "IF" ] # List of non-reserved keywords to add; @@ -70,6 +82,16 @@ data: { "DATETIME" "VARIANT" "UUID" + # Pinot DDL keywords kept non-reserved so users can still use them as identifiers + "DIMENSION" + "METRIC" + "GRANULARITY" + "OFFLINE" + "REALTIME" + "PROPERTIES" + "TABLES" + "TABLE_TYPE" + "IF" # The following keywords are reserved in core Calcite, # are reserved in some version of SQL, @@ -568,6 +590,9 @@ data: { statementParserMethods: [ "SqlInsertFromFile()" "SqlPhysicalExplain()" + "SqlPinotCreateTable()" + "SqlPinotDropTable()" + "SqlPinotShow()" ] # List of methods for parsing custom literals. diff --git a/pinot-common/src/main/codegen/includes/parserImpls.ftl b/pinot-common/src/main/codegen/includes/parserImpls.ftl index 6e1283b075ba..f12204829d78 100644 --- a/pinot-common/src/main/codegen/includes/parserImpls.ftl +++ b/pinot-common/src/main/codegen/includes/parserImpls.ftl @@ -109,3 +109,190 @@ SqlNode SqlPhysicalExplain() : nDynamicParams); } } + +/** + * CREATE TABLE [IF NOT EXISTS] [db.]name ( + * col TYPE [NULL | NOT NULL] [DEFAULT literal] + * [ DIMENSION | METRIC | DATETIME FORMAT 'fmt' GRANULARITY 'gran' ], + * ... + * ) + * TABLE_TYPE = OFFLINE | REALTIME + * [ PROPERTIES ( 'k' = 'v', ... ) ] + */ +SqlNode SqlPinotCreateTable() : +{ + SqlParserPos pos; + SqlIdentifier name; + boolean ifNotExists = false; + SqlNodeList columns; + SqlLiteral tableType; + SqlNodeList properties = null; +} +{ + { pos = getPos(); } + + [ LOOKAHEAD(3) { ifNotExists = true; } ] + name = CompoundIdentifier() + columns = PinotColumnList() + + tableType = PinotTableTypeLiteral() + [ properties = PinotPropertyList() ] + { + return new SqlPinotCreateTable(pos, name, ifNotExists, columns, tableType, properties); + } +} + +SqlNodeList PinotColumnList() : +{ + SqlParserPos pos; + SqlPinotColumnDeclaration col; + List list = new ArrayList(); +} +{ + { pos = getPos(); } + col = PinotColumnDeclaration() { list.add(col); } + ( col = PinotColumnDeclaration() { list.add(col); } )* + + { + return new SqlNodeList(list, pos.plus(getPos())); + } +} + +SqlPinotColumnDeclaration PinotColumnDeclaration() : +{ + SqlParserPos pos; + SqlIdentifier name; + SqlDataTypeSpec type; + boolean nullable = true; + SqlNode defaultValue = null; + String role = null; + SqlNode fmtNode = null; + SqlNode granNode = null; + boolean multiValue = false; +} +{ + name = SimpleIdentifier() { pos = getPos(); } + type = DataType() + [ + LOOKAHEAD(2) + { nullable = false; } + | + { nullable = true; } + ] + [ defaultValue = Literal() ] + [ + { role = "DIMENSION"; } [ { multiValue = true; } ] + | + { role = "METRIC"; } + | + { role = "DATETIME"; } + fmtNode = StringLiteral() + granNode = StringLiteral() + ] + { + return new SqlPinotColumnDeclaration(pos, name, type, nullable, defaultValue, role, + (SqlLiteral) fmtNode, (SqlLiteral) granNode, multiValue); + } +} + +SqlLiteral PinotTableTypeLiteral() : +{ + String value; + SqlParserPos pos; +} +{ + ( + { value = "OFFLINE"; pos = getPos(); } + | + { value = "REALTIME"; pos = getPos(); } + ) + { + return SqlLiteral.createCharString(value, pos); + } +} + +SqlNodeList PinotPropertyList() : +{ + SqlParserPos pos; + SqlPinotProperty prop; + List list = new ArrayList(); +} +{ + { pos = getPos(); } + [ + prop = PinotProperty() { list.add(prop); } + ( prop = PinotProperty() { list.add(prop); } )* + ] + + { + return new SqlNodeList(list, pos.plus(getPos())); + } +} + +SqlPinotProperty PinotProperty() : +{ + SqlParserPos pos; + SqlNode keyNode; + SqlNode valueNode; +} +{ + keyNode = StringLiteral() { pos = getPos(); } + valueNode = StringLiteral() + { + return new SqlPinotProperty(pos, (SqlLiteral) keyNode, (SqlLiteral) valueNode); + } +} + +/** + * DROP TABLE [IF EXISTS] [db.]name [TYPE OFFLINE | REALTIME] + */ +SqlNode SqlPinotDropTable() : +{ + SqlParserPos pos; + SqlIdentifier name; + boolean ifExists = false; + SqlLiteral tableType = null; +} +{ + { pos = getPos(); } +
+ [ LOOKAHEAD(2) { ifExists = true; } ] + name = CompoundIdentifier() + [ tableType = PinotTableTypeLiteral() ] + { + return new SqlPinotDropTable(pos, name, ifExists, tableType); + } +} + +/** + * SHOW TABLES [FROM db] + * | SHOW CREATE TABLE [db.]name [TYPE OFFLINE | REALTIME] + * + * Both grammar branches share a leading token; combining them into a single entry point + * keeps the JavaCC choice unambiguous (no need for LOOKAHEAD across multiple statementParser + * methods that all start with SHOW). + */ +SqlNode SqlPinotShow() : +{ + SqlParserPos pos; + SqlIdentifier name; + SqlIdentifier database = null; + SqlLiteral tableType = null; +} +{ + { pos = getPos(); } + ( + + [ database = SimpleIdentifier() ] + { + return new SqlPinotShowTables(pos, database); + } + | +
+ name = CompoundIdentifier() + [ tableType = PinotTableTypeLiteral() ] + { + return new SqlPinotShowCreateTable(pos, name, tableType); + } + ) +} diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java index 05ec83d6fc5e..8159836520bd 100644 --- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java +++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java @@ -66,6 +66,10 @@ import org.apache.pinot.sql.FilterKind; import org.apache.pinot.sql.parsers.parser.SqlInsertFromFile; import org.apache.pinot.sql.parsers.parser.SqlParserImpl; +import org.apache.pinot.sql.parsers.parser.SqlPinotCreateTable; +import org.apache.pinot.sql.parsers.parser.SqlPinotDropTable; +import org.apache.pinot.sql.parsers.parser.SqlPinotShowCreateTable; +import org.apache.pinot.sql.parsers.parser.SqlPinotShowTables; import org.apache.pinot.sql.parsers.rewriter.QueryRewriter; import org.apache.pinot.sql.parsers.rewriter.QueryRewriterFactory; import org.slf4j.Logger; @@ -137,6 +141,17 @@ private static SqlNodeAndOptions extractSqlNodeAndOptions(SqlNodeList sqlNodeLis } else { throw new SqlCompilationException("SqlNode with executable statement already exist with type: " + sqlType); } + } else if (sqlNode instanceof SqlPinotCreateTable + || sqlNode instanceof SqlPinotDropTable + || sqlNode instanceof SqlPinotShowTables + || sqlNode instanceof SqlPinotShowCreateTable) { + // Pinot-native DDL statements; the controller dispatches these via the DDL endpoint. + if (sqlType == null) { + sqlType = PinotSqlType.DDL; + statementNode = sqlNode; + } else { + throw new SqlCompilationException("SqlNode with executable statement already exist with type: " + sqlType); + } } else if (sqlNode instanceof SqlSetOption) { // extract options, these are non-execution statements List operandList = ((SqlSetOption) sqlNode).getOperandList(); diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotColumnDeclaration.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotColumnDeclaration.java new file mode 100644 index 000000000000..70b40c96c31c --- /dev/null +++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotColumnDeclaration.java @@ -0,0 +1,168 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.parsers.parser; + +import java.util.Arrays; +import java.util.List; +import javax.annotation.Nullable; +import org.apache.calcite.sql.SqlCall; +import org.apache.calcite.sql.SqlDataTypeSpec; +import org.apache.calcite.sql.SqlIdentifier; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.SqlLiteral; +import org.apache.calcite.sql.SqlNode; +import org.apache.calcite.sql.SqlOperator; +import org.apache.calcite.sql.SqlSpecialOperator; +import org.apache.calcite.sql.SqlWriter; +import org.apache.calcite.sql.parser.SqlParserPos; + + +/** + * Single column declaration inside the column list of a Pinot {@link SqlPinotCreateTable}. + * + *

Grammar: + * 

{@code
+ *   col_name DATA_TYPE [NULL | NOT NULL] [DEFAULT literal]
+ *     [ DIMENSION | METRIC | DATETIME FORMAT 'fmt' GRANULARITY 'gran' ]
+ * }
+ * + *

The {@code role} field is one of "DIMENSION", "METRIC", "DATETIME", or {@code null} + * (unspecified, defer inference to the compiler). When {@code role} is "DATETIME", the + * {@code dateTimeFormat} and {@code dateTimeGranularity} string literals are required. + * When {@code role} is "DIMENSION", the optional {@code ARRAY} suffix marks the column as + * multi-value (Pinot MV dimension). + * + *

DEFAULT semantics: the {@code DEFAULT literal} clause maps to Pinot's + * {@code FieldSpec.defaultNullValue}, NOT to standard SQL's "value substituted on insert when + * column is omitted". Pinot's defaultNullValue is applied at ingestion time when the source + * record contains a null/missing value for the column. Users coming from standard SQL should + * not expect this clause to interact with INSERT statements. + * + *

This class is not thread-safe; instances should not be mutated after construction. + */ +public class SqlPinotColumnDeclaration extends SqlCall { + private static final SqlSpecialOperator OPERATOR = + new SqlSpecialOperator("COLUMN_DECL", SqlKind.COLUMN_DECL); + + private final SqlIdentifier _columnName; + private final SqlDataTypeSpec _dataType; + private final boolean _nullable; + private final SqlNode _defaultValue; + private final String _role; + private final SqlLiteral _dateTimeFormat; + private final SqlLiteral _dateTimeGranularity; + private final boolean _multiValue; + + public SqlPinotColumnDeclaration(SqlParserPos pos, SqlIdentifier columnName, SqlDataTypeSpec dataType, + boolean nullable, @Nullable SqlNode defaultValue, @Nullable String role, + @Nullable SqlLiteral dateTimeFormat, @Nullable SqlLiteral dateTimeGranularity) { + this(pos, columnName, dataType, nullable, defaultValue, role, dateTimeFormat, dateTimeGranularity, false); + } + + public SqlPinotColumnDeclaration(SqlParserPos pos, SqlIdentifier columnName, SqlDataTypeSpec dataType, + boolean nullable, @Nullable SqlNode defaultValue, @Nullable String role, + @Nullable SqlLiteral dateTimeFormat, @Nullable SqlLiteral dateTimeGranularity, boolean multiValue) { + super(pos); + _columnName = columnName; + _dataType = dataType; + _nullable = nullable; + _defaultValue = defaultValue; + _role = role; + _dateTimeFormat = dateTimeFormat; + _dateTimeGranularity = dateTimeGranularity; + _multiValue = multiValue; + } + + public SqlIdentifier getColumnName() { + return _columnName; + } + + public SqlDataTypeSpec getDataType() { + return _dataType; + } + + public boolean isNullable() { + return _nullable; + } + + @Nullable + public SqlNode getDefaultValue() { + return _defaultValue; + } + + /** + * @return one of "DIMENSION", "METRIC", "DATETIME", or {@code null} when unspecified. + */ + @Nullable + public String getRole() { + return _role; + } + + @Nullable + public SqlLiteral getDateTimeFormat() { + return _dateTimeFormat; + } + + @Nullable + public SqlLiteral getDateTimeGranularity() { + return _dateTimeGranularity; + } + + /** Returns true if this is a multi-value dimension (declared with {@code DIMENSION ARRAY}). */ + public boolean isMultiValue() { + return _multiValue; + } + + @Override + public SqlOperator getOperator() { + return OPERATOR; + } + + @Override + public List getOperandList() { + return Arrays.asList(_columnName, _dataType, _defaultValue, _dateTimeFormat, _dateTimeGranularity); + } + + @Override + public void unparse(SqlWriter writer, int leftPrec, int rightPrec) { + _columnName.unparse(writer, 0, 0); + _dataType.unparse(writer, 0, 0); + if (!_nullable) { + writer.keyword("NOT NULL"); + } + if (_defaultValue != null) { + writer.keyword("DEFAULT"); + _defaultValue.unparse(writer, 0, 0); + } + if (_role != null) { + if ("DATETIME".equals(_role)) { + writer.keyword("DATETIME"); + writer.keyword("FORMAT"); + _dateTimeFormat.unparse(writer, 0, 0); + writer.keyword("GRANULARITY"); + _dateTimeGranularity.unparse(writer, 0, 0); + } else { + writer.keyword(_role); + if (_multiValue) { + writer.keyword("ARRAY"); + } + } + } + } +} diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java new file mode 100644 index 000000000000..66a20e829fd2 --- /dev/null +++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java @@ -0,0 +1,134 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.parsers.parser; + +import java.util.Arrays; +import java.util.List; +import javax.annotation.Nullable; +import org.apache.calcite.sql.SqlCall; +import org.apache.calcite.sql.SqlIdentifier; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.SqlLiteral; +import org.apache.calcite.sql.SqlNode; +import org.apache.calcite.sql.SqlNodeList; +import org.apache.calcite.sql.SqlOperator; +import org.apache.calcite.sql.SqlSpecialOperator; +import org.apache.calcite.sql.SqlWriter; +import org.apache.calcite.sql.parser.SqlParserPos; + + +/** + * Pinot-native {@code CREATE TABLE} DDL statement. + * + *

Syntax (Phase 1, Slice 1): + * 

{@code
+ *   CREATE TABLE [IF NOT EXISTS] [db.]name (
+ *     col TYPE [NULL | NOT NULL] [DEFAULT literal] [DIMENSION | METRIC],
+ *     col TYPE DATETIME FORMAT 'fmt' GRANULARITY 'gran',
+ *     ...
+ *   )
+ *   TABLE_TYPE = OFFLINE | REALTIME
+ *   PROPERTIES (
+ *     'key' = 'value',
+ *     ...
+ *   )
+ * }
+ * + *

This is a parse-time AST node only. Semantic validation (data type recognition, role + * inference, property mapping) happens in {@code DdlCompiler} in the {@code pinot-sql-ddl} module. + * + *

This class is not thread-safe; instances should not be mutated after construction. + */ +public class SqlPinotCreateTable extends SqlCall { + private static final SqlSpecialOperator OPERATOR = + new SqlSpecialOperator("CREATE_TABLE", SqlKind.CREATE_TABLE); + + private final SqlIdentifier _name; + private final boolean _ifNotExists; + private final SqlNodeList _columns; + private final SqlLiteral _tableType; + private final SqlNodeList _properties; + + public SqlPinotCreateTable(SqlParserPos pos, SqlIdentifier name, boolean ifNotExists, SqlNodeList columns, + SqlLiteral tableType, @Nullable SqlNodeList properties) { + super(pos); + _name = name; + _ifNotExists = ifNotExists; + _columns = columns; + _tableType = tableType; + _properties = properties == null ? SqlNodeList.EMPTY : properties; + } + + public SqlIdentifier getName() { + return _name; + } + + public boolean isIfNotExists() { + return _ifNotExists; + } + + public SqlNodeList getColumns() { + return _columns; + } + + public SqlLiteral getTableType() { + return _tableType; + } + + public SqlNodeList getProperties() { + return _properties; + } + + @Override + public SqlOperator getOperator() { + return OPERATOR; + } + + @Override + public List getOperandList() { + return Arrays.asList(_name, _columns, _tableType, _properties); + } + + @Override + public void unparse(SqlWriter writer, int leftPrec, int rightPrec) { + writer.keyword("CREATE TABLE"); + if (_ifNotExists) { + writer.keyword("IF NOT EXISTS"); + } + _name.unparse(writer, leftPrec, rightPrec); + SqlWriter.Frame columnFrame = writer.startList("(", ")"); + for (SqlNode column : _columns) { + writer.sep(","); + column.unparse(writer, 0, 0); + } + writer.endList(columnFrame); + writer.keyword("TABLE_TYPE"); + writer.keyword("="); + writer.keyword(_tableType.toValue()); + if (!_properties.isEmpty()) { + writer.keyword("PROPERTIES"); + SqlWriter.Frame propFrame = writer.startList("(", ")"); + for (SqlNode property : _properties) { + writer.sep(","); + property.unparse(writer, 0, 0); + } + writer.endList(propFrame); + } + } +} diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotDropTable.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotDropTable.java new file mode 100644 index 000000000000..2640872da0d4 --- /dev/null +++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotDropTable.java @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.parsers.parser; + +import java.util.Arrays; +import java.util.List; +import javax.annotation.Nullable; +import org.apache.calcite.sql.SqlCall; +import org.apache.calcite.sql.SqlIdentifier; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.SqlLiteral; +import org.apache.calcite.sql.SqlNode; +import org.apache.calcite.sql.SqlOperator; +import org.apache.calcite.sql.SqlSpecialOperator; +import org.apache.calcite.sql.SqlWriter; +import org.apache.calcite.sql.parser.SqlParserPos; + + +/** + * Pinot-native {@code DROP TABLE} DDL statement. + * + *

Syntax: {@code DROP TABLE [IF EXISTS] [db.]name [TYPE OFFLINE | REALTIME]} + * + *

The optional {@code TYPE} clause restricts the drop to one physical table when the logical + * name has both OFFLINE and REALTIME variants. When absent, both variants are dropped. + * + *

This class is not thread-safe; instances should not be mutated after construction. + */ +public class SqlPinotDropTable extends SqlCall { + private static final SqlSpecialOperator OPERATOR = + new SqlSpecialOperator("DROP_TABLE", SqlKind.DROP_TABLE); + + private final SqlIdentifier _name; + private final boolean _ifExists; + private final SqlLiteral _tableType; + + public SqlPinotDropTable(SqlParserPos pos, SqlIdentifier name, boolean ifExists, + @Nullable SqlLiteral tableType) { + super(pos); + _name = name; + _ifExists = ifExists; + _tableType = tableType; + } + + public SqlIdentifier getName() { + return _name; + } + + public boolean isIfExists() { + return _ifExists; + } + + /** + * @return the explicit table type ("OFFLINE" or "REALTIME") if specified, or {@code null} for + * "drop both variants". + */ + @Nullable + public SqlLiteral getTableType() { + return _tableType; + } + + @Override + public SqlOperator getOperator() { + return OPERATOR; + } + + @Override + public List getOperandList() { + return Arrays.asList(_name, _tableType); + } + + @Override + public void unparse(SqlWriter writer, int leftPrec, int rightPrec) { + writer.keyword("DROP TABLE"); + if (_ifExists) { + writer.keyword("IF EXISTS"); + } + _name.unparse(writer, leftPrec, rightPrec); + if (_tableType != null) { + writer.keyword("TYPE"); + writer.keyword(_tableType.toValue()); + } + } +} diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotProperty.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotProperty.java new file mode 100644 index 000000000000..b0077ab752c1 --- /dev/null +++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotProperty.java @@ -0,0 +1,87 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.parsers.parser; + +import java.util.Arrays; +import java.util.List; +import org.apache.calcite.sql.SqlCall; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.SqlLiteral; +import org.apache.calcite.sql.SqlNode; +import org.apache.calcite.sql.SqlOperator; +import org.apache.calcite.sql.SqlSpecialOperator; +import org.apache.calcite.sql.SqlWriter; +import org.apache.calcite.sql.parser.SqlParserPos; + + +/** + * One {@code 'key' = 'value'} entry inside a {@code PROPERTIES (...)} clause. + * + *

Both key and value are parsed as quoted string literals to keep the property surface area + * uniform and forward-compatible. This lets stream/minion-task configs (whose keys evolve outside + * the DDL grammar) be passed through verbatim without grammar changes. + * + *

This class is not thread-safe; instances should not be mutated after construction. + */ +public class SqlPinotProperty extends SqlCall { + private static final SqlSpecialOperator OPERATOR = + new SqlSpecialOperator("PROPERTY", SqlKind.OTHER); + + private final SqlLiteral _key; + private final SqlLiteral _value; + + public SqlPinotProperty(SqlParserPos pos, SqlLiteral key, SqlLiteral value) { + super(pos); + _key = key; + _value = value; + } + + public SqlLiteral getKey() { + return _key; + } + + public SqlLiteral getValue() { + return _value; + } + + public String getKeyString() { + return _key.toValue(); + } + + public String getValueString() { + return _value.toValue(); + } + + @Override + public SqlOperator getOperator() { + return OPERATOR; + } + + @Override + public List getOperandList() { + return Arrays.asList(_key, _value); + } + + @Override + public void unparse(SqlWriter writer, int leftPrec, int rightPrec) { + _key.unparse(writer, 0, 0); + writer.keyword("="); + _value.unparse(writer, 0, 0); + } +} diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowCreateTable.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowCreateTable.java new file mode 100644 index 000000000000..2f4016243a92 --- /dev/null +++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowCreateTable.java @@ -0,0 +1,88 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.parsers.parser; + +import java.util.Arrays; +import java.util.List; +import javax.annotation.Nullable; +import org.apache.calcite.sql.SqlCall; +import org.apache.calcite.sql.SqlIdentifier; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.SqlLiteral; +import org.apache.calcite.sql.SqlNode; +import org.apache.calcite.sql.SqlOperator; +import org.apache.calcite.sql.SqlSpecialOperator; +import org.apache.calcite.sql.SqlWriter; +import org.apache.calcite.sql.parser.SqlParserPos; + + +/** + * Pinot-native {@code SHOW CREATE TABLE [db.]name [TYPE OFFLINE | REALTIME]} DDL statement. + * + *

Parse-time AST node only. The optional {@code TYPE OFFLINE | REALTIME} clause carries the + * caller's explicit preference; absence leaves the choice to the executor. Disambiguation policy + * for the both-variants-exist case is the controller's responsibility, not the parser's. + * + *

This class is not thread-safe; instances should not be mutated after construction. + */ +public class SqlPinotShowCreateTable extends SqlCall { + private static final SqlSpecialOperator OPERATOR = + new SqlSpecialOperator("SHOW_CREATE_TABLE", SqlKind.OTHER_DDL); + + private final SqlIdentifier _name; + private final SqlLiteral _tableType; + + public SqlPinotShowCreateTable(SqlParserPos pos, SqlIdentifier name, @Nullable SqlLiteral tableType) { + super(pos); + _name = name; + _tableType = tableType; + } + + public SqlIdentifier getName() { + return _name; + } + + /** + * @return the explicit table type ("OFFLINE" or "REALTIME") if specified, else {@code null}. + */ + @Nullable + public SqlLiteral getTableType() { + return _tableType; + } + + @Override + public SqlOperator getOperator() { + return OPERATOR; + } + + @Override + public List getOperandList() { + return Arrays.asList(_name, _tableType); + } + + @Override + public void unparse(SqlWriter writer, int leftPrec, int rightPrec) { + writer.keyword("SHOW CREATE TABLE"); + _name.unparse(writer, leftPrec, rightPrec); + if (_tableType != null) { + writer.keyword("TYPE"); + writer.keyword(_tableType.toValue()); + } + } +} diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java new file mode 100644 index 000000000000..13793d45b6ee --- /dev/null +++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.parsers.parser; + +import java.util.Collections; +import java.util.List; +import javax.annotation.Nullable; +import org.apache.calcite.sql.SqlCall; +import org.apache.calcite.sql.SqlIdentifier; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.SqlNode; +import org.apache.calcite.sql.SqlOperator; +import org.apache.calcite.sql.SqlSpecialOperator; +import org.apache.calcite.sql.SqlWriter; +import org.apache.calcite.sql.parser.SqlParserPos; + + +/** + * Pinot-native {@code SHOW TABLES [FROM db]} DDL statement. Lists tables in the given database + * (or the default database when none is specified). + * + *

This class is not thread-safe; instances should not be mutated after construction. + */ +public class SqlPinotShowTables extends SqlCall { + private static final SqlSpecialOperator OPERATOR = + new SqlSpecialOperator("SHOW_TABLES", SqlKind.OTHER_DDL); + + private final SqlIdentifier _database; + + public SqlPinotShowTables(SqlParserPos pos, @Nullable SqlIdentifier database) { + super(pos); + _database = database; + } + + /** + * @return the explicit database identifier when {@code FROM } is supplied, else {@code null}. + */ + @Nullable + public SqlIdentifier getDatabase() { + return _database; + } + + @Override + public SqlOperator getOperator() { + return OPERATOR; + } + + @Override + public List getOperandList() { + return _database == null ? Collections.emptyList() : Collections.singletonList(_database); + } + + @Override + public void unparse(SqlWriter writer, int leftPrec, int rightPrec) { + writer.keyword("SHOW TABLES"); + if (_database != null) { + writer.keyword("FROM"); + _database.unparse(writer, leftPrec, rightPrec); + } + } +} diff --git a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java new file mode 100644 index 000000000000..b638a69a0ae0 --- /dev/null +++ b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java @@ -0,0 +1,357 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.parsers; + +import org.apache.calcite.sql.SqlNode; +import org.apache.pinot.sql.parsers.parser.SqlPinotColumnDeclaration; +import org.apache.pinot.sql.parsers.parser.SqlPinotCreateTable; +import org.apache.pinot.sql.parsers.parser.SqlPinotDropTable; +import org.apache.pinot.sql.parsers.parser.SqlPinotProperty; +import org.apache.pinot.sql.parsers.parser.SqlPinotShowCreateTable; +import org.apache.pinot.sql.parsers.parser.SqlPinotShowTables; +import org.testng.annotations.Test; + +import static org.testng.Assert.assertEquals; +import static org.testng.Assert.assertFalse; +import static org.testng.Assert.assertNotNull; +import static org.testng.Assert.assertNull; +import static org.testng.Assert.assertTrue; +import static org.testng.Assert.expectThrows; +import static org.testng.Assert.fail; + + +/** + * Parser-layer tests for the new Pinot DDL grammar (CREATE TABLE / DROP TABLE / SHOW TABLES). + * Verifies that statements parse, produce the expected AST shape, and that {@link CalciteSqlParser} + * classifies them as {@link PinotSqlType#DDL}. + */ +public class PinotDdlParserTest { + + // ------------------------------------------------------------------------------------------- + // CREATE TABLE + // ------------------------------------------------------------------------------------------- + + @Test + public void minimalCreateTable() { + SqlPinotCreateTable node = parseCreate( + "CREATE TABLE events (id INT, name STRING) TABLE_TYPE = OFFLINE"); + assertEquals(node.getName().getSimple(), "events"); + assertFalse(node.isIfNotExists()); + assertEquals(node.getColumns().size(), 2); + assertEquals(node.getTableType().toValue(), "OFFLINE"); + assertTrue(node.getProperties().isEmpty()); + } + + @Test + public void createTableWithIfNotExists() { + SqlPinotCreateTable node = parseCreate( + "CREATE TABLE IF NOT EXISTS events (id INT) TABLE_TYPE = OFFLINE"); + assertTrue(node.isIfNotExists()); + assertEquals(node.getName().getSimple(), "events"); + } + + @Test + public void createTableQualifiedName() { + SqlPinotCreateTable node = parseCreate( + "CREATE TABLE myDb.events (id INT) TABLE_TYPE = OFFLINE"); + assertEquals(node.getName().names.size(), 2); + assertEquals(node.getName().names.get(0), "myDb"); + assertEquals(node.getName().names.get(1), "events"); + } + + @Test + public void createTableRealtime() { + SqlPinotCreateTable node = parseCreate( + "CREATE TABLE events (id INT, ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' " + + "GRANULARITY '1:MILLISECONDS') TABLE_TYPE = REALTIME"); + assertEquals(node.getTableType().toValue(), "REALTIME"); + } + + @Test + public void createTableAllColumnModifiers() { + SqlPinotCreateTable node = parseCreate( + "CREATE TABLE t (" + + " d1 STRING DIMENSION," + + " d2 INT NOT NULL DIMENSION," + + " m1 LONG METRIC," + + " m2 DOUBLE NOT NULL DEFAULT 0.0 METRIC," + + " ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'" + + ") TABLE_TYPE = OFFLINE"); + assertEquals(node.getColumns().size(), 5); + SqlPinotColumnDeclaration d2 = (SqlPinotColumnDeclaration) node.getColumns().get(1); + assertFalse(d2.isNullable()); + assertEquals(d2.getRole(), "DIMENSION"); + + SqlPinotColumnDeclaration m2 = (SqlPinotColumnDeclaration) node.getColumns().get(3); + assertEquals(m2.getRole(), "METRIC"); + assertNotNull(m2.getDefaultValue()); + + SqlPinotColumnDeclaration ts = (SqlPinotColumnDeclaration) node.getColumns().get(4); + assertEquals(ts.getRole(), "DATETIME"); + assertNotNull(ts.getDateTimeFormat()); + assertEquals(ts.getDateTimeFormat().toValue(), "1:MILLISECONDS:EPOCH"); + assertEquals(ts.getDateTimeGranularity().toValue(), "1:MILLISECONDS"); + } + + @Test + public void createTableWithProperties() { + SqlPinotCreateTable node = parseCreate( + "CREATE TABLE events (id INT) TABLE_TYPE = OFFLINE PROPERTIES (" + + " 'replication' = '3'," + + " 'brokerTenant' = 'DefaultTenant'," + + " 'stream.kafka.topic.name' = 'orders'" + + ")"); + assertEquals(node.getProperties().size(), 3); + SqlPinotProperty first = (SqlPinotProperty) node.getProperties().get(0); + assertEquals(first.getKeyString(), "replication"); + assertEquals(first.getValueString(), "3"); + SqlPinotProperty third = (SqlPinotProperty) node.getProperties().get(2); + assertEquals(third.getKeyString(), "stream.kafka.topic.name"); + } + + @Test + public void createTableEmptyPropertiesIsAllowed() { + SqlPinotCreateTable node = parseCreate( + "CREATE TABLE events (id INT) TABLE_TYPE = OFFLINE PROPERTIES ()"); + assertTrue(node.getProperties().isEmpty()); + } + + @Test + public void createTablePropertyKeysWithDotsAndDashes() { + // Property keys are quoted literals so any character is acceptable. This is the + // forward-compatibility hook for stream/task/minion configs that evolve outside the grammar. + SqlPinotCreateTable node = parseCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES (" + + " 'task.RealtimeToOfflineSegmentsTask.bucketTimePeriod' = '1d'," + + " 'realtime.segment.flush.threshold.rows' = '500000'," + + " 'kafka.client-id' = 'pinot-events'" + + ")"); + assertEquals(node.getProperties().size(), 3); + } + + // ------------------------------------------------------------------------------------------- + // DROP TABLE + // ------------------------------------------------------------------------------------------- + + @Test + public void dropTableMinimal() { + SqlPinotDropTable node = parseDrop("DROP TABLE events"); + assertEquals(node.getName().getSimple(), "events"); + assertFalse(node.isIfExists()); + assertNull(node.getTableType()); + } + + @Test + public void dropTableIfExists() { + SqlPinotDropTable node = parseDrop("DROP TABLE IF EXISTS events"); + assertTrue(node.isIfExists()); + } + + @Test + public void dropTableWithType() { + SqlPinotDropTable node = parseDrop("DROP TABLE events TYPE OFFLINE"); + assertNotNull(node.getTableType()); + assertEquals(node.getTableType().toValue(), "OFFLINE"); + } + + @Test + public void dropTableQualifiedName() { + SqlPinotDropTable node = parseDrop("DROP TABLE IF EXISTS myDb.events TYPE REALTIME"); + assertEquals(node.getName().names.size(), 2); + assertTrue(node.isIfExists()); + assertEquals(node.getTableType().toValue(), "REALTIME"); + } + + // ------------------------------------------------------------------------------------------- + // SHOW TABLES + // ------------------------------------------------------------------------------------------- + + @Test + public void showTables() { + SqlPinotShowTables node = parseShow("SHOW TABLES"); + assertNull(node.getDatabase()); + } + + @Test + public void showTablesFromDatabase() { + SqlPinotShowTables node = parseShow("SHOW TABLES FROM analytics"); + assertNotNull(node.getDatabase()); + assertEquals(node.getDatabase().getSimple(), "analytics"); + } + + @Test + public void showCreateTableMinimal() { + SqlPinotShowCreateTable node = parseShowCreate("SHOW CREATE TABLE events"); + assertEquals(node.getName().getSimple(), "events"); + assertNull(node.getTableType()); + } + + @Test + public void showCreateTableQualifiedNameWithType() { + SqlPinotShowCreateTable node = parseShowCreate( + "SHOW CREATE TABLE analytics.events TYPE OFFLINE"); + assertEquals(node.getName().names.size(), 2); + assertEquals(node.getName().names.get(0), "analytics"); + assertEquals(node.getName().names.get(1), "events"); + assertNotNull(node.getTableType()); + assertEquals(node.getTableType().toValue(), "OFFLINE"); + } + + // ------------------------------------------------------------------------------------------- + // PinotSqlType classification + // ------------------------------------------------------------------------------------------- + + @Test + public void allDdlNodesClassifyAsDdl() { + String[] ddlStatements = { + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE", + "DROP TABLE t", + "DROP TABLE IF EXISTS t TYPE OFFLINE", + "SHOW TABLES", + "SHOW TABLES FROM db" + }; + for (String sql : ddlStatements) { + SqlNodeAndOptions parsed = CalciteSqlParser.compileToSqlNodeAndOptions(sql); + assertEquals(parsed.getSqlType(), PinotSqlType.DDL, + "Expected DDL classification for: " + sql); + } + } + + @Test + public void selectStillClassifiesAsDql() { + SqlNodeAndOptions parsed = CalciteSqlParser.compileToSqlNodeAndOptions("SELECT 1"); + assertEquals(parsed.getSqlType(), PinotSqlType.DQL); + } + + // ------------------------------------------------------------------------------------------- + // Negative cases + // ------------------------------------------------------------------------------------------- + + @Test + public void createTableMissingTableTypeFails() { + expectThrows(SqlCompilationException.class, + () -> CalciteSqlParser.compileToSqlNodeAndOptions("CREATE TABLE t (id INT)")); + } + + @Test + public void createTableEmptyColumnListFails() { + expectThrows(SqlCompilationException.class, + () -> CalciteSqlParser.compileToSqlNodeAndOptions("CREATE TABLE t () TABLE_TYPE = OFFLINE")); + } + + @Test + public void createTableInvalidTableTypeFails() { + expectThrows(SqlCompilationException.class, + () -> CalciteSqlParser.compileToSqlNodeAndOptions( + "CREATE TABLE t (id INT) TABLE_TYPE = HYBRID")); + } + + @Test + public void datetimeWithoutFormatFails() { + expectThrows(SqlCompilationException.class, + () -> CalciteSqlParser.compileToSqlNodeAndOptions( + "CREATE TABLE t (ts LONG DATETIME) TABLE_TYPE = OFFLINE")); + } + + @Test + public void datetimeWithoutGranularityFails() { + expectThrows(SqlCompilationException.class, + () -> CalciteSqlParser.compileToSqlNodeAndOptions( + "CREATE TABLE t (ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH') TABLE_TYPE = OFFLINE")); + } + + @Test + public void propertyWithoutQuotesFails() { + expectThrows(SqlCompilationException.class, + () -> CalciteSqlParser.compileToSqlNodeAndOptions( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES (replication = 3)")); + } + + @Test + public void multipleColumnRolesFails() { + expectThrows(SqlCompilationException.class, + () -> CalciteSqlParser.compileToSqlNodeAndOptions( + "CREATE TABLE t (id INT DIMENSION METRIC) TABLE_TYPE = OFFLINE")); + } + + @Test + public void dimensionArrayParsedAsMultiValue() { + SqlPinotCreateTable stmt = parseCreate( + "CREATE TABLE t (tags STRING DIMENSION ARRAY, id INT DIMENSION) TABLE_TYPE = OFFLINE"); + SqlPinotColumnDeclaration tags = (SqlPinotColumnDeclaration) stmt.getColumns().get(0); + SqlPinotColumnDeclaration id = (SqlPinotColumnDeclaration) stmt.getColumns().get(1); + assertTrue(tags.isMultiValue(), "tags should be multi-value"); + assertFalse(id.isMultiValue(), "id should be single-value"); + assertEquals(tags.getRole(), "DIMENSION"); + } + + @Test + public void ifIsUsableAsIdentifier() { + // IF must be non-reserved so existing user code with column or table names like "if" still + // parses. The grammar uses LOOKAHEAD(3) on `IF NOT EXISTS` to avoid committing to the + // optional branch when IF is just an identifier. + SqlPinotCreateTable asColumn = parseCreate( + "CREATE TABLE t (\"if\" INT) TABLE_TYPE = OFFLINE"); + assertEquals(asColumn.getColumns().size(), 1); + SqlPinotCreateTable asTable = parseCreate( + "CREATE TABLE \"if\" (id INT) TABLE_TYPE = OFFLINE"); + assertEquals(asTable.getName().getSimple(), "if"); + } + + // ------------------------------------------------------------------------------------------- + // Helpers + // ------------------------------------------------------------------------------------------- + + private static SqlPinotCreateTable parseCreate(String sql) { + SqlNode node = parseSingle(sql); + if (!(node instanceof SqlPinotCreateTable)) { + fail("Expected SqlPinotCreateTable; got " + node.getClass().getSimpleName()); + } + return (SqlPinotCreateTable) node; + } + + private static SqlPinotDropTable parseDrop(String sql) { + SqlNode node = parseSingle(sql); + if (!(node instanceof SqlPinotDropTable)) { + fail("Expected SqlPinotDropTable; got " + node.getClass().getSimpleName()); + } + return (SqlPinotDropTable) node; + } + + private static SqlPinotShowTables parseShow(String sql) { + SqlNode node = parseSingle(sql); + if (!(node instanceof SqlPinotShowTables)) { + fail("Expected SqlPinotShowTables; got " + node.getClass().getSimpleName()); + } + return (SqlPinotShowTables) node; + } + + private static SqlPinotShowCreateTable parseShowCreate(String sql) { + SqlNode node = parseSingle(sql); + if (!(node instanceof SqlPinotShowCreateTable)) { + fail("Expected SqlPinotShowCreateTable; got " + node.getClass().getSimpleName()); + } + return (SqlPinotShowCreateTable) node; + } + + private static SqlNode parseSingle(String sql) { + SqlNodeAndOptions parsed = CalciteSqlParser.compileToSqlNodeAndOptions(sql); + return parsed.getSqlNode(); + } +} diff --git a/pinot-controller/pom.xml b/pinot-controller/pom.xml index f89f57c5d674..1ae7adf62238 100644 --- a/pinot-controller/pom.xml +++ b/pinot-controller/pom.xml @@ -56,6 +56,10 @@ org.apache.pinot pinot-timeseries-planner + + org.apache.pinot + pinot-sql-ddl + com.101tec diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java new file mode 100644 index 000000000000..19785928b374 --- /dev/null +++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java @@ -0,0 +1,546 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.controller.api.resources; + +import com.fasterxml.jackson.databind.JsonNode; +import io.swagger.annotations.Api; +import io.swagger.annotations.ApiKeyAuthDefinition; +import io.swagger.annotations.ApiOperation; +import io.swagger.annotations.ApiParam; +import io.swagger.annotations.ApiResponse; +import io.swagger.annotations.ApiResponses; +import io.swagger.annotations.Authorization; +import io.swagger.annotations.SecurityDefinition; +import io.swagger.annotations.SwaggerDefinition; +import java.util.ArrayList; +import java.util.List; +import javax.inject.Inject; +import javax.ws.rs.Consumes; +import javax.ws.rs.DefaultValue; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.QueryParam; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.HttpHeaders; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import org.apache.commons.lang3.StringUtils; +import org.apache.pinot.common.exception.SchemaAlreadyExistsException; +import org.apache.pinot.common.utils.DatabaseUtils; +import org.apache.pinot.controller.api.access.AccessControlFactory; +import org.apache.pinot.controller.api.access.AccessType; +import org.apache.pinot.controller.api.exception.ControllerApplicationException; +import org.apache.pinot.controller.api.exception.TableAlreadyExistsException; +import org.apache.pinot.controller.api.resources.ddl.DdlExecutionRequest; +import org.apache.pinot.controller.api.resources.ddl.DdlExecutionResponse; +import org.apache.pinot.controller.helix.core.PinotHelixResourceManager; +import org.apache.pinot.controller.helix.core.minion.PinotTaskManager; +import org.apache.pinot.controller.util.TaskConfigUtils; +import org.apache.pinot.core.auth.Actions; +import org.apache.pinot.core.auth.ManualAuthorization; +import org.apache.pinot.segment.local.utils.TableConfigUtils; +import org.apache.pinot.spi.config.table.TableConfigValidatorRegistry; +import org.apache.pinot.spi.config.table.TableType; +import org.apache.pinot.spi.data.Schema; +import org.apache.pinot.spi.utils.CommonConstants; +import org.apache.pinot.spi.utils.JsonUtils; +import org.apache.pinot.spi.utils.builder.TableNameBuilder; +import org.apache.pinot.sql.ddl.compile.CompiledCreateTable; +import org.apache.pinot.sql.ddl.compile.CompiledDdl; +import org.apache.pinot.sql.ddl.compile.CompiledDropTable; +import org.apache.pinot.sql.ddl.compile.CompiledShowCreateTable; +import org.apache.pinot.sql.ddl.compile.DdlCompilationException; +import org.apache.pinot.sql.ddl.compile.DdlCompiler; +import org.apache.pinot.sql.ddl.compile.DdlOperation; +import org.apache.pinot.sql.ddl.reverse.CanonicalDdlEmitter; +import org.glassfish.grizzly.http.server.Request; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import static org.apache.pinot.spi.utils.CommonConstants.DATABASE; +import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY; + + +/** + * Controller endpoint for executing Pinot SQL DDL statements. Currently supports CREATE TABLE, + * DROP TABLE, SHOW TABLES, and SHOW CREATE TABLE. + * + *

Pipeline: + *

    + *
  1. {@link DdlCompiler} parses + compiles the SQL into a {@link CompiledDdl}.
    2. + *
  2. Database/table names are translated through {@link DatabaseUtils#translateTableName} + * so the {@code Database} HTTP header is honoured uniformly.
    3. + *
  3. Authorization is invoked based on the operation type.
    4. + *
  4. Execution either persists via {@link PinotHelixResourceManager} or, when {@code dryRun} + * is true, returns the compiled artifacts without mutating cluster state.
    5. + *
+ * + *

The endpoint is intentionally a single POST that dispatches by operation. This keeps the + * client surface area small and matches the canonical {@code POST /sql/ddl} contract from the + * design. + */ +@Api(tags = "SQL DDL", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY), + @Authorization(value = DATABASE)}) +@SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = { + @ApiKeyAuthDefinition(name = HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, + key = SWAGGER_AUTHORIZATION_KEY, + description = "The format of the key is ```\"Basic \" or \"Bearer \"```"), + @ApiKeyAuthDefinition(name = DATABASE, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = DATABASE, + description = "Database context passed through http header. If no context is provided 'default' database " + + "context will be considered.")})) +@Path("/") +public class PinotDdlRestletResource { + private static final Logger LOGGER = LoggerFactory.getLogger(PinotDdlRestletResource.class); + + @Inject + PinotHelixResourceManager _pinotHelixResourceManager; + + @Inject + PinotTaskManager _pinotTaskManager; + + @Inject + AccessControlFactory _accessControlFactory; + + @POST + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + @Path("/sql/ddl") + @ManualAuthorization // permission check is done after parsing the DDL so we know the operation + @ApiOperation(value = "Execute a Pinot SQL DDL statement", + notes = "Supports CREATE TABLE, DROP TABLE, SHOW TABLES, and SHOW CREATE TABLE. Returns the " + + "generated Schema/TableConfig (CREATE), operation outcome (DROP/SHOW TABLES), or " + + "canonical DDL string (SHOW CREATE TABLE).") + @ApiResponses(value = { + @ApiResponse(code = 200, message = "Success"), + @ApiResponse(code = 400, message = "Bad request (parse/semantic error)"), + @ApiResponse(code = 409, message = "Table already exists"), + @ApiResponse(code = 500, message = "Internal server error") + }) + public DdlExecutionResponse executeDdl( + @ApiParam(value = "DDL request body with 'sql' field", required = true) + DdlExecutionRequest request, + @ApiParam(value = "When true, compile and validate but do not persist.") + @QueryParam("dryRun") @DefaultValue("false") boolean dryRun, + @Context HttpHeaders httpHeaders, @Context Request httpRequest) { + if (request == null || StringUtils.isBlank(request.getSql())) { + throw badRequest("Request body must include a non-empty 'sql' field."); + } + + CompiledDdl compiled; + try { + compiled = DdlCompiler.compile(request.getSql()); + } catch (DdlCompilationException e) { + throw badRequest(e.getMessage()); + } catch (RuntimeException e) { + LOGGER.warn("Unexpected DDL compilation failure", e); + throw badRequest("DDL compilation failed: " + e.getMessage()); + } + + DdlOperation op = compiled.getOperation(); + String requestedDatabase = compiled.getDatabaseName(); + String effectiveDatabase = resolveDatabase(requestedDatabase, httpHeaders); + + // Authorization is performed inside each execute*() method, AFTER the target table name has + // been DB-translated. Authorizing on the pre-translation name would let a header-supplied + // database substitute past the auth check. + switch (op) { + case CREATE_TABLE: + return executeCreate((CompiledCreateTable) compiled, effectiveDatabase, dryRun, + httpHeaders, httpRequest); + case DROP_TABLE: + return executeDrop((CompiledDropTable) compiled, effectiveDatabase, dryRun, + httpHeaders, httpRequest); + case SHOW_TABLES: + return executeShow(effectiveDatabase, httpHeaders, httpRequest); + case SHOW_CREATE_TABLE: + return executeShowCreate((CompiledShowCreateTable) compiled, effectiveDatabase, + httpHeaders, httpRequest); + default: + throw new ControllerApplicationException(LOGGER, "Unhandled DDL operation: " + op, + Response.Status.INTERNAL_SERVER_ERROR); + } + } + + // ------------------------------------------------------------------------------------------- + // CREATE + // ------------------------------------------------------------------------------------------- + + private DdlExecutionResponse executeCreate(CompiledCreateTable create, String database, + boolean dryRun, HttpHeaders headers, Request httpRequest) { + // The compiled TableConfig.tableName carries the SQL `db.tbl` qualifier when one was given; + // translateTableName then reconciles it against the Database header (and rejects conflicts). + String tableNameWithType = DatabaseUtils.translateTableName( + TableNameBuilder.forType(create.getTableConfig().getTableType()) + .tableNameWithType(create.getTableConfig().getTableName()), headers); + create.getTableConfig().setTableName(tableNameWithType); + + // The schema name has the same DB-scoping requirement: PinotHelixResourceManager.addTable + // looks up the schema using the raw table name extracted from tableNameWithType, so the two + // must agree on the database prefix. + String compiledSchemaName = create.getSchema().getSchemaName(); + String dottedSchemaName = create.getDatabaseName() == null + ? compiledSchemaName + : create.getDatabaseName() + "." + compiledSchemaName; + String schemaName = DatabaseUtils.translateTableName(dottedSchemaName, headers); + create.getSchema().setSchemaName(schemaName); + + // Authorize against the FULLY-QUALIFIED, post-translation table name. Checking the bare + // compiled name would let a Database header substitute the resource we're checking against, + // enabling cross-DB privilege escalation. + ResourceUtils.checkPermissionAndAccess(tableNameWithType, httpRequest, headers, + AccessType.CREATE, Actions.Table.CREATE_TABLE, _accessControlFactory, LOGGER); + + DdlExecutionResponse response = new DdlExecutionResponse() + .setOperation(DdlOperation.CREATE_TABLE) + .setDryRun(dryRun) + .setDatabaseName(database) + .setTableName(tableNameWithType) + .setTableType(create.getTableConfig().getTableType().toString()) + .setIfNotExists(create.isIfNotExists()) + .setSchema(toJson(create.getSchema())) + .setTableConfig(toJson(create.getTableConfig())) + .setWarnings(create.getWarnings()); + + // Run the full schema/table validation stack that the existing /tables and /tableConfigs APIs + // apply before any ZK write. This catches invalid combinations (upsert without primary keys, + // field configs referencing non-existent columns, task configs with bad column references, + // etc.) that the compiler alone cannot detect. Runs for both dry-run and live create. + validateTableConfig(create.getSchema(), create.getTableConfig()); + + if (dryRun) { + response.setMessage("Dry run: validated CREATE TABLE without persisting."); + return response; + } + + if (_pinotHelixResourceManager.hasTable(tableNameWithType)) { + if (create.isIfNotExists()) { + response.setMessage("Table " + tableNameWithType + " already exists; CREATE IF NOT EXISTS is a no-op."); + return response; + } + throw new ControllerApplicationException(LOGGER, + "Table " + tableNameWithType + " already exists.", Response.Status.CONFLICT); + } + + // When a schema for this raw table name already exists (the common case when adding the + // second physical variant of a hybrid pair), verify the DDL column list is equivalent to + // the stored schema. Silently accepting a mismatched column list would create a table whose + // runtime schema differs from what the DDL described. + Schema storedSchema = _pinotHelixResourceManager.getSchema(schemaName); + boolean schemaPreexisted = storedSchema != null; + if (schemaPreexisted) { + try { + String storedJson = JsonUtils.objectToString(storedSchema); + String compiledJson = JsonUtils.objectToString(create.getSchema()); + if (!storedJson.equals(compiledJson)) { + throw new ControllerApplicationException(LOGGER, + "Schema '" + schemaName + "' already exists and does not match the column list in the DDL. " + + "Either omit the column list to reuse the existing schema, or drop and recreate the table pair " + + "if the schema has genuinely changed.", + Response.Status.CONFLICT); + } + } catch (ControllerApplicationException e) { + throw e; + } catch (Exception e) { + throw new ControllerApplicationException(LOGGER, + "Failed to compare schemas for '" + schemaName + "': " + e.getMessage(), + Response.Status.INTERNAL_SERVER_ERROR, e); + } + } + + boolean schemaCreatedHere = false; + try { + // override=false: an existing schema with the same name is a precondition violation, not + // something we silently overwrite. The other-typed table variant or another caller's + // schema would otherwise be clobbered out from under them. + if (!schemaPreexisted) { + _pinotHelixResourceManager.addSchema(create.getSchema(), false, false); + schemaCreatedHere = true; + } + _pinotHelixResourceManager.addTable(create.getTableConfig()); + response.setMessage("Successfully created table " + tableNameWithType); + LOGGER.info("DDL created table {}", tableNameWithType); + return response; + } catch (TableAlreadyExistsException e) { + // Race: another caller added the table between our hasTable check and addTable. Map to + // 409 Conflict so the failure mode is consistent with the pre-check path. + rollbackSchemaIfCreated(schemaName, schemaCreatedHere); + throw new ControllerApplicationException(LOGGER, e.getMessage(), Response.Status.CONFLICT, e); + } catch (SchemaAlreadyExistsException e) { + // The override=false addSchema call lost a race with another schema writer. Surface 409. + throw new ControllerApplicationException(LOGGER, e.getMessage(), Response.Status.CONFLICT, e); + } catch (Exception e) { + // Roll back the schema we created in this call so we don't leak partial state. We only + // touch the schema when this call created it; a pre-existing schema is left alone. + rollbackSchemaIfCreated(schemaName, schemaCreatedHere); + // ControllerApplicationException(LOGGER, ...) logs the exception, so don't double-log here. + throw new ControllerApplicationException(LOGGER, + "Failed to create table " + tableNameWithType + ": " + e.getMessage(), + Response.Status.INTERNAL_SERVER_ERROR, e); + } + } + + private void rollbackSchemaIfCreated(String schemaName, boolean schemaCreatedHere) { + if (!schemaCreatedHere) { + return; + } + try { + _pinotHelixResourceManager.deleteSchema(schemaName); + LOGGER.info("Rolled back schema {} after failed table create", schemaName); + } catch (Exception rollbackEx) { + LOGGER.error("Failed to roll back schema {} after failed table create; manual cleanup " + + "may be required", schemaName, rollbackEx); + } + } + + /** + * Runs the same schema/table validation stack that {@code /tables} and {@code /tableConfigs} + * apply before any ZK write, so DDL-created configs are subject to the same rules as + * JSON-API-created configs (upsert/dedup primary-key requirements, field config column + * references, task config validation, registry-level semantic checks, etc.). + */ + private void validateTableConfig(Schema schema, + org.apache.pinot.spi.config.table.TableConfig tableConfig) { + try { + TableConfigUtils.validateTableName(tableConfig); + TableConfigUtils.validate(tableConfig, schema, null); + _pinotHelixResourceManager.validateTableTenantConfig(tableConfig); + _pinotHelixResourceManager.validateTableTaskMinionInstanceTagConfig(tableConfig); + TaskConfigUtils.validateTaskConfigs(tableConfig, schema, _pinotTaskManager, null); + TableConfigValidatorRegistry.validate(tableConfig, schema); + } catch (ControllerApplicationException e) { + throw e; + } catch (Exception e) { + throw new ControllerApplicationException(LOGGER, + "Table config validation failed: " + e.getMessage(), Response.Status.BAD_REQUEST, e); + } + } + + // ------------------------------------------------------------------------------------------- + // DROP + // ------------------------------------------------------------------------------------------- + + private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database, boolean dryRun, + HttpHeaders headers, Request httpRequest) { + // The SQL `db.tbl` qualifier is preserved through to translateTableName so the resolved + // raw name carries the right database scope. Without this, `DROP TABLE analyticsDb.events` + // with no Database header would silently target `events` in the default DB. + String dottedRaw = drop.getDatabaseName() == null + ? drop.getRawTableName() + : drop.getDatabaseName() + "." + drop.getRawTableName(); + String fullyQualifiedRaw = DatabaseUtils.translateTableName(dottedRaw, headers); + + // Compute the candidate typed names BEFORE existence filtering so we can authorize against + // the user's intent (not just whatever happens to exist now). This prevents an unauthorized + // caller from probing existence via 200/404 vs 403. + List candidates = new ArrayList<>(2); + if (drop.getTableType() == null || drop.getTableType() == TableType.OFFLINE) { + candidates.add(TableNameBuilder.OFFLINE.tableNameWithType(fullyQualifiedRaw)); + } + if (drop.getTableType() == null || drop.getTableType() == TableType.REALTIME) { + candidates.add(TableNameBuilder.REALTIME.tableNameWithType(fullyQualifiedRaw)); + } + for (String candidate : candidates) { + ResourceUtils.checkPermissionAndAccess(candidate, httpRequest, headers, + AccessType.DELETE, Actions.Table.DELETE_TABLE, _accessControlFactory, LOGGER); + } + + List targets = new ArrayList<>(2); + for (String candidate : candidates) { + if (_pinotHelixResourceManager.hasTable(candidate)) { + targets.add(candidate); + } + } + + if (targets.isEmpty()) { + if (drop.isIfExists()) { + return new DdlExecutionResponse() + .setOperation(DdlOperation.DROP_TABLE) + .setDryRun(dryRun) + .setDatabaseName(database) + .setTableName(fullyQualifiedRaw) + .setIfExists(true) + .setDeletedTables(targets) + .setMessage("No matching table to drop; IF EXISTS satisfied."); + } + throw new ControllerApplicationException(LOGGER, + "Table not found: " + fullyQualifiedRaw, Response.Status.NOT_FOUND); + } + + DdlExecutionResponse response = new DdlExecutionResponse() + .setOperation(DdlOperation.DROP_TABLE) + .setDryRun(dryRun) + .setDatabaseName(database) + .setTableName(fullyQualifiedRaw) + .setIfExists(drop.isIfExists()) + .setTableType(drop.getTableType() == null ? null : drop.getTableType().toString()) + .setDeletedTables(targets); + + if (dryRun) { + response.setMessage("Dry run: " + targets.size() + " table(s) would be dropped."); + return response; + } + + try { + for (String target : targets) { + TableType type = TableNameBuilder.getTableTypeFromTableName(target); + _pinotHelixResourceManager.deleteTable(fullyQualifiedRaw, type, null); + } + response.setMessage("Dropped " + targets.size() + " table(s)."); + LOGGER.info("DDL dropped tables {}", targets); + return response; + } catch (Exception e) { + // ControllerApplicationException(LOGGER, ...) logs the exception itself; don't double-log. + throw new ControllerApplicationException(LOGGER, + "Failed to drop table " + fullyQualifiedRaw + ": " + e.getMessage(), + Response.Status.INTERNAL_SERVER_ERROR, e); + } + } + + // ------------------------------------------------------------------------------------------- + // SHOW + // ------------------------------------------------------------------------------------------- + + // ------------------------------------------------------------------------------------------- + // SHOW CREATE TABLE + // ------------------------------------------------------------------------------------------- + + private DdlExecutionResponse executeShowCreate(CompiledShowCreateTable show, String database, + HttpHeaders headers, Request httpRequest) { + String dottedRaw = show.getDatabaseName() == null + ? show.getRawTableName() + : show.getDatabaseName() + "." + show.getRawTableName(); + String fullyQualifiedRaw = DatabaseUtils.translateTableName(dottedRaw, headers); + + // Resolve which typed variant to render. Explicit TYPE clause wins; otherwise check both. + // Authorize BEFORE existence checks so an unauthorized caller cannot distinguish + // "exists but forbidden" (403) from "not found" (404) — mirroring the DROP path. + TableType resolvedType = show.getTableType(); + String tableNameWithType; + if (resolvedType != null) { + tableNameWithType = TableNameBuilder.forType(resolvedType).tableNameWithType(fullyQualifiedRaw); + ResourceUtils.checkPermissionAndAccess(tableNameWithType, httpRequest, headers, + AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER); + if (!_pinotHelixResourceManager.hasTable(tableNameWithType)) { + throw new ControllerApplicationException(LOGGER, + "Table not found: " + tableNameWithType, Response.Status.NOT_FOUND); + } + } else { + // Authorize BOTH candidates before probing existence so the caller cannot infer which + // variant exists from a 403 vs 404 response, and so an auth failure on the realtime + // variant is not masked by a 404 produced before the check runs. + String off = TableNameBuilder.OFFLINE.tableNameWithType(fullyQualifiedRaw); + String rt = TableNameBuilder.REALTIME.tableNameWithType(fullyQualifiedRaw); + ResourceUtils.checkPermissionAndAccess(off, httpRequest, headers, + AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER); + ResourceUtils.checkPermissionAndAccess(rt, httpRequest, headers, + AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER); + if (_pinotHelixResourceManager.hasTable(off)) { + tableNameWithType = off; + resolvedType = TableType.OFFLINE; + } else if (_pinotHelixResourceManager.hasTable(rt)) { + tableNameWithType = rt; + resolvedType = TableType.REALTIME; + } else { + throw new ControllerApplicationException(LOGGER, + "Table not found: " + fullyQualifiedRaw, Response.Status.NOT_FOUND); + } + } + + org.apache.pinot.spi.config.table.TableConfig tableConfig = + _pinotHelixResourceManager.getTableConfig(tableNameWithType); + if (tableConfig == null) { + // Should not happen — hasTable just succeeded — but treat as not-found rather than 500. + throw new ControllerApplicationException(LOGGER, + "Table " + tableNameWithType + " disappeared during SHOW CREATE.", + Response.Status.NOT_FOUND); + } + org.apache.pinot.spi.data.Schema schema = + _pinotHelixResourceManager.getTableSchema(tableNameWithType); + if (schema == null) { + throw new ControllerApplicationException(LOGGER, + "Schema not found for " + tableNameWithType + + "; SHOW CREATE TABLE requires both schema and config to exist.", + Response.Status.INTERNAL_SERVER_ERROR); + } + + // Use the resolved database (which incorporates the Database header) so the emitted DDL + // carries the correct qualifier even when the caller omits the db. prefix in the SQL. + String ddl = CanonicalDdlEmitter.emit(schema, tableConfig, database); + + return new DdlExecutionResponse() + .setOperation(DdlOperation.SHOW_CREATE_TABLE) + .setDatabaseName(database) + .setTableName(tableNameWithType) + .setTableType(resolvedType.toString()) + .setDdl(ddl) + .setMessage("Rendered canonical CREATE TABLE for " + tableNameWithType + "."); + } + + private DdlExecutionResponse executeShow(String database, HttpHeaders headers, Request httpRequest) { + // SHOW TABLES is scoped to a single database to prevent silently leaking table names across + // databases the caller may not have access to. The database resolution chain (SQL FROM + // clause -> Database header -> DEFAULT_DATABASE) ensures we always have an explicit scope. + String scopedDatabase = database == null ? CommonConstants.DEFAULT_DATABASE : database; + // Use the cluster-scoped GetTable action that the existing GET /tables endpoint uses. We + // pass the database name as the resource handle so AccessControl impls that key on + // database scope can still authorise per-database, but the action itself is cluster-level + // (matching the listing semantics) rather than table-level. + ResourceUtils.checkPermissionAndAccess(scopedDatabase, httpRequest, headers, + AccessType.READ, Actions.Cluster.GET_TABLE, _accessControlFactory, LOGGER); + List tables = _pinotHelixResourceManager.getAllRawTables(scopedDatabase); + return new DdlExecutionResponse() + .setOperation(DdlOperation.SHOW_TABLES) + .setDatabaseName(scopedDatabase) + .setTableNames(tables) + .setMessage("Found " + tables.size() + " table(s)."); + } + + // ------------------------------------------------------------------------------------------- + // Helpers + // ------------------------------------------------------------------------------------------- + + /** + * Returns the database name to use for this request. Precedence: explicit {@code db.name} in + * the SQL statement → {@code Database} HTTP header → {@code null} (default database). + */ + private static String resolveDatabase(String fromSql, HttpHeaders headers) { + if (fromSql != null) { + return fromSql; + } + if (headers == null) { + return null; + } + return headers.getHeaderString(DATABASE); + } + + private static JsonNode toJson(Object obj) { + try { + return JsonUtils.objectToJsonNode(obj); + } catch (Exception e) { + throw new ControllerApplicationException(LOGGER, + "Failed to serialize compiled DDL artifact: " + e.getMessage(), + Response.Status.INTERNAL_SERVER_ERROR, e); + } + } + + private static ControllerApplicationException badRequest(String message) { + return new ControllerApplicationException(LOGGER, message, Response.Status.BAD_REQUEST); + } +} diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionRequest.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionRequest.java new file mode 100644 index 000000000000..e42c17631287 --- /dev/null +++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionRequest.java @@ -0,0 +1,37 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.controller.api.resources.ddl; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonProperty; + + +/** Request body for {@code POST /sql/ddl}. */ +public class DdlExecutionRequest { + private final String _sql; + + @JsonCreator + public DdlExecutionRequest(@JsonProperty("sql") String sql) { + _sql = sql; + } + + public String getSql() { + return _sql; + } +} diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java new file mode 100644 index 000000000000..5fb97a8aac1b --- /dev/null +++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java @@ -0,0 +1,196 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.controller.api.resources.ddl; + +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.databind.JsonNode; +import java.util.List; +import javax.annotation.Nullable; +import org.apache.pinot.sql.ddl.compile.DdlOperation; + + +/** + * Response body for {@code POST /sql/ddl}. + * + *

The shape of the response varies by operation. {@link JsonInclude.Include#NON_NULL} keeps + * the wire payload focused on the fields that actually apply to the operation that ran. + * + *

    + *
  • CREATE_TABLE: {@code tableName, tableType, schema, tableConfig, ifNotExists, warnings}
    • + *
  • DROP_TABLE: {@code tableName, tableType, deletedTables, ifExists}
    • + *
  • SHOW_TABLES: {@code tableNames}
    • + *
  • SHOW_CREATE_TABLE: {@code tableName, tableType, ddl}
    • + *
+ */ +@JsonInclude(JsonInclude.Include.NON_NULL) +public class DdlExecutionResponse { + private DdlOperation _operation; + private boolean _dryRun; + private String _databaseName; + private String _tableName; + private String _tableType; + private Boolean _ifNotExists; + private Boolean _ifExists; + private JsonNode _schema; + private JsonNode _tableConfig; + private List _warnings; + private List _deletedTables; + private List _tableNames; + private String _ddl; + private String _message; + + public DdlOperation getOperation() { + return _operation; + } + + public DdlExecutionResponse setOperation(DdlOperation operation) { + _operation = operation; + return this; + } + + public boolean isDryRun() { + return _dryRun; + } + + public DdlExecutionResponse setDryRun(boolean dryRun) { + _dryRun = dryRun; + return this; + } + + @Nullable + public String getDatabaseName() { + return _databaseName; + } + + public DdlExecutionResponse setDatabaseName(@Nullable String databaseName) { + _databaseName = databaseName; + return this; + } + + @Nullable + public String getTableName() { + return _tableName; + } + + public DdlExecutionResponse setTableName(@Nullable String tableName) { + _tableName = tableName; + return this; + } + + @Nullable + public String getTableType() { + return _tableType; + } + + public DdlExecutionResponse setTableType(@Nullable String tableType) { + _tableType = tableType; + return this; + } + + @Nullable + public Boolean getIfNotExists() { + return _ifNotExists; + } + + public DdlExecutionResponse setIfNotExists(@Nullable Boolean ifNotExists) { + _ifNotExists = ifNotExists; + return this; + } + + @Nullable + public Boolean getIfExists() { + return _ifExists; + } + + public DdlExecutionResponse setIfExists(@Nullable Boolean ifExists) { + _ifExists = ifExists; + return this; + } + + @Nullable + public JsonNode getSchema() { + return _schema; + } + + public DdlExecutionResponse setSchema(@Nullable JsonNode schema) { + _schema = schema; + return this; + } + + @Nullable + public JsonNode getTableConfig() { + return _tableConfig; + } + + public DdlExecutionResponse setTableConfig(@Nullable JsonNode tableConfig) { + _tableConfig = tableConfig; + return this; + } + + @Nullable + public List getWarnings() { + return _warnings; + } + + public DdlExecutionResponse setWarnings(@Nullable List warnings) { + _warnings = warnings == null || warnings.isEmpty() ? null : warnings; + return this; + } + + @Nullable + public List getDeletedTables() { + return _deletedTables; + } + + public DdlExecutionResponse setDeletedTables(@Nullable List deletedTables) { + _deletedTables = deletedTables; + return this; + } + + @Nullable + public List getTableNames() { + return _tableNames; + } + + public DdlExecutionResponse setTableNames(@Nullable List tableNames) { + _tableNames = tableNames; + return this; + } + + @Nullable + public String getDdl() { + return _ddl; + } + + /** Canonical CREATE TABLE statement returned by {@code SHOW CREATE TABLE}. */ + public DdlExecutionResponse setDdl(@Nullable String ddl) { + _ddl = ddl; + return this; + } + + @Nullable + public String getMessage() { + return _message; + } + + public DdlExecutionResponse setMessage(@Nullable String message) { + _message = message; + return this; + } +} diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java new file mode 100644 index 000000000000..dda400016b03 --- /dev/null +++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java @@ -0,0 +1,337 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.controller.api; + +import com.fasterxml.jackson.databind.JsonNode; +import java.io.IOException; +import java.util.Collections; +import java.util.Map; +import org.apache.pinot.controller.helix.ControllerTest; +import org.apache.pinot.spi.utils.JsonUtils; +import org.testng.annotations.AfterClass; +import org.testng.annotations.BeforeClass; +import org.testng.annotations.Test; + +import static org.testng.Assert.assertEquals; +import static org.testng.Assert.assertFalse; +import static org.testng.Assert.assertNotNull; +import static org.testng.Assert.assertTrue; +import static org.testng.Assert.fail; + + +/** + * End-to-end integration tests for {@code POST /sql/ddl}. + * + *

Each test uses a unique table name prefix so failures do not cascade through the shared + * controller test instance. + */ +public class PinotDdlRestletResourceTest extends ControllerTest { + + private static final String TBL_BASIC = "ddlBasicOffline"; + private static final String TBL_DRY_RUN = "ddlDryRunOffline"; + private static final String TBL_IF_NOT_EXISTS = "ddlIfNotExistsOffline"; + private static final String TBL_DROP = "ddlDropOffline"; + + @BeforeClass + public void setUp() + throws Exception { + DEFAULT_INSTANCE.setupSharedStateAndValidate(); + } + + @AfterClass + public void tearDown() { + DEFAULT_INSTANCE.cleanup(); + } + + // ------------------------------------------------------------------------------------------- + // CREATE TABLE + // ------------------------------------------------------------------------------------------- + + @Test + public void createOfflineTableRoundTrip() + throws IOException { + String sql = "CREATE TABLE " + TBL_BASIC + " (" + + " id INT NOT NULL DIMENSION," + + " ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'" + + ") TABLE_TYPE = OFFLINE PROPERTIES (" + + " 'timeColumnName' = 'ts'," + + " 'replication' = '1'" + + ")"; + + JsonNode response = postDdl(sql, false); + assertEquals(response.get("operation").asText(), "CREATE_TABLE"); + assertEquals(response.get("tableName").asText(), TBL_BASIC + "_OFFLINE"); + assertEquals(response.get("tableType").asText(), "OFFLINE"); + assertNotNull(response.get("schema")); + assertNotNull(response.get("tableConfig")); + assertFalse(response.get("dryRun").asBoolean()); + + // Verify it actually persisted: the table should now show up in the existing tables list. + String listResponse = sendGetRequest(DEFAULT_INSTANCE.getControllerBaseApiUrl() + "/tables"); + assertTrue(listResponse.contains(TBL_BASIC), + "Created table should appear in /tables; got " + listResponse); + } + + @Test + public void dryRunDoesNotPersist() + throws IOException { + String sql = "CREATE TABLE " + TBL_DRY_RUN + " (id INT) TABLE_TYPE = OFFLINE"; + JsonNode response = postDdl(sql, true); + assertTrue(response.get("dryRun").asBoolean()); + assertNotNull(response.get("schema")); + assertNotNull(response.get("tableConfig")); + + // Verify nothing was persisted: the table should NOT be in the listing. + String listResponse = sendGetRequest(DEFAULT_INSTANCE.getControllerBaseApiUrl() + "/tables"); + assertFalse(listResponse.contains(TBL_DRY_RUN), + "Dry-run table must not be persisted; got " + listResponse); + } + + @Test + public void createIfNotExistsIsIdempotent() + throws IOException { + String sql = "CREATE TABLE IF NOT EXISTS " + TBL_IF_NOT_EXISTS + + " (id INT) TABLE_TYPE = OFFLINE"; + + // First call: creates. + postDdl(sql, false); + // Second call: succeeds without error. + JsonNode second = postDdl(sql, false); + assertEquals(second.get("operation").asText(), "CREATE_TABLE"); + assertTrue(second.get("message").asText().toLowerCase().contains("exist"), + "Expected idempotent message, got: " + second.get("message").asText()); + } + + @Test + public void createWithoutIfNotExistsConflicts() + throws IOException { + String sql = "CREATE TABLE conflictTable (id INT) TABLE_TYPE = OFFLINE"; + postDdl(sql, false); // first time — succeeds + int status = postDdlExpectFailure(sql); + assertEquals(status, 409, "Expected 409 Conflict on duplicate create"); + } + + // ------------------------------------------------------------------------------------------- + // DROP TABLE + // ------------------------------------------------------------------------------------------- + + @Test + public void dropTableSucceeds() + throws IOException { + postDdl("CREATE TABLE " + TBL_DROP + " (id INT) TABLE_TYPE = OFFLINE", false); + + JsonNode response = postDdl("DROP TABLE " + TBL_DROP, false); + assertEquals(response.get("operation").asText(), "DROP_TABLE"); + JsonNode deleted = response.get("deletedTables"); + assertNotNull(deleted); + assertTrue(deleted.toString().contains(TBL_DROP), + "Expected " + TBL_DROP + " in deletedTables; got " + deleted); + } + + @Test + public void dropMissingTableWithoutIfExistsReturns404() + throws IOException { + int status = postDdlExpectFailure("DROP TABLE noSuchTableEverCreated"); + assertEquals(status, 404); + } + + @Test + public void dropMissingTableWithIfExistsSucceeds() + throws IOException { + JsonNode response = postDdl("DROP TABLE IF EXISTS noSuchTableEverCreated2", false); + assertEquals(response.get("operation").asText(), "DROP_TABLE"); + assertTrue(response.get("ifExists").asBoolean()); + } + + // ------------------------------------------------------------------------------------------- + // SHOW TABLES + // ------------------------------------------------------------------------------------------- + + @Test + public void showTablesListsExistingTables() + throws IOException { + // Make sure at least one DDL-created table exists for this test. + String tbl = "ddlShowList"; + postDdl("CREATE TABLE " + tbl + " (id INT) TABLE_TYPE = OFFLINE", false); + + JsonNode response = postDdl("SHOW TABLES", false); + assertEquals(response.get("operation").asText(), "SHOW_TABLES"); + assertNotNull(response.get("tableNames")); + assertTrue(response.get("tableNames").toString().contains(tbl), + "Expected " + tbl + " in tableNames; got " + response.get("tableNames")); + } + + @Test + public void showCreateRendersCanonicalDdlAndRoundTrips() + throws IOException { + String tbl = "ddlShowCreateRoundtrip"; + String createSql = "CREATE TABLE " + tbl + " (" + + " id INT NOT NULL DIMENSION," + + " ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'" + + ") TABLE_TYPE = OFFLINE PROPERTIES (" + + " 'timeColumnName' = 'ts'," + + " 'replication' = '2'," + + " 'brokerTenant' = 'DefaultTenant'" + + ")"; + postDdl(createSql, false); + + JsonNode response = postDdl("SHOW CREATE TABLE " + tbl, false); + assertEquals(response.get("operation").asText(), "SHOW_CREATE_TABLE"); + assertEquals(response.get("tableName").asText(), tbl + "_OFFLINE"); + String ddl = response.get("ddl").asText(); + // Canonical clause order: column block, TABLE_TYPE, PROPERTIES. + assertTrue(ddl.startsWith("CREATE TABLE " + tbl + " ("), ddl); + assertTrue(ddl.contains("TABLE_TYPE = OFFLINE"), ddl); + assertTrue(ddl.contains("'replication' = '2'"), ddl); + assertTrue(ddl.contains("'timeColumnName' = 'ts'"), ddl); + // Properties are emitted lexicographically: brokerTenant < replication < timeColumnName. + int brokerIdx = ddl.indexOf("'brokerTenant'"); + int replicationIdx = ddl.indexOf("'replication'"); + int timeColIdx = ddl.indexOf("'timeColumnName'"); + assertTrue(brokerIdx < replicationIdx && replicationIdx < timeColIdx, + "Properties not in lex order:\n" + ddl); + } + + @Test + public void showCreateOnMissingTableReturns404() + throws IOException { + int status = postDdlExpectFailure("SHOW CREATE TABLE noSuchTableEverDeclared"); + assertEquals(status, 404); + } + + @Test + public void showCreateWithDatabaseHeaderEmitsDatabaseQualifiedDdl() + throws IOException { + // Regression: executeShowCreate() was passing show.getDatabaseName() (always null when the + // SQL uses a bare table name) into CanonicalDdlEmitter instead of the resolved `database` + // that incorporates the Database: header. The emitted DDL would therefore carry no qualifier, + // making the statement non-idempotent when replayed without the same header. + String tbl = "ddlShowCreateWithHeader"; + postDdl("CREATE TABLE " + tbl + " (id INT) TABLE_TYPE = OFFLINE", false); + + String url = DEFAULT_INSTANCE.getControllerBaseApiUrl() + "/sql/ddl"; + String body = "{\"sql\": \"SHOW CREATE TABLE " + tbl + "\"}"; + java.util.Map hdrs = new java.util.LinkedHashMap<>(); + hdrs.put("Content-Type", "application/json"); + // Pass database via header with no SQL db. qualifier — the resolved database must appear in + // the emitted DDL so replaying without the header still targets the correct tenant. + hdrs.put("database", "default"); + String raw = sendPostRequest(url, body, hdrs); + JsonNode response = JsonUtils.stringToJsonNode(raw); + assertEquals(response.get("operation").asText(), "SHOW_CREATE_TABLE"); + String ddl = response.get("ddl").asText(); + assertTrue(ddl.startsWith("CREATE TABLE default." + tbl), + "Expected DDL to carry db-qualified name; got:\n" + ddl); + } + + @Test + public void databaseQualifiedDropTargetsCorrectTable() + throws IOException { + // Regression: DROP TABLE was previously discarding the SQL `db.` qualifier and silently + // targeting the default-database table of the same bare name. With no Database header, the + // qualified DROP must report "table not found in .", not a stale 200 against + // some unrelated default-DB table. + String bareName = "ddlDbQualifiedDropTarget"; + postDdl("CREATE TABLE " + bareName + " (id INT) TABLE_TYPE = OFFLINE", false); + + // Drop with a fake DB qualifier that does not match the table's actual (default) database + // should NOT delete the default-DB copy. Without IF EXISTS, expect 404. + int status = postDdlExpectFailure("DROP TABLE noSuchDb." + bareName); + assertEquals(status, 404); + + // The default-DB table should still be there. + String listResponse = sendGetRequest(DEFAULT_INSTANCE.getControllerBaseApiUrl() + "/tables"); + assertTrue(listResponse.contains(bareName), + "Default-DB table must not be silently dropped by a qualified DROP; got " + listResponse); + } + + // ------------------------------------------------------------------------------------------- + // Negative parse / compile cases + // ------------------------------------------------------------------------------------------- + + @Test + public void emptyBodyReturnsBadRequest() + throws IOException { + int status = postRawExpectFailure("{}"); + assertEquals(status, 400); + } + + @Test + public void parseErrorReturnsBadRequest() + throws IOException { + int status = postDdlExpectFailure("CREATE NOT_A_THING (id INT) TABLE_TYPE = OFFLINE"); + assertEquals(status, 400); + } + + @Test + public void semanticErrorReturnsBadRequest() + throws IOException { + int status = postDdlExpectFailure( + "CREATE TABLE foo (id INT) TABLE_TYPE = OFFLINE PROPERTIES ('timeColumnName' = 'missing')"); + assertEquals(status, 400); + } + + // ------------------------------------------------------------------------------------------- + // Helpers + // ------------------------------------------------------------------------------------------- + + private static JsonNode postDdl(String sql, boolean dryRun) + throws IOException { + String url = DEFAULT_INSTANCE.getControllerBaseApiUrl() + "/sql/ddl?dryRun=" + dryRun; + String body = "{\"sql\": " + JsonUtils.objectToString(sql) + "}"; + String response = sendPostRequest(url, body, + Collections.singletonMap("Content-Type", "application/json")); + return JsonUtils.stringToJsonNode(response); + } + + private static int postDdlExpectFailure(String sql) { + String url = DEFAULT_INSTANCE.getControllerBaseApiUrl() + "/sql/ddl"; + String body; + try { + body = "{\"sql\": " + JsonUtils.objectToString(sql) + "}"; + } catch (IOException ioe) { + throw new RuntimeException(ioe); + } + return postRawExpectFailureBody(url, body); + } + + private static int postRawExpectFailure(String body) { + return postRawExpectFailureBody( + DEFAULT_INSTANCE.getControllerBaseApiUrl() + "/sql/ddl", body); + } + + private static int postRawExpectFailureBody(String url, String body) { + Map headers = Collections.singletonMap("Content-Type", "application/json"); + try { + sendPostRequest(url, body, headers); + fail("Expected request to fail with HTTP error"); + return -1; + } catch (IOException e) { + // The wrapper throws an HttpErrorStatusException whose message starts with the status code. + String msg = e.getMessage() == null ? "" : e.getMessage(); + // Status code is the first integer prefix in the message; default to 500 if not parseable. + for (int code : new int[]{400, 404, 409, 500}) { + if (msg.contains(String.valueOf(code))) { + return code; + } + } + return 500; + } + } +} diff --git a/pinot-sql-ddl/pom.xml b/pinot-sql-ddl/pom.xml new file mode 100644 index 000000000000..0de88f5e1c82 --- /dev/null +++ b/pinot-sql-ddl/pom.xml @@ -0,0 +1,65 @@ + + + + 4.0.0 + + pinot + org.apache.pinot + 1.6.0-SNAPSHOT + + pinot-sql-ddl + Pinot SQL DDL + SQL DDL compiler for Apache Pinot. Translates Pinot-native CREATE/DROP/SHOW TABLE + statements parsed by pinot-common into Pinot Schema + TableConfig metadata for execution by + the controller. + https://pinot.apache.org/ + + ${basedir}/.. + + + + + org.apache.pinot + pinot-common + + + org.apache.pinot + pinot-spi + + + org.apache.calcite + calcite-babel + + + + + org.testng + testng + test + + + org.assertj + assertj-core + test + + + diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledCreateTable.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledCreateTable.java new file mode 100644 index 000000000000..f5d4a2fd13f1 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledCreateTable.java @@ -0,0 +1,52 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.compile; + +import java.util.List; +import javax.annotation.Nullable; +import org.apache.pinot.spi.config.table.TableConfig; +import org.apache.pinot.spi.data.Schema; + + +/** Result of compiling {@code CREATE TABLE ...}. */ +public final class CompiledCreateTable extends CompiledDdl { + private final Schema _schema; + private final TableConfig _tableConfig; + private final boolean _ifNotExists; + + public CompiledCreateTable(@Nullable String databaseName, Schema schema, TableConfig tableConfig, + boolean ifNotExists, List warnings) { + super(DdlOperation.CREATE_TABLE, databaseName, warnings); + _schema = schema; + _tableConfig = tableConfig; + _ifNotExists = ifNotExists; + } + + public Schema getSchema() { + return _schema; + } + + public TableConfig getTableConfig() { + return _tableConfig; + } + + public boolean isIfNotExists() { + return _ifNotExists; + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDdl.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDdl.java new file mode 100644 index 000000000000..a9b8789b1af4 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDdl.java @@ -0,0 +1,56 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.compile; + +import java.util.Collections; +import java.util.List; +import javax.annotation.Nullable; + + +/** + * Base type for the result of compiling a Pinot DDL statement. Concrete subtypes carry the + * operation-specific payload (Schema/TableConfig for {@code CREATE}, target name for {@code DROP}, + * etc.). The controller dispatches on {@link #getOperation()}. + */ +public abstract class CompiledDdl { + private final DdlOperation _operation; + private final String _databaseName; + private final List _warnings; + + protected CompiledDdl(DdlOperation operation, @Nullable String databaseName, List warnings) { + _operation = operation; + _databaseName = databaseName; + _warnings = warnings == null ? Collections.emptyList() : Collections.unmodifiableList(warnings); + } + + public DdlOperation getOperation() { + return _operation; + } + + /** Database name from {@code db.table} or {@code SHOW TABLES FROM db}; may be {@code null}. */ + @Nullable + public String getDatabaseName() { + return _databaseName; + } + + /** Non-fatal compile-time warnings (e.g. unknown property routed to TableCustomConfig). */ + public List getWarnings() { + return _warnings; + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDropTable.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDropTable.java new file mode 100644 index 000000000000..029877ec7ae0 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDropTable.java @@ -0,0 +1,57 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.compile; + +import java.util.Collections; +import javax.annotation.Nullable; +import org.apache.pinot.spi.config.table.TableType; + + +/** Result of compiling {@code DROP TABLE ...}. */ +public final class CompiledDropTable extends CompiledDdl { + private final String _rawTableName; + private final TableType _tableType; + private final boolean _ifExists; + + public CompiledDropTable(@Nullable String databaseName, String rawTableName, + @Nullable TableType tableType, boolean ifExists) { + super(DdlOperation.DROP_TABLE, databaseName, Collections.emptyList()); + _rawTableName = rawTableName; + _tableType = tableType; + _ifExists = ifExists; + } + + /** Bare table name with no database prefix and no _OFFLINE/_REALTIME suffix. */ + public String getRawTableName() { + return _rawTableName; + } + + /** + * @return the requested type to drop, or {@code null} when both OFFLINE and REALTIME variants + * should be dropped. + */ + @Nullable + public TableType getTableType() { + return _tableType; + } + + public boolean isIfExists() { + return _ifExists; + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java new file mode 100644 index 000000000000..4818cd09650f --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java @@ -0,0 +1,57 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.compile; + +import java.util.Collections; +import javax.annotation.Nullable; +import org.apache.pinot.spi.config.table.TableType; + + +/** + * Result of compiling {@code SHOW CREATE TABLE [db.]name [TYPE OFFLINE | REALTIME]}. + * + *

This is a lookup-only compile result: it carries the target identifier and (optional) + * type filter; the controller is responsible for fetching the persisted Schema + TableConfig and + * running them through the canonical DDL emitter. + */ +public final class CompiledShowCreateTable extends CompiledDdl { + private final String _rawTableName; + private final TableType _tableType; + + public CompiledShowCreateTable(@Nullable String databaseName, String rawTableName, + @Nullable TableType tableType) { + super(DdlOperation.SHOW_CREATE_TABLE, databaseName, Collections.emptyList()); + _rawTableName = rawTableName; + _tableType = tableType; + } + + /** Bare table name with no database prefix and no _OFFLINE/_REALTIME suffix. */ + public String getRawTableName() { + return _rawTableName; + } + + /** + * @return the requested type, or {@code null} when the controller should auto-pick (defaults + * to OFFLINE when both variants exist). + */ + @Nullable + public TableType getTableType() { + return _tableType; + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowTables.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowTables.java new file mode 100644 index 000000000000..39edd5a4110a --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowTables.java @@ -0,0 +1,30 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.compile; + +import java.util.Collections; +import javax.annotation.Nullable; + + +/** Result of compiling {@code SHOW TABLES [FROM db]}. Carries no payload beyond the database. */ +public final class CompiledShowTables extends CompiledDdl { + public CompiledShowTables(@Nullable String databaseName) { + super(DdlOperation.SHOW_TABLES, databaseName, Collections.emptyList()); + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java new file mode 100644 index 000000000000..f54046c5e6b5 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java @@ -0,0 +1,78 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.compile; + +import java.util.HashMap; +import java.util.Locale; +import java.util.Map; +import org.apache.pinot.spi.data.FieldSpec.DataType; + + +/** + * Maps SQL data type names to Pinot {@link DataType} values. Recognizes both standard SQL names + * (BIGINT, VARCHAR, etc.) and Pinot-native aliases (LONG, STRING, BIG_DECIMAL, BYTES) that the + * Calcite grammar already exposes via {@code config.fmpp}. + */ +public final class DataTypeMapper { + private static final Map NAME_TO_DATATYPE; + + static { + Map map = new HashMap<>(); + map.put("INT", DataType.INT); + map.put("INTEGER", DataType.INT); + map.put("SMALLINT", DataType.INT); + map.put("TINYINT", DataType.INT); + map.put("BIGINT", DataType.LONG); + map.put("LONG", DataType.LONG); + map.put("FLOAT", DataType.FLOAT); + map.put("REAL", DataType.FLOAT); + map.put("DOUBLE", DataType.DOUBLE); + map.put("DECIMAL", DataType.BIG_DECIMAL); + map.put("NUMERIC", DataType.BIG_DECIMAL); + map.put("BIG_DECIMAL", DataType.BIG_DECIMAL); + map.put("BOOLEAN", DataType.BOOLEAN); + map.put("TIMESTAMP", DataType.TIMESTAMP); + map.put("VARCHAR", DataType.STRING); + map.put("CHAR", DataType.STRING); + map.put("STRING", DataType.STRING); + map.put("VARBINARY", DataType.BYTES); + map.put("BINARY", DataType.BYTES); + map.put("BYTES", DataType.BYTES); + map.put("JSON", DataType.JSON); + NAME_TO_DATATYPE = map; + } + + private DataTypeMapper() { + } + + /** + * Resolves a SQL type name (case-insensitive) to a Pinot {@link DataType}. + * + * @throws DdlCompilationException if the type is not supported. + */ + public static DataType resolve(String sqlTypeName) { + // Locale.ROOT: in Turkish locale "int".toUpperCase() yields "İNT" which fails the lookup. + String upper = sqlTypeName.toUpperCase(Locale.ROOT); + DataType dt = NAME_TO_DATATYPE.get(upper); + if (dt == null) { + throw new DdlCompilationException("Unsupported column data type: " + sqlTypeName); + } + return dt; + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompilationException.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompilationException.java new file mode 100644 index 000000000000..ca66223e6934 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompilationException.java @@ -0,0 +1,34 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.compile; + +/** + * Runtime exception raised when Pinot DDL compilation fails (parse errors, semantic errors, + * unsupported syntax, conflicting clauses, etc.). The controller surfaces the message verbatim + * to the API caller. + */ +public class DdlCompilationException extends RuntimeException { + public DdlCompilationException(String message) { + super(message); + } + + public DdlCompilationException(String message, Throwable cause) { + super(message, cause); + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java new file mode 100644 index 000000000000..149b77352584 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java @@ -0,0 +1,385 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.compile; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import javax.annotation.Nullable; +import org.apache.calcite.sql.SqlIdentifier; +import org.apache.calcite.sql.SqlLiteral; +import org.apache.calcite.sql.SqlNode; +import org.apache.pinot.spi.config.table.TableConfig; +import org.apache.pinot.spi.config.table.TableType; +import org.apache.pinot.spi.data.DateTimeFieldSpec; +import org.apache.pinot.spi.data.DimensionFieldSpec; +import org.apache.pinot.spi.data.FieldSpec; +import org.apache.pinot.spi.data.FieldSpec.DataType; +import org.apache.pinot.spi.data.MetricFieldSpec; +import org.apache.pinot.spi.data.Schema; +import org.apache.pinot.spi.utils.builder.TableConfigBuilder; +import org.apache.pinot.sql.ddl.resolved.ColumnRole; +import org.apache.pinot.sql.ddl.resolved.ResolvedColumnDefinition; +import org.apache.pinot.sql.ddl.resolved.ResolvedTableDefinition; +import org.apache.pinot.sql.parsers.CalciteSqlParser; +import org.apache.pinot.sql.parsers.SqlCompilationException; +import org.apache.pinot.sql.parsers.SqlNodeAndOptions; +import org.apache.pinot.sql.parsers.parser.SqlPinotColumnDeclaration; +import org.apache.pinot.sql.parsers.parser.SqlPinotCreateTable; +import org.apache.pinot.sql.parsers.parser.SqlPinotDropTable; +import org.apache.pinot.sql.parsers.parser.SqlPinotProperty; +import org.apache.pinot.sql.parsers.parser.SqlPinotShowCreateTable; +import org.apache.pinot.sql.parsers.parser.SqlPinotShowTables; + + +/** + * Compiles a Pinot SQL DDL statement into an executable {@link CompiledDdl}. + * + *

Top-level pipeline: + * 

+ *   SQL String
+ *     → CalciteSqlParser (in pinot-common, generated parser)
+ *     → SqlNode (one of SqlPinot{CreateTable,DropTable,ShowTables})
+ *     → ResolvedTableDefinition (CREATE only)
+ *     → Schema + TableConfig (CREATE only) via {@link PropertyMapping}
+ *     → CompiledDdl
+ *
+ * + *

Stateless and thread-safe. All entry points are static. + */ +public final class DdlCompiler { + + private DdlCompiler() { + } + + /** + * Parses and compiles a DDL statement. + * + * @param sql the raw SQL string (single statement) + * @return a {@link CompiledDdl} subclass appropriate for the operation + * @throws DdlCompilationException for parse failures or semantic errors + */ + public static CompiledDdl compile(String sql) { + SqlNode node = parse(sql); + if (node instanceof SqlPinotCreateTable) { + return compileCreate((SqlPinotCreateTable) node); + } + if (node instanceof SqlPinotDropTable) { + return compileDrop((SqlPinotDropTable) node); + } + if (node instanceof SqlPinotShowTables) { + return compileShow((SqlPinotShowTables) node); + } + if (node instanceof SqlPinotShowCreateTable) { + return compileShowCreate((SqlPinotShowCreateTable) node); + } + throw new DdlCompilationException( + "Unsupported DDL statement; expected CREATE TABLE, DROP TABLE, SHOW TABLES, " + + "or SHOW CREATE TABLE."); + } + + private static CompiledShowCreateTable compileShowCreate(SqlPinotShowCreateTable node) { + QualifiedName name = parseQualifiedName(node.getName()); + TableType tableType = node.getTableType() == null + ? null + : parseTableType(node.getTableType().toValue()); + return new CompiledShowCreateTable(name._databaseName, name._tableName, tableType); + } + + private static SqlNode parse(String sql) { + try { + SqlNodeAndOptions parsed = CalciteSqlParser.compileToSqlNodeAndOptions(sql); + return parsed.getSqlNode(); + } catch (SqlCompilationException e) { + throw new DdlCompilationException("Failed to parse DDL: " + e.getMessage(), e); + } + } + + // ------------------------------------------------------------------------------------------- + // CREATE TABLE + // ------------------------------------------------------------------------------------------- + + private static CompiledCreateTable compileCreate(SqlPinotCreateTable node) { + QualifiedName name = parseQualifiedName(node.getName()); + TableType tableType = parseTableType(node.getTableType().toValue()); + + List columns = resolveColumns(node.getColumns().getList()); + Map properties = resolveProperties(node.getProperties().getList()); + + ResolvedTableDefinition resolved = new ResolvedTableDefinition( + name._databaseName, name._tableName, tableType, node.isIfNotExists(), columns, properties); + + Schema schema = buildSchema(resolved); + List warnings = new ArrayList<>(); + TableConfig tableConfig = buildTableConfig(resolved, warnings); + validateConsistency(resolved, schema, tableConfig, warnings); + + return new CompiledCreateTable(resolved.getDatabaseName(), schema, tableConfig, + resolved.isIfNotExists(), warnings); + } + + private static List resolveColumns(List columnNodes) { + if (columnNodes.isEmpty()) { + throw new DdlCompilationException("CREATE TABLE requires at least one column."); + } + List result = new ArrayList<>(columnNodes.size()); + Set seen = new HashSet<>(); + for (SqlNode raw : columnNodes) { + if (!(raw instanceof SqlPinotColumnDeclaration)) { + throw new DdlCompilationException( + "Unexpected column node type: " + raw.getClass().getSimpleName()); + } + SqlPinotColumnDeclaration col = (SqlPinotColumnDeclaration) raw; + String name = col.getColumnName().getSimple(); + if (!seen.add(name.toLowerCase(Locale.ROOT))) { + throw new DdlCompilationException("Duplicate column name: " + name); + } + DataType dt = DataTypeMapper.resolve(col.getDataType().getTypeName().getSimple()); + ColumnRole role = inferRole(col, dt); + + String fmt = col.getDateTimeFormat() == null ? null : col.getDateTimeFormat().toValue(); + String gran = col.getDateTimeGranularity() == null ? null : col.getDateTimeGranularity().toValue(); + if (role == ColumnRole.DATETIME && (fmt == null || gran == null)) { + // Defensive: parser should have enforced this via grammar. + throw new DdlCompilationException( + "DATETIME column '" + name + "' requires both FORMAT and GRANULARITY clauses."); + } + String defaultValue = extractLiteralValue(col.getDefaultValue()); + boolean singleValue = !col.isMultiValue(); + result.add(new ResolvedColumnDefinition(name, dt, role, singleValue, !col.isNullable(), + defaultValue, fmt, gran)); + } + return result; + } + + /** + * Extracts the bare string value of a {@link SqlLiteral} (e.g. {@code 'foo'} → {@code foo}, + * {@code 0.0} → {@code "0.0"}). Uses {@link SqlLiteral#toValue()} which strips the SQL-wire + * quoting that {@code toString()} would otherwise leak into Pinot's defaultNullValue field. + * + *

Calcite's {@code toValue()} throws {@link UnsupportedOperationException} for binary + * string and interval literals; we catch and surface a typed {@link DdlCompilationException} + * so the caller sees a 400 with a useful message rather than a 500. + */ + private static String extractLiteralValue(@Nullable SqlNode literal) { + if (literal == null) { + return null; + } + if (literal instanceof SqlLiteral) { + try { + return ((SqlLiteral) literal).toValue(); + } catch (UnsupportedOperationException e) { + throw new DdlCompilationException("Unsupported DEFAULT literal: " + literal, e); + } + } + return literal.toString(); + } + + private static ColumnRole inferRole(SqlPinotColumnDeclaration col, DataType dt) { + String role = col.getRole(); + if (role == null) { + // Default: DIMENSION. Numeric columns can be promoted to METRIC by explicit annotation; + // we never silently infer METRIC because misclassification causes aggregation surprises. + return ColumnRole.DIMENSION; + } + switch (role) { + case "DIMENSION": + return ColumnRole.DIMENSION; + case "METRIC": + if (!isMetricCompatible(dt)) { + throw new DdlCompilationException( + "METRIC role requires a numeric data type; column '" + col.getColumnName().getSimple() + + "' is " + dt + "."); + } + return ColumnRole.METRIC; + case "DATETIME": + return ColumnRole.DATETIME; + default: + throw new DdlCompilationException("Unknown column role: " + role); + } + } + + private static boolean isMetricCompatible(DataType dt) { + switch (dt) { + case INT: + case LONG: + case FLOAT: + case DOUBLE: + case BIG_DECIMAL: + return true; + default: + return false; + } + } + + private static Map resolveProperties(List propertyNodes) { + Map result = new LinkedHashMap<>(propertyNodes.size()); + Set seenLower = new HashSet<>(); + for (SqlNode raw : propertyNodes) { + if (!(raw instanceof SqlPinotProperty)) { + throw new DdlCompilationException( + "Unexpected property node type: " + raw.getClass().getSimpleName()); + } + SqlPinotProperty prop = (SqlPinotProperty) raw; + String key = prop.getKeyString(); + if (!seenLower.add(key.toLowerCase(Locale.ROOT))) { + throw new DdlCompilationException("Duplicate property key: " + key); + } + result.put(key, prop.getValueString()); + } + return result; + } + + private static Schema buildSchema(ResolvedTableDefinition resolved) { + Schema schema = new Schema(); + schema.setSchemaName(resolved.getRawTableName()); + for (ResolvedColumnDefinition col : resolved.getColumns()) { + schema.addField(toFieldSpec(col)); + } + return schema; + } + + private static FieldSpec toFieldSpec(ResolvedColumnDefinition col) { + FieldSpec spec; + switch (col.getRole()) { + case METRIC: + spec = new MetricFieldSpec(col.getName(), col.getDataType()); + break; + case DATETIME: + spec = new DateTimeFieldSpec(col.getName(), col.getDataType(), + col.getDateTimeFormat(), col.getDateTimeGranularity()); + break; + case DIMENSION: + default: + spec = new DimensionFieldSpec(col.getName(), col.getDataType(), col.isSingleValue()); + break; + } + if (col.isNotNull()) { + spec.setNotNull(true); + } + if (col.getDefaultValue() != null) { + spec.setDefaultNullValue(col.getDefaultValue()); + } + return spec; + } + + private static TableConfig buildTableConfig(ResolvedTableDefinition resolved, List warnings) { + // If SQL specified `db.tableName`, prepend the db so the resulting tableName carries it + // through to the controller (DatabaseUtils.translateTableName is idempotent for already- + // qualified names). Otherwise the controller resolves the database from the HTTP header. + String tableNameForConfig = resolved.getDatabaseName() == null + ? resolved.getRawTableName() + : resolved.getDatabaseName() + "." + resolved.getRawTableName(); + TableConfigBuilder builder = new TableConfigBuilder(resolved.getTableType()) + .setTableName(tableNameForConfig); + PropertyMapping.apply(resolved, builder); + return builder.build(); + } + + /** + * Cross-checks resolved fields against the produced TableConfig (e.g. {@code timeColumnName} + * must reference a DATETIME column). Adds advisory warnings for missing-but-recommended fields. + */ + private static void validateConsistency(ResolvedTableDefinition resolved, Schema schema, + TableConfig tableConfig, List warnings) { + String timeColumnName = tableConfig.getValidationConfig().getTimeColumnName(); + if (timeColumnName != null) { + FieldSpec spec = schema.getFieldSpecFor(timeColumnName); + if (spec == null) { + throw new DdlCompilationException( + "timeColumnName '" + timeColumnName + "' does not match any declared column."); + } + if (!(spec instanceof DateTimeFieldSpec)) { + throw new DdlCompilationException( + "timeColumnName '" + timeColumnName + "' must reference a DATETIME column."); + } + } else if (resolved.getTableType() == TableType.REALTIME) { + warnings.add("REALTIME table created without 'timeColumnName' property; segment time-based " + + "operations will be unavailable."); + } + if (resolved.getTableType() == TableType.REALTIME + && tableConfig.getIndexingConfig().getStreamConfigs() == null) { + warnings.add("REALTIME table created without any 'stream.*' properties; ingestion will not " + + "start until stream configs are provided."); + } + } + + // ------------------------------------------------------------------------------------------- + // DROP TABLE + // ------------------------------------------------------------------------------------------- + + private static CompiledDropTable compileDrop(SqlPinotDropTable node) { + QualifiedName name = parseQualifiedName(node.getName()); + TableType tableType = node.getTableType() == null + ? null + : parseTableType(node.getTableType().toValue()); + return new CompiledDropTable(name._databaseName, name._tableName, tableType, node.isIfExists()); + } + + // ------------------------------------------------------------------------------------------- + // SHOW TABLES + // ------------------------------------------------------------------------------------------- + + private static CompiledShowTables compileShow(SqlPinotShowTables node) { + SqlIdentifier db = node.getDatabase(); + return new CompiledShowTables(db == null ? null : db.getSimple()); + } + + // ------------------------------------------------------------------------------------------- + // Helpers + // ------------------------------------------------------------------------------------------- + + private static TableType parseTableType(String value) { + if ("OFFLINE".equalsIgnoreCase(value)) { + return TableType.OFFLINE; + } + if ("REALTIME".equalsIgnoreCase(value)) { + return TableType.REALTIME; + } + throw new DdlCompilationException("Unknown table type: " + value); + } + + /** Splits a parser identifier into (databaseName, tableName). */ + private static QualifiedName parseQualifiedName(SqlIdentifier identifier) { + List names = identifier.names; + if (names.size() == 1) { + return new QualifiedName(null, names.get(0)); + } + if (names.size() == 2) { + return new QualifiedName(names.get(0), names.get(1)); + } + throw new DdlCompilationException( + "Table identifier must be 'name' or 'database.name'; got: " + identifier); + } + + private static final class QualifiedName { + @Nullable + final String _databaseName; + final String _tableName; + + QualifiedName(@Nullable String databaseName, String tableName) { + _databaseName = databaseName; + _tableName = tableName; + } + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlOperation.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlOperation.java new file mode 100644 index 000000000000..478c8e478158 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlOperation.java @@ -0,0 +1,27 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.compile; + +/** Discriminator for the operation a {@link CompiledDdl} represents. */ +public enum DdlOperation { + CREATE_TABLE, + DROP_TABLE, + SHOW_TABLES, + SHOW_CREATE_TABLE +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java new file mode 100644 index 000000000000..593b0b9a66c1 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java @@ -0,0 +1,473 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.compile; + +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.JsonNode; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import org.apache.pinot.spi.config.table.CompletionConfig; +import org.apache.pinot.spi.config.table.DedupConfig; +import org.apache.pinot.spi.config.table.DimensionTableConfig; +import org.apache.pinot.spi.config.table.FieldConfig; +import org.apache.pinot.spi.config.table.JsonIndexConfig; +import org.apache.pinot.spi.config.table.MultiColumnTextIndexConfig; +import org.apache.pinot.spi.config.table.QueryConfig; +import org.apache.pinot.spi.config.table.QuotaConfig; +import org.apache.pinot.spi.config.table.ReplicaGroupStrategyConfig; +import org.apache.pinot.spi.config.table.RoutingConfig; +import org.apache.pinot.spi.config.table.SegmentPartitionConfig; +import org.apache.pinot.spi.config.table.StarTreeIndexConfig; +import org.apache.pinot.spi.config.table.TableCustomConfig; +import org.apache.pinot.spi.config.table.TableTaskConfig; +import org.apache.pinot.spi.config.table.TableType; +import org.apache.pinot.spi.config.table.TagOverrideConfig; +import org.apache.pinot.spi.config.table.TierConfig; +import org.apache.pinot.spi.config.table.TunerConfig; +import org.apache.pinot.spi.config.table.UpsertConfig; +import org.apache.pinot.spi.config.table.assignment.InstanceAssignmentConfig; +import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType; +import org.apache.pinot.spi.config.table.assignment.SegmentAssignmentConfig; +import org.apache.pinot.spi.config.table.ingestion.IngestionConfig; +import org.apache.pinot.spi.config.table.sampler.TableSamplerConfig; +import org.apache.pinot.spi.utils.JsonUtils; +import org.apache.pinot.spi.utils.builder.TableConfigBuilder; +import org.apache.pinot.sql.ddl.resolved.ResolvedTableDefinition; + + +/** + * Routes Pinot DDL {@code PROPERTIES (...)} entries onto a {@link TableConfigBuilder}. + * + *

Routing rules (applied in order): + *

    + *
  1. If the key (case-insensitive) is in the promoted catalog, set the corresponding + * {@link TableConfigBuilder} field directly.
    2. + *
  2. If the key starts with {@code stream.}, route into {@code IndexingConfig.streamConfigs}. + * The {@code stream.} prefix is stripped so existing Pinot stream property keys (e.g. + * {@code stream.kafka.topic.name}) round-trip cleanly. REALTIME-only.
    3. + *
  3. If the key starts with {@code task..}, route the remainder into + * {@code TableTaskConfig.taskTypeConfigsMap[taskType]}.
    4. + *
  4. Otherwise, store verbatim in {@link TableCustomConfig#getCustomConfigs()}. + * This guarantees no silent loss of meaningful config: every DDL property survives the + * compile / persist round-trip, even when the DDL grammar is older than the property.
    5. + *
+ * + *

The free-form pass-through (rules 2-4) is the forward-compatibility hook: stream and minion + * task config schemas evolve independently and need not be in lock-step with the DDL grammar. + */ +public final class PropertyMapping { + + private static final String STREAM_PREFIX = "stream."; + private static final String REALTIME_PREFIX = "realtime."; + private static final String TASK_PREFIX = "task."; + + /** Property keys that carry table-type semantics, handled separately by the caller. */ + static final Set RESERVED_KEYS = ImmutableSet.of("tabletype", "tablename", "ifnotexists"); + + /** + * Lowercase property keys that have a dedicated PropertyMapping handler (promoted scalar or + * JSON-blob deserialization). Used by the reverse compiler to detect TableCustomConfig keys + * that would shadow a reserved key on round-trip and reject them up front. + */ + private static final Set RESERVED_ROUND_TRIP_KEYS; + static { + Set keys = new HashSet<>(); + keys.add("timecolumnname"); + keys.add("timecolumn"); + keys.add("timetype"); + keys.add("replication"); + keys.add("retentiontimeunit"); + keys.add("retentiontimevalue"); + keys.add("brokertenant"); + keys.add("servertenant"); + keys.add("loadmode"); + keys.add("sortedcolumn"); + keys.add("nullhandlingenabled"); + keys.add("isdimtable"); + keys.add("invertedindexcolumns"); + keys.add("nodictionarycolumns"); + keys.add("bloomfiltercolumns"); + keys.add("rangeindexcolumns"); + keys.add("jsonindexcolumns"); + keys.add("varlengthdictionarycolumns"); + keys.add("onheapdictionarycolumns"); + keys.add("peersegmentdownloadscheme"); + keys.add("crypterclassname"); + keys.add("deletedsegmentsretentionperiod"); + keys.add("segmentversion"); + keys.add("aggregatemetrics"); + keys.add("description"); + keys.add("tags"); + keys.add("ingestionconfig"); + keys.add("upsertconfig"); + keys.add("dedupconfig"); + keys.add("dimensiontableconfig"); + keys.add("routingconfig"); + keys.add("queryconfig"); + keys.add("quotaconfig"); + keys.add("tierconfigs"); + keys.add("tunerconfigs"); + keys.add("fieldconfigs"); + keys.add("instanceassignmentconfigmap"); + keys.add("tagoverrideconfig"); + keys.add("replicagroupstrategyconfig"); + keys.add("completionconfig"); + keys.add("startreeindexconfigs"); + keys.add("segmentpartitionconfig"); + keys.add("multicolumntextindexconfig"); + keys.add("jsonindexconfigs"); + keys.add("instancepartitionsmap"); + keys.add("segmentassignmentconfigmap"); + keys.add("tablesamplers"); + keys.add("tieroverwrites"); + keys.addAll(RESERVED_KEYS); + RESERVED_ROUND_TRIP_KEYS = Collections.unmodifiableSet(keys); + } + + /** + * Returns {@code true} if {@code lowerKey} (already lower-cased) is consumed by a dedicated + * PropertyMapping handler — i.e. it would not round-trip safely as a TableCustomConfig entry. + * + *

Catches both exact-match keys (promoted scalars + JSON-blob keys) and the + * prefix-routed paths ({@code stream.}, {@code realtime.}, {@code task.}). A custom-config + * entry with a key like {@code task.MinionTask.foo} would otherwise be silently routed into + * {@code TableTaskConfig} on re-parse. + */ + public static boolean isReservedRoundTripKey(String lowerKey) { + if (RESERVED_ROUND_TRIP_KEYS.contains(lowerKey)) { + return true; + } + return lowerKey.startsWith(STREAM_PREFIX) + || lowerKey.startsWith(REALTIME_PREFIX) + || lowerKey.startsWith(TASK_PREFIX); + } + + private PropertyMapping() { + } + + /** + * Applies all properties from {@code definition} onto {@code builder}. + * + * @throws DdlCompilationException if a promoted key has a non-coercible value (e.g. non-integer + * replication). + */ + public static void apply(ResolvedTableDefinition definition, TableConfigBuilder builder) { + Map streamConfigs = new LinkedHashMap<>(); + Map> taskConfigs = new LinkedHashMap<>(); + Map customConfigs = new LinkedHashMap<>(); + + for (Map.Entry entry : definition.getProperties().entrySet()) { + String rawKey = entry.getKey(); + String value = entry.getValue(); + String lower = rawKey.toLowerCase(Locale.ROOT); + + if (RESERVED_KEYS.contains(lower)) { + // Reserved keys are handled by the caller (e.g. tableType from the TABLE_TYPE clause). + // If they appear in PROPERTIES we treat that as a conflicting clause. + throw new DdlCompilationException( + "Property '" + rawKey + "' is reserved and must be expressed via a first-class clause, " + + "not PROPERTIES."); + } + + if (applyPromoted(lower, value, builder)) { + continue; + } + + // JSON-blob property keys: complex nested configs (ingestionConfig, upsertConfig, etc.) + // round-trip via PROPERTIES('' = ''). The reverse compiler emits these the + // same way so canonical DDL preserves all meaningful TableConfig state. + if (applyJsonBlob(rawKey, lower, value, builder)) { + continue; + } + + if (lower.startsWith(STREAM_PREFIX) || lower.startsWith(REALTIME_PREFIX)) { + // Both "stream.*" (Pinot stream connection configs) and "realtime.*" (e.g. + // realtime.segment.flush.threshold.rows / .size / .segment.size) live in + // IndexingConfig.streamConfigs in real Pinot table configs. Routing them anywhere else + // makes them silently inert. + if (definition.getTableType() != TableType.REALTIME) { + String kind = lower.startsWith(REALTIME_PREFIX) ? "Realtime" : "Stream"; + throw new DdlCompilationException( + kind + " property '" + rawKey + "' is only valid for REALTIME tables."); + } + // Preserve the original key (including its prefix) so existing Pinot stream configs + // round-trip identically. + streamConfigs.put(rawKey, value); + continue; + } + + if (lower.startsWith(TASK_PREFIX)) { + // task.. = value + String afterPrefix = rawKey.substring(TASK_PREFIX.length()); + int dot = afterPrefix.indexOf('.'); + if (dot <= 0 || dot == afterPrefix.length() - 1) { + throw new DdlCompilationException( + "Task property '" + rawKey + "' must follow the form task.."); + } + String taskType = afterPrefix.substring(0, dot); + String taskKey = afterPrefix.substring(dot + 1); + taskConfigs.computeIfAbsent(taskType, k -> new LinkedHashMap<>()).put(taskKey, value); + continue; + } + + customConfigs.put(rawKey, value); + } + + if (!streamConfigs.isEmpty()) { + builder.setStreamConfigs(streamConfigs); + } + if (!taskConfigs.isEmpty()) { + builder.setTaskConfig(new TableTaskConfig(taskConfigs)); + } + if (!customConfigs.isEmpty()) { + builder.setCustomConfig(new TableCustomConfig(customConfigs)); + } + } + + /** Returns true if {@code lowerKey} matched a promoted property and was applied. */ + private static boolean applyPromoted(String lowerKey, String value, TableConfigBuilder builder) { + switch (lowerKey) { + case "timecolumnname": + case "timecolumn": + builder.setTimeColumnName(value); + return true; + case "timetype": + builder.setTimeType(value); + return true; + case "replication": + builder.setNumReplicas(parseInt(lowerKey, value)); + return true; + case "retentiontimeunit": + builder.setRetentionTimeUnit(value); + return true; + case "retentiontimevalue": + builder.setRetentionTimeValue(value); + return true; + case "brokertenant": + builder.setBrokerTenant(value); + return true; + case "servertenant": + builder.setServerTenant(value); + return true; + case "loadmode": + builder.setLoadMode(value); + return true; + case "sortedcolumn": + // The canonical DDL emits all sort columns as CSV but TableConfigBuilder.setSortedColumn + // only accepts a single string. Use the first element so the primary sort column is + // preserved; multi-column sort configs are intentionally deferred to Slice 4. + List sortCols = splitCsv(value); + if (!sortCols.isEmpty()) { + builder.setSortedColumn(sortCols.get(0)); + } + return true; + case "nullhandlingenabled": + builder.setNullHandlingEnabled(parseBool(lowerKey, value)); + return true; + case "isdimtable": + builder.setIsDimTable(parseBool(lowerKey, value)); + return true; + case "invertedindexcolumns": + builder.setInvertedIndexColumns(splitCsv(value)); + return true; + case "nodictionarycolumns": + builder.setNoDictionaryColumns(splitCsv(value)); + return true; + case "bloomfiltercolumns": + builder.setBloomFilterColumns(splitCsv(value)); + return true; + case "rangeindexcolumns": + builder.setRangeIndexColumns(splitCsv(value)); + return true; + case "jsonindexcolumns": + builder.setJsonIndexColumns(splitCsv(value)); + return true; + case "varlengthdictionarycolumns": + builder.setVarLengthDictionaryColumns(splitCsv(value)); + return true; + case "onheapdictionarycolumns": + builder.setOnHeapDictionaryColumns(splitCsv(value)); + return true; + case "peersegmentdownloadscheme": + builder.setPeerSegmentDownloadScheme(value); + return true; + case "crypterclassname": + builder.setCrypterClassName(value); + return true; + case "deletedsegmentsretentionperiod": + builder.setDeletedSegmentsRetentionPeriod(value); + return true; + case "segmentversion": + builder.setSegmentVersion(value); + return true; + case "aggregatemetrics": + builder.setAggregateMetrics(parseBool(lowerKey, value)); + return true; + case "description": + builder.setDescription(value); + return true; + case "tags": + builder.setTags(splitCsv(value)); + return true; + default: + return false; + } + } + + /** + * Recognizes JSON-blob property keys for complex nested TableConfig fields and deserializes + * them back into the right setter on {@link TableConfigBuilder}. Returns true when the key + * was handled (regardless of value validity — invalid JSON throws so the user sees a 400). + */ + private static boolean applyJsonBlob(String rawKey, String lowerKey, String value, + TableConfigBuilder builder) { + try { + switch (lowerKey) { + case "ingestionconfig": + builder.setIngestionConfig(JsonUtils.stringToObject(value, IngestionConfig.class)); + return true; + case "upsertconfig": + builder.setUpsertConfig(JsonUtils.stringToObject(value, UpsertConfig.class)); + return true; + case "dedupconfig": + builder.setDedupConfig(JsonUtils.stringToObject(value, DedupConfig.class)); + return true; + case "dimensiontableconfig": + builder.setDimensionTableConfig( + JsonUtils.stringToObject(value, DimensionTableConfig.class)); + return true; + case "routingconfig": + builder.setRoutingConfig(JsonUtils.stringToObject(value, RoutingConfig.class)); + return true; + case "queryconfig": + builder.setQueryConfig(JsonUtils.stringToObject(value, QueryConfig.class)); + return true; + case "quotaconfig": + builder.setQuotaConfig(JsonUtils.stringToObject(value, QuotaConfig.class)); + return true; + case "tierconfigs": + builder.setTierConfigList(JsonUtils.stringToObject(value, + new TypeReference>() { })); + return true; + case "tunerconfigs": + builder.setTunerConfigList(JsonUtils.stringToObject(value, + new TypeReference>() { })); + return true; + case "fieldconfigs": + builder.setFieldConfigList(JsonUtils.stringToObject(value, + new TypeReference>() { })); + return true; + case "instanceassignmentconfigmap": + builder.setInstanceAssignmentConfigMap(JsonUtils.stringToObject(value, + new TypeReference>() { })); + return true; + case "tagoverrideconfig": + builder.setTagOverrideConfig( + JsonUtils.stringToObject(value, TagOverrideConfig.class)); + return true; + case "replicagroupstrategyconfig": + builder.setReplicaGroupStrategyConfig( + JsonUtils.stringToObject(value, ReplicaGroupStrategyConfig.class)); + return true; + case "completionconfig": + builder.setCompletionConfig(JsonUtils.stringToObject(value, CompletionConfig.class)); + return true; + case "startreeindexconfigs": + builder.setStarTreeIndexConfigs(JsonUtils.stringToObject(value, + new TypeReference>() { })); + return true; + case "segmentpartitionconfig": + builder.setSegmentPartitionConfig( + JsonUtils.stringToObject(value, SegmentPartitionConfig.class)); + return true; + case "multicolumntextindexconfig": + builder.setMultiColumnTextIndexConfig( + JsonUtils.stringToObject(value, MultiColumnTextIndexConfig.class)); + return true; + case "jsonindexconfigs": + builder.setJsonIndexConfigs(JsonUtils.stringToObject(value, + new TypeReference>() { })); + return true; + case "instancepartitionsmap": + builder.setInstancePartitionsMap(JsonUtils.stringToObject(value, + new TypeReference>() { })); + return true; + case "segmentassignmentconfigmap": + builder.setSegmentAssignmentConfigMap(JsonUtils.stringToObject(value, + new TypeReference>() { })); + return true; + case "tablesamplers": + builder.setTableSamplers(JsonUtils.stringToObject(value, + new TypeReference>() { })); + return true; + case "tieroverwrites": + builder.setTierOverwrites(JsonUtils.stringToObject(value, JsonNode.class)); + return true; + default: + return false; + } + } catch (Exception e) { + throw new DdlCompilationException( + "Failed to parse JSON value for property '" + rawKey + "': " + e.getMessage(), e); + } + } + + private static int parseInt(String key, String value) { + try { + return Integer.parseInt(value.trim()); + } catch (NumberFormatException e) { + throw new DdlCompilationException( + "Property '" + key + "' must be an integer; got '" + value + "'."); + } + } + + private static boolean parseBool(String key, String value) { + String trimmed = value.trim(); + if ("true".equalsIgnoreCase(trimmed)) { + return true; + } + if ("false".equalsIgnoreCase(trimmed)) { + return false; + } + throw new DdlCompilationException( + "Property '" + key + "' must be 'true' or 'false'; got '" + value + "'."); + } + + private static List splitCsv(String value) { + if (value == null || value.isEmpty()) { + return Collections.emptyList(); + } + String[] parts = value.split(","); + List result = new ArrayList<>(parts.length); + for (String part : parts) { + String trimmed = part.trim(); + if (!trimmed.isEmpty()) { + result.add(trimmed); + } + } + return result; + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ColumnRole.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ColumnRole.java new file mode 100644 index 000000000000..8372365622cb --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ColumnRole.java @@ -0,0 +1,32 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.resolved; + +/** + * Schema role for a resolved DDL column. This drives which Pinot {@code FieldSpec} subclass the + * column compiles to (DimensionFieldSpec / MetricFieldSpec / DateTimeFieldSpec). + */ +public enum ColumnRole { + /** Maps to {@code DimensionFieldSpec}. Filterable / groupable column. */ + DIMENSION, + /** Maps to {@code MetricFieldSpec}. Aggregatable numeric column. */ + METRIC, + /** Maps to {@code DateTimeFieldSpec}. Requires explicit format and granularity. */ + DATETIME +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedColumnDefinition.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedColumnDefinition.java new file mode 100644 index 000000000000..e726671197e1 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedColumnDefinition.java @@ -0,0 +1,88 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.resolved; + +import javax.annotation.Nullable; +import org.apache.pinot.spi.data.FieldSpec.DataType; + + +/** + * Normalized column definition produced by the DDL compiler. Independent of the Calcite parse + * tree so downstream stages (Schema generation, reverse compilation) do not depend on the parser. + * + *

Immutable; instances are safe to share across threads. + */ +public final class ResolvedColumnDefinition { + private final String _name; + private final DataType _dataType; + private final ColumnRole _role; + private final boolean _singleValue; + private final boolean _notNull; + private final String _defaultValue; + private final String _dateTimeFormat; + private final String _dateTimeGranularity; + + public ResolvedColumnDefinition(String name, DataType dataType, ColumnRole role, boolean singleValue, + boolean notNull, @Nullable String defaultValue, @Nullable String dateTimeFormat, + @Nullable String dateTimeGranularity) { + _name = name; + _dataType = dataType; + _role = role; + _singleValue = singleValue; + _notNull = notNull; + _defaultValue = defaultValue; + _dateTimeFormat = dateTimeFormat; + _dateTimeGranularity = dateTimeGranularity; + } + + public String getName() { + return _name; + } + + public DataType getDataType() { + return _dataType; + } + + public ColumnRole getRole() { + return _role; + } + + public boolean isSingleValue() { + return _singleValue; + } + + public boolean isNotNull() { + return _notNull; + } + + @Nullable + public String getDefaultValue() { + return _defaultValue; + } + + @Nullable + public String getDateTimeFormat() { + return _dateTimeFormat; + } + + @Nullable + public String getDateTimeGranularity() { + return _dateTimeGranularity; + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedTableDefinition.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedTableDefinition.java new file mode 100644 index 000000000000..0f1792b59a66 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedTableDefinition.java @@ -0,0 +1,83 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.resolved; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import javax.annotation.Nullable; +import org.apache.pinot.spi.config.table.TableType; + + +/** + * Normalized {@code CREATE TABLE} definition produced by the DDL compiler. Independent of the + * Calcite parse tree. + * + *

Properties retain insertion order via a {@link LinkedHashMap} so that downstream reverse + * compilation can produce deterministic canonical DDL. + * + *

Immutable; instances are safe to share across threads. + */ +public final class ResolvedTableDefinition { + private final String _databaseName; + private final String _rawTableName; + private final TableType _tableType; + private final boolean _ifNotExists; + private final List _columns; + private final Map _properties; + + public ResolvedTableDefinition(@Nullable String databaseName, String rawTableName, TableType tableType, + boolean ifNotExists, List columns, Map properties) { + _databaseName = databaseName; + _rawTableName = rawTableName; + _tableType = tableType; + _ifNotExists = ifNotExists; + _columns = Collections.unmodifiableList(columns); + _properties = Collections.unmodifiableMap(new LinkedHashMap<>(properties)); + } + + /** Returns the database name when one was supplied via {@code db.tableName}, else {@code null}. */ + @Nullable + public String getDatabaseName() { + return _databaseName; + } + + /** Returns the bare table name (no database prefix, no _OFFLINE/_REALTIME suffix). */ + public String getRawTableName() { + return _rawTableName; + } + + public TableType getTableType() { + return _tableType; + } + + public boolean isIfNotExists() { + return _ifNotExists; + } + + public List getColumns() { + return _columns; + } + + /** Returns property map in declaration order. */ + public Map getProperties() { + return _properties; + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java new file mode 100644 index 000000000000..741861a8dc58 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java @@ -0,0 +1,132 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.reverse; + +import java.util.List; +import java.util.Map; +import javax.annotation.Nullable; +import org.apache.pinot.spi.config.table.TableConfig; +import org.apache.pinot.spi.config.table.TableType; +import org.apache.pinot.spi.data.Schema; +import org.apache.pinot.spi.utils.builder.TableNameBuilder; + + +/** + * Renders a {@code CREATE TABLE} statement in canonical Pinot DDL form from a {@link Schema} and + * {@link TableConfig}. Designed so that {@code parse(emit(schema, config))} round-trips back to + * a semantically-equivalent (Schema, TableConfig) pair. + * + *

Canonical formatting rules: + *

    + *
  • Two-space indentation, one column per line, trailing comma between entries.
    • + *
  • Clause order: column block, {@code TABLE_TYPE}, {@code PROPERTIES}.
    • + *
  • Property keys in lexicographic order (provided by {@link PropertyExtractor}).
    • + *
  • Identifiers double-quoted only when required (reserved words, special chars).
    • + *
  • String literals always single-quoted; embedded single quotes doubled.
    • + *
+ * + *

Stateless and thread-safe. + */ +public final class CanonicalDdlEmitter { + + private static final String INDENT = " "; + + private CanonicalDdlEmitter() { + } + + /** + * Renders the canonical DDL for the given schema + table config. + * + * @param schema the table's schema; column declarations are derived from its field specs. + * @param config the table config; the table name (with type suffix stripped) and all + * non-default settings are emitted. + * @return canonical DDL ending with a semicolon and trailing newline. + */ + public static String emit(Schema schema, TableConfig config) { + return emit(schema, config, null); + } + + /** + * Renders the canonical DDL, scoped under {@code databaseName} when non-null. The database name + * is rendered as a leading {@code db.} qualifier on the table name; this matches the parser's + * {@code [db.]name} grammar. + */ + public static String emit(Schema schema, TableConfig config, @Nullable String databaseName) { + StringBuilder sb = new StringBuilder(512); + + String rawTableName = TableNameBuilder.extractRawTableName(config.getTableName()); + + // Derive the effective database: explicit argument wins; fall back to any db. prefix already + // embedded in the raw table name (e.g. analytics.events_OFFLINE → "analytics"). This ensures + // the no-database overload emit(schema, config) preserves a db-qualified table name rather + // than silently stripping it to a bare name that would land in the wrong database on replay. + String effectiveDb = databaseName; + if ((effectiveDb == null || effectiveDb.isEmpty()) && rawTableName.contains(".")) { + int dot = rawTableName.indexOf('.'); + effectiveDb = rawTableName.substring(0, dot); + } + String displayName = rawTableName.contains(".") ? rawTableName.substring(rawTableName.indexOf('.') + 1) + : rawTableName; + + sb.append("CREATE TABLE "); + if (effectiveDb != null && !effectiveDb.isEmpty()) { + sb.append(SqlIdentifiers.quote(effectiveDb)).append('.'); + } + sb.append(SqlIdentifiers.quote(displayName)); + sb.append(" (\n"); + List columns = SchemaEmitter.emitColumns(schema); + for (int i = 0; i < columns.size(); i++) { + sb.append(INDENT).append(columns.get(i)); + if (i < columns.size() - 1) { + sb.append(','); + } + sb.append('\n'); + } + sb.append(")\n"); + + sb.append("TABLE_TYPE = ").append(emitTableType(config.getTableType())).append('\n'); + + Map props = PropertyExtractor.extract(config); + if (!props.isEmpty()) { + sb.append("PROPERTIES (\n"); + int idx = 0; + int last = props.size() - 1; + for (Map.Entry e : props.entrySet()) { + sb.append(INDENT) + .append(SqlIdentifiers.quoteString(e.getKey())) + .append(" = ") + .append(SqlIdentifiers.quoteString(e.getValue())); + if (idx++ < last) { + sb.append(','); + } + sb.append('\n'); + } + sb.append(")"); + } else { + // Trim the trailing newline before the semicolon when there's no PROPERTIES block. + sb.setLength(sb.length() - 1); + } + sb.append(";\n"); + return sb.toString(); + } + + private static String emitTableType(TableType tableType) { + return tableType == TableType.OFFLINE ? "OFFLINE" : "REALTIME"; + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java new file mode 100644 index 000000000000..25988d4544f7 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java @@ -0,0 +1,298 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.reverse; + +import java.util.Collection; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.TreeMap; +import javax.annotation.Nullable; +import org.apache.pinot.spi.config.table.IndexingConfig; +import org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig; +import org.apache.pinot.spi.config.table.TableConfig; +import org.apache.pinot.spi.config.table.TableCustomConfig; +import org.apache.pinot.spi.config.table.TableTaskConfig; +import org.apache.pinot.spi.config.table.TenantConfig; +import org.apache.pinot.spi.utils.JsonUtils; +import org.apache.pinot.sql.ddl.compile.PropertyMapping; + + +/** + * Inverse of {@code PropertyMapping}: walks a {@link TableConfig} and emits a property map + * suitable for the {@code PROPERTIES (...)} clause in canonical DDL. The map is sorted + * lexicographically so canonical output is deterministic. + * + *

Routing rules mirror {@code PropertyMapping}: + *

    + *
  • Promoted fields (timeColumnName, replication, brokerTenant, …) emit under their + * known property keys.
    • + *
  • Stream configs are emitted verbatim using their original key (preserving the + * {@code stream.}/{@code realtime.} prefix).
    • + *
  • Task configs are emitted as {@code task.<type>.<key> = <value>}.
    • + *
  • Table custom-config entries are emitted verbatim.
    • + *
  • Complex nested configs (ingestionConfig, upsertConfig, routingConfig, etc.) that have no + * first-class clause yet are emitted as JSON-string property values under the same key + * used by Pinot's own JSON serialization. The forward {@code PropertyMapping} parses + * these blobs back into the corresponding TableConfig fields.
    • + *
+ * + *

Defaults are skipped where doing so doesn't lose information (e.g. {@code loadMode=MMAP} + * or {@code replication=1}) so canonical output stays compact. + * + *

Coverage scope (Slice 2): this extractor covers the TableConfig fields most users + * configure today plus a JSON-blob fallback for the major nested configs. Several long-tail + * fields are intentionally not yet emitted — round-trip will silently lose them — and are + * tracked for Slice 4 hardening: + *

    + *
  • SegmentsValidationAndRetentionConfig: {@code segmentAssignmentStrategy} (deprecated), + * {@code segmentPushType}/{@code segmentPushFrequency} (deprecated; use IngestionConfig), + * {@code replicasPerPartition}, {@code untrackedSegmentsDeletion*} fields.
    • + *
  • IndexingConfig: {@code rangeIndexVersion}, {@code enableDefaultStarTree}, + * {@code enableDynamicStarTreeCreation}, {@code columnMajorSegmentBuilderEnabled}, + * {@code skipSegmentPreprocess}, {@code optimizeNoDictStatsCollection}, + * {@code optimizeDictionary*}, {@code noDictionarySize/CardinalityRatioThreshold}, + * {@code segmentNameGeneratorType}, {@code columnMinMaxValueGeneratorMode}, + * {@code autoGeneratedInvertedIndex}, {@code createInvertedIndexDuringSegmentGeneration}.
    • + *
+ * Slice 4 will either add explicit handlers for each or introduce a guard test that fails + * compilation when a TableConfig sets a non-default value for an unhandled field. + */ +public final class PropertyExtractor { + + /** Property keys whose values are JSON-serialized blobs of the corresponding nested config. */ + static final String KEY_INGESTION_CONFIG = "ingestionConfig"; + static final String KEY_UPSERT_CONFIG = "upsertConfig"; + static final String KEY_DEDUP_CONFIG = "dedupConfig"; + static final String KEY_ROUTING_CONFIG = "routingConfig"; + static final String KEY_QUERY_CONFIG = "queryConfig"; + static final String KEY_QUOTA_CONFIG = "quotaConfig"; + static final String KEY_TIER_CONFIGS = "tierConfigs"; + static final String KEY_TUNER_CONFIGS = "tunerConfigs"; + static final String KEY_FIELD_CONFIGS = "fieldConfigs"; + static final String KEY_INSTANCE_ASSIGNMENT = "instanceAssignmentConfigMap"; + static final String KEY_INSTANCE_PARTITIONS = "instancePartitionsMap"; + static final String KEY_SEGMENT_ASSIGNMENT = "segmentAssignmentConfigMap"; + static final String KEY_DIMENSION_TABLE_CONFIG = "dimensionTableConfig"; + static final String KEY_TAG_OVERRIDE_CONFIG = "tagOverrideConfig"; + static final String KEY_REPLICA_GROUP_STRATEGY = "replicaGroupStrategyConfig"; + static final String KEY_COMPLETION_CONFIG = "completionConfig"; + static final String KEY_STAR_TREE_INDEX_CONFIGS = "starTreeIndexConfigs"; + static final String KEY_MULTI_COLUMN_TEXT_INDEX = "multiColumnTextIndexConfig"; + static final String KEY_JSON_INDEX_CONFIGS = "jsonIndexConfigs"; + static final String KEY_SEGMENT_PARTITION_CONFIG = "segmentPartitionConfig"; + static final String KEY_TIER_OVERWRITES = "tierOverwrites"; + static final String KEY_TABLE_SAMPLERS = "tableSamplers"; + + private PropertyExtractor() { + } + + /** Extracts a deterministic, lexicographically-sorted property map from {@code config}. */ + public static Map extract(TableConfig config) { + Map props = new TreeMap<>(); + extractValidation(config.getValidationConfig(), props); + extractTenant(config.getTenantConfig(), props); + extractIndexing(config.getIndexingConfig(), props); + extractTask(config.getTaskConfig(), props); + extractCustom(config.getCustomConfig(), props); + extractTopLevel(config, props); + extractNestedAsJsonBlobs(config, props); + return props; + } + + private static void extractValidation(@Nullable SegmentsValidationAndRetentionConfig v, + Map props) { + if (v == null) { + return; + } + putIfPresent(props, "timeColumnName", v.getTimeColumnName()); + if (v.getTimeType() != null) { + props.put("timeType", v.getTimeType().name()); + } + putIfPresent(props, "retentionTimeUnit", v.getRetentionTimeUnit()); + putIfPresent(props, "retentionTimeValue", v.getRetentionTimeValue()); + // replication defaults to "1"; emit when explicitly different to keep canonical compact. + String replication = v.getReplication(); + if (replication != null && !"1".equals(replication)) { + props.put("replication", replication); + } + putIfPresent(props, "peerSegmentDownloadScheme", v.getPeerSegmentDownloadScheme()); + putIfPresent(props, "crypterClassName", v.getCrypterClassName()); + putIfPresent(props, "deletedSegmentsRetentionPeriod", + skipDefault(v.getDeletedSegmentsRetentionPeriod(), "7d")); + putJsonIfPresent(props, KEY_REPLICA_GROUP_STRATEGY, v.getReplicaGroupStrategyConfig()); + putJsonIfPresent(props, KEY_COMPLETION_CONFIG, v.getCompletionConfig()); + } + + private static void extractTenant(@Nullable TenantConfig t, Map props) { + if (t == null) { + return; + } + putIfPresent(props, "brokerTenant", t.getBroker()); + putIfPresent(props, "serverTenant", t.getServer()); + putJsonIfPresent(props, KEY_TAG_OVERRIDE_CONFIG, t.getTagOverrideConfig()); + } + + private static void extractIndexing(@Nullable IndexingConfig i, Map props) { + if (i == null) { + return; + } + // loadMode defaults to MMAP. + String loadMode = i.getLoadMode(); + if (loadMode != null && !"MMAP".equalsIgnoreCase(loadMode)) { + props.put("loadMode", loadMode); + } + putIfPresent(props, "segmentVersion", i.getSegmentFormatVersion()); + // Emit all sort columns as CSV. TableConfigBuilder.setSortedColumn only accepts a single + // string so PropertyMapping restores only the first column on re-parse; the full list is + // preserved in canonical DDL so the value is not silently dropped. + putCsvIfPresent(props, "sortedColumn", i.getSortedColumn(), false); + putCsvIfPresent(props, "invertedIndexColumns", i.getInvertedIndexColumns(), false); + putCsvIfPresent(props, "noDictionaryColumns", i.getNoDictionaryColumns(), false); + putCsvIfPresent(props, "onHeapDictionaryColumns", i.getOnHeapDictionaryColumns(), false); + putCsvIfPresent(props, "varLengthDictionaryColumns", i.getVarLengthDictionaryColumns(), false); + putCsvIfPresent(props, "bloomFilterColumns", i.getBloomFilterColumns(), false); + putCsvIfPresent(props, "rangeIndexColumns", i.getRangeIndexColumns(), false); + putCsvIfPresent(props, "jsonIndexColumns", i.getJsonIndexColumns(), false); + if (i.isNullHandlingEnabled()) { + props.put("nullHandlingEnabled", "true"); + } + if (i.isAggregateMetrics()) { + props.put("aggregateMetrics", "true"); + } + // Stream configs are routed verbatim with their original keys (stream.* / realtime.*). + Map streamConfigs = i.getStreamConfigs(); + if (streamConfigs != null) { + props.putAll(streamConfigs); + } + putJsonIfPresent(props, KEY_SEGMENT_PARTITION_CONFIG, i.getSegmentPartitionConfig()); + putJsonIfPresent(props, KEY_STAR_TREE_INDEX_CONFIGS, i.getStarTreeIndexConfigs()); + putJsonIfPresent(props, KEY_MULTI_COLUMN_TEXT_INDEX, i.getMultiColumnTextIndexConfig()); + putJsonIfPresent(props, KEY_JSON_INDEX_CONFIGS, i.getJsonIndexConfigs()); + putJsonIfPresent(props, KEY_TIER_OVERWRITES, i.getTierOverwrites()); + } + + private static void extractTask(@Nullable TableTaskConfig t, Map props) { + if (t == null || t.getTaskTypeConfigsMap() == null) { + return; + } + for (Map.Entry> taskEntry : t.getTaskTypeConfigsMap().entrySet()) { + String taskType = taskEntry.getKey(); + for (Map.Entry kv : taskEntry.getValue().entrySet()) { + props.put("task." + taskType + "." + kv.getKey(), kv.getValue()); + } + } + } + + private static void extractCustom(@Nullable TableCustomConfig c, Map props) { + if (c == null || c.getCustomConfigs() == null) { + return; + } + // Custom configs are emitted verbatim and round-trip via the property fall-through case in + // PropertyMapping. We refuse keys that would shadow a promoted scalar or a JSON-blob key: + // emitting them would cause re-parse to either misroute the value to the wrong TableConfig + // field (silent corruption) or fail JSON deserialization. The caller can rename the + // colliding key to dodge the conflict. + for (Map.Entry e : c.getCustomConfigs().entrySet()) { + String key = e.getKey(); + String lower = key.toLowerCase(Locale.ROOT); + if (PropertyMapping.isReservedRoundTripKey(lower)) { + throw new IllegalArgumentException("TableCustomConfig key '" + key + + "' collides with a reserved DDL property name (promoted scalar or JSON blob); " + + "rename the custom-config entry so canonical DDL emission is unambiguous."); + } + props.put(key, e.getValue()); + } + } + + private static void extractTopLevel(TableConfig config, Map props) { + if (config.isDimTable()) { + props.put("isDimTable", "true"); + } + putIfPresent(props, "description", config.getDescription()); + List tags = config.getTags(); + if (tags != null && !tags.isEmpty()) { + props.put("tags", String.join(",", tags)); + } + } + + private static void extractNestedAsJsonBlobs(TableConfig config, Map props) { + putJsonIfPresent(props, KEY_INGESTION_CONFIG, config.getIngestionConfig()); + putJsonIfPresent(props, KEY_UPSERT_CONFIG, config.getUpsertConfig()); + putJsonIfPresent(props, KEY_DEDUP_CONFIG, config.getDedupConfig()); + putJsonIfPresent(props, KEY_DIMENSION_TABLE_CONFIG, config.getDimensionTableConfig()); + putJsonIfPresent(props, KEY_ROUTING_CONFIG, config.getRoutingConfig()); + putJsonIfPresent(props, KEY_QUERY_CONFIG, config.getQueryConfig()); + putJsonIfPresent(props, KEY_QUOTA_CONFIG, config.getQuotaConfig()); + putJsonIfPresent(props, KEY_TIER_CONFIGS, config.getTierConfigsList()); + putJsonIfPresent(props, KEY_TUNER_CONFIGS, config.getTunerConfigsList()); + putJsonIfPresent(props, KEY_FIELD_CONFIGS, config.getFieldConfigList()); + putJsonIfPresent(props, KEY_INSTANCE_ASSIGNMENT, config.getInstanceAssignmentConfigMap()); + putJsonIfPresent(props, KEY_INSTANCE_PARTITIONS, config.getInstancePartitionsMap()); + putJsonIfPresent(props, KEY_SEGMENT_ASSIGNMENT, config.getSegmentAssignmentConfigMap()); + putJsonIfPresent(props, KEY_TABLE_SAMPLERS, config.getTableSamplers()); + } + + // --------------------------------------------------------------------------------------------- + + private static void putIfPresent(Map props, String key, @Nullable String value) { + if (value != null && !value.isEmpty()) { + props.put(key, value); + } + } + + private static void putCsvIfPresent(Map props, String key, + @Nullable List list, boolean firstOnly) { + if (list == null || list.isEmpty()) { + return; + } + if (firstOnly) { + props.put(key, list.get(0)); + } else { + props.put(key, String.join(",", list)); + } + } + + private static void putJsonIfPresent(Map props, String key, + @Nullable Object obj) { + if (obj == null) { + return; + } + if (obj instanceof Collection && ((Collection) obj).isEmpty()) { + return; + } + if (obj instanceof Map && ((Map) obj).isEmpty()) { + return; + } + try { + props.put(key, JsonUtils.objectToString(obj)); + } catch (Exception e) { + // Defensive: any field we can't serialize can't round-trip; log and skip rather than + // crash the SHOW CREATE TABLE call. + throw new IllegalStateException( + "Failed to serialize TableConfig field '" + key + "' to JSON for canonical DDL emission", + e); + } + } + + @Nullable + private static String skipDefault(@Nullable String value, String defaultValue) { + return value == null || defaultValue.equals(value) ? null : value; + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java new file mode 100644 index 000000000000..4aacdef7c6bb --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java @@ -0,0 +1,142 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.reverse; + +import java.util.ArrayList; +import java.util.List; +import org.apache.pinot.spi.data.DateTimeFieldSpec; +import org.apache.pinot.spi.data.DimensionFieldSpec; +import org.apache.pinot.spi.data.FieldSpec; +import org.apache.pinot.spi.data.FieldSpec.DataType; +import org.apache.pinot.spi.data.MetricFieldSpec; +import org.apache.pinot.spi.data.Schema; + + +/** + * Emits a list of canonical column-declaration strings from a Pinot {@link Schema}. Each + * declaration is formatted to match the grammar consumed by {@code SqlPinotColumnDeclaration}. + * + *

Column ordering follows the schema's natural ordering: dimensions first (insertion order), + * then metrics, then date-time columns. This mirrors what users see in JSON config and is stable + * across emit/parse round-trips. + */ +final class SchemaEmitter { + + private SchemaEmitter() { + } + + static List emitColumns(Schema schema) { + List out = new ArrayList<>(); + for (DimensionFieldSpec dim : schema.getDimensionFieldSpecs()) { + out.add(emitColumn(dim)); + } + for (MetricFieldSpec metric : schema.getMetricFieldSpecs()) { + out.add(emitColumn(metric)); + } + for (DateTimeFieldSpec dt : schema.getDateTimeFieldSpecs()) { + out.add(emitColumn(dt)); + } + return out; + } + + private static String emitColumn(FieldSpec spec) { + StringBuilder sb = new StringBuilder(); + sb.append(SqlIdentifiers.quote(spec.getName())); + sb.append(' ').append(emitDataType(spec.getDataType())); + if (spec.isNotNull()) { + sb.append(" NOT NULL"); + } + // Only emit DEFAULT when the user-supplied value differs from the data-type's natural + // default; this matches Pinot's own JSON serialization rule and keeps canonical output + // free of redundant defaults. + Object defaultValue = spec.getDefaultNullValue(); + Object naturalDefault = + FieldSpec.getDefaultNullValue(spec.getFieldType(), spec.getDataType(), null); + if (defaultValue != null && !defaultValue.equals(naturalDefault)) { + sb.append(" DEFAULT ").append(emitDefault(defaultValue, spec.getDataType())); + } + if (spec instanceof DateTimeFieldSpec) { + DateTimeFieldSpec dt = (DateTimeFieldSpec) spec; + sb.append(" DATETIME FORMAT ").append(SqlIdentifiers.quoteString(dt.getFormat())); + sb.append(" GRANULARITY ").append(SqlIdentifiers.quoteString(dt.getGranularity())); + } else if (spec instanceof MetricFieldSpec) { + sb.append(" METRIC"); + } else { + // DimensionFieldSpec — emit DIMENSION (with ARRAY for multi-value) so the round-trip + // preserves the MV shape. Single-value is the default and needs no suffix. + sb.append(" DIMENSION"); + if (!spec.isSingleValueField()) { + sb.append(" ARRAY"); + } + } + return sb.toString(); + } + + /** + * Maps a Pinot {@link DataType} to the canonical SQL keyword we emit. We pick the + * Pinot-native names (LONG, BIG_DECIMAL, STRING, BYTES) where they exist so the round-trip + * matches what a human would write. + */ + private static String emitDataType(DataType dt) { + switch (dt) { + case INT: + return "INT"; + case LONG: + return "LONG"; + case FLOAT: + return "FLOAT"; + case DOUBLE: + return "DOUBLE"; + case BIG_DECIMAL: + return "BIG_DECIMAL"; + case BOOLEAN: + return "BOOLEAN"; + case TIMESTAMP: + return "TIMESTAMP"; + case STRING: + return "STRING"; + case JSON: + return "JSON"; + case BYTES: + return "BYTES"; + default: + // Fall back to the enum name for types we don't have a canonical short name for + // (LIST, MAP, STRUCT, UNKNOWN). These are not yet expressible in DDL but emitting the + // name lets a future grammar accept them. + return dt.name(); + } + } + + private static String emitDefault(Object value, DataType dt) { + if (value == null) { + return "NULL"; + } + switch (dt) { + case INT: + case LONG: + case FLOAT: + case DOUBLE: + case BIG_DECIMAL: + case BOOLEAN: + return value.toString(); + default: + return SqlIdentifiers.quoteString(value.toString()); + } + } +} diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SqlIdentifiers.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SqlIdentifiers.java new file mode 100644 index 000000000000..ec9bd2176d35 --- /dev/null +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SqlIdentifiers.java @@ -0,0 +1,84 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.reverse; + +import com.google.common.collect.ImmutableSet; +import java.util.Locale; +import java.util.Set; +import java.util.regex.Pattern; + + +/** Quoting helpers used by the canonical DDL emitter. */ +final class SqlIdentifiers { + + /** + * SQL identifiers that need double-quoting because they conflict with the Pinot DDL grammar + * (reserved or context-sensitive keywords) or because they are not safe bare identifiers. + * + *

This set is intentionally over-cautious: a few extra quotes never break round-trip, + * but a missing quote would cause a re-parse to misinterpret the identifier as a keyword. + */ + private static final Set ALWAYS_QUOTE = ImmutableSet.of( + // grammar keywords introduced by Pinot DDL that could appear as user identifiers + "DIMENSION", "METRIC", "DATETIME", "FORMAT", "GRANULARITY", "OFFLINE", "REALTIME", + "PROPERTIES", "TABLES", "TABLE_TYPE", "IF", "TYPE", "FROM", + // standard SQL keywords that round-trip incorrectly when bare + "TABLE", "CREATE", "DROP", "SHOW", "NOT", "NULL", "DEFAULT", "EXISTS", "AS", "BY", + "ORDER", "PRIMARY", "KEY", "WITH", "ON", "AND", "OR", "SELECT", "WHERE", + // data-type keywords consumed by the column-declaration grammar's DataType() rule. + // Without quoting, a column literally named e.g. "int" would re-parse as a type token + // rather than an identifier and the column declaration would fail. + "INT", "INTEGER", "SMALLINT", "TINYINT", "BIGINT", "LONG", "FLOAT", "REAL", "DOUBLE", + "DECIMAL", "NUMERIC", "BIG_DECIMAL", "BOOLEAN", "TIMESTAMP", "VARCHAR", "CHAR", "STRING", + "VARBINARY", "BINARY", "BYTES", "JSON"); + + private static final Pattern BARE_IDENTIFIER = Pattern.compile("[A-Za-z_][A-Za-z0-9_]*"); + + private SqlIdentifiers() { + } + + /** + * Returns {@code identifier} ready to embed in canonical DDL: double-quoted if required, + * bare otherwise. Embedded double quotes are escaped per SQL convention ({@code "} → {@code ""}). + */ + static String quote(String identifier) { + if (mustQuote(identifier)) { + return "\"" + identifier.replace("\"", "\"\"") + "\""; + } + return identifier; + } + + /** + * Returns a single-quoted SQL string literal for {@code value}; embedded single quotes are + * doubled per SQL convention. + */ + static String quoteString(String value) { + return "'" + value.replace("'", "''") + "'"; + } + + private static boolean mustQuote(String identifier) { + if (identifier == null || identifier.isEmpty()) { + return true; + } + if (!BARE_IDENTIFIER.matcher(identifier).matches()) { + return true; + } + return ALWAYS_QUOTE.contains(identifier.toUpperCase(Locale.ROOT)); + } +} diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java new file mode 100644 index 000000000000..72107a6d6ba8 --- /dev/null +++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java @@ -0,0 +1,359 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.compile; + +import java.util.Map; +import org.apache.pinot.spi.config.table.TableConfig; +import org.apache.pinot.spi.config.table.TableType; +import org.apache.pinot.spi.data.DateTimeFieldSpec; +import org.apache.pinot.spi.data.DimensionFieldSpec; +import org.apache.pinot.spi.data.FieldSpec; +import org.apache.pinot.spi.data.FieldSpec.DataType; +import org.apache.pinot.spi.data.MetricFieldSpec; +import org.apache.pinot.spi.data.Schema; +import org.testng.annotations.Test; + +import static org.testng.Assert.assertEquals; +import static org.testng.Assert.assertFalse; +import static org.testng.Assert.assertNotNull; +import static org.testng.Assert.assertNull; +import static org.testng.Assert.assertTrue; +import static org.testng.Assert.expectThrows; + + +/** End-to-end compiler tests: SQL → CompiledDdl → Schema + TableConfig. */ +public class DdlCompilerTest { + + // ------------------------------------------------------------------------------------------- + // CREATE TABLE: schema mapping + // ------------------------------------------------------------------------------------------- + + @Test + public void offlineMinimal() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE events (id INT, name STRING) TABLE_TYPE = OFFLINE"); + assertEquals(c.getTableConfig().getTableType(), TableType.OFFLINE); + // TableConfig auto-appends the type suffix; schema name retains the bare table name. + assertEquals(c.getTableConfig().getTableName(), "events_OFFLINE"); + assertEquals(c.getSchema().getSchemaName(), "events"); + assertEquals(c.getSchema().getDimensionNames().size(), 2); + // Default replication is "1" from TableConfigBuilder. + assertEquals(c.getTableConfig().getValidationConfig().getReplication(), "1"); + } + + @Test + public void databaseQualifiedNamePreservedInTableConfig() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE analytics.events (id INT) TABLE_TYPE = OFFLINE"); + assertEquals(c.getDatabaseName(), "analytics"); + assertEquals(c.getTableConfig().getTableName(), "analytics.events_OFFLINE"); + // Schema name is the bare raw name; database scoping is on the table side. + assertEquals(c.getSchema().getSchemaName(), "events"); + } + + @Test + public void columnRolesProduceCorrectFieldSpecs() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE t (" + + " d1 STRING DIMENSION," + + " m1 LONG METRIC," + + " ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'" + + ") TABLE_TYPE = OFFLINE"); + + Schema s = c.getSchema(); + FieldSpec d1 = s.getFieldSpecFor("d1"); + FieldSpec m1 = s.getFieldSpecFor("m1"); + FieldSpec ts = s.getFieldSpecFor("ts"); + assertTrue(d1 instanceof DimensionFieldSpec); + assertTrue(m1 instanceof MetricFieldSpec); + assertTrue(ts instanceof DateTimeFieldSpec); + assertEquals(((DateTimeFieldSpec) ts).getFormat(), "1:MILLISECONDS:EPOCH"); + assertEquals(((DateTimeFieldSpec) ts).getGranularity(), "1:MILLISECONDS"); + } + + @Test + public void unspecifiedRoleDefaultsToDimension() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE t (id INT, name STRING) TABLE_TYPE = OFFLINE"); + assertTrue(c.getSchema().getFieldSpecFor("id") instanceof DimensionFieldSpec); + assertTrue(c.getSchema().getFieldSpecFor("name") instanceof DimensionFieldSpec); + } + + @Test + public void notNullSetsFieldSpecFlag() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE t (id INT NOT NULL) TABLE_TYPE = OFFLINE"); + assertTrue(c.getSchema().getFieldSpecFor("id").isNotNull()); + } + + @Test + public void dataTypeMappingCoversCommonTypes() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE t (" + + " c_int INT," + + " c_bigint BIGINT," + + " c_long LONG," + + " c_float FLOAT," + + " c_double DOUBLE," + + " c_decimal DECIMAL," + + " c_bool BOOLEAN," + + " c_str STRING," + + " c_var VARCHAR," + + " c_bin BYTES," + + " c_ts TIMESTAMP" + + ") TABLE_TYPE = OFFLINE"); + Schema s = c.getSchema(); + assertEquals(s.getFieldSpecFor("c_int").getDataType(), DataType.INT); + assertEquals(s.getFieldSpecFor("c_bigint").getDataType(), DataType.LONG); + assertEquals(s.getFieldSpecFor("c_long").getDataType(), DataType.LONG); + assertEquals(s.getFieldSpecFor("c_float").getDataType(), DataType.FLOAT); + assertEquals(s.getFieldSpecFor("c_double").getDataType(), DataType.DOUBLE); + assertEquals(s.getFieldSpecFor("c_decimal").getDataType(), DataType.BIG_DECIMAL); + assertEquals(s.getFieldSpecFor("c_bool").getDataType(), DataType.BOOLEAN); + assertEquals(s.getFieldSpecFor("c_str").getDataType(), DataType.STRING); + assertEquals(s.getFieldSpecFor("c_var").getDataType(), DataType.STRING); + assertEquals(s.getFieldSpecFor("c_bin").getDataType(), DataType.BYTES); + assertEquals(s.getFieldSpecFor("c_ts").getDataType(), DataType.TIMESTAMP); + } + + // ------------------------------------------------------------------------------------------- + // CREATE TABLE: property mapping + // ------------------------------------------------------------------------------------------- + + @Test + public void promotedPropertiesMapToTableConfigFields() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE events (" + + " ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'" + + ") TABLE_TYPE = OFFLINE PROPERTIES (" + + " 'replication' = '3'," + + " 'retentionTimeUnit' = 'DAYS'," + + " 'retentionTimeValue' = '30'," + + " 'brokerTenant' = 'tenantA'," + + " 'serverTenant' = 'tenantB'," + + " 'timeColumnName' = 'ts'," + + " 'sortedColumn' = 'ts'" + + ")"); + TableConfig cfg = c.getTableConfig(); + assertEquals(cfg.getValidationConfig().getReplication(), "3"); + assertEquals(cfg.getValidationConfig().getRetentionTimeUnit(), "DAYS"); + assertEquals(cfg.getValidationConfig().getRetentionTimeValue(), "30"); + assertEquals(cfg.getValidationConfig().getTimeColumnName(), "ts"); + assertEquals(cfg.getTenantConfig().getBroker(), "tenantA"); + assertEquals(cfg.getTenantConfig().getServer(), "tenantB"); + assertEquals(cfg.getIndexingConfig().getSortedColumn().get(0), "ts"); + } + + @Test + public void streamPropertiesRoutedToStreamConfigsForRealtime() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE events (" + + " ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'" + + ") TABLE_TYPE = REALTIME PROPERTIES (" + + " 'timeColumnName' = 'ts'," + + " 'stream.kafka.topic.name' = 'orders'," + + " 'stream.kafka.consumer.factory.class.name' = 'KafkaConsumerFactory'," + + " 'realtime.segment.flush.threshold.rows' = '500000'" + + ")"); + Map stream = c.getTableConfig().getIndexingConfig().getStreamConfigs(); + assertNotNull(stream); + // Both "stream.*" and "realtime.*" prefixes route to streamConfigs because that is where + // Pinot actually reads them; routing elsewhere would make them silently inert. + assertEquals(stream.get("stream.kafka.topic.name"), "orders"); + assertEquals(stream.get("stream.kafka.consumer.factory.class.name"), "KafkaConsumerFactory"); + assertEquals(stream.get("realtime.segment.flush.threshold.rows"), "500000"); + } + + @Test + public void realtimePropertyOnOfflineTableRejected() { + expectThrows(DdlCompilationException.class, () -> compileCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES (" + + " 'realtime.segment.flush.threshold.rows' = '500000')")); + } + + @Test + public void stringDefaultDoesNotLeakSqlQuotes() { + // Regression for a bug where DEFAULT 'foo' produced a defaultNullValue of "'foo'" (with + // surrounding quotes) because we were calling SqlNode.toString() instead of + // SqlLiteral.getValue().toString(). The latter strips the SQL-wire quoting. + CompiledCreateTable c = compileCreate( + "CREATE TABLE t (s STRING DEFAULT 'unknown') TABLE_TYPE = OFFLINE"); + Object defaultValue = c.getSchema().getFieldSpecFor("s").getDefaultNullValue(); + assertEquals(defaultValue, "unknown"); + } + + @Test + public void numericDefaultRoundTripsCorrectly() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE t (m DOUBLE DEFAULT 0.0 METRIC) TABLE_TYPE = OFFLINE"); + Object defaultValue = c.getSchema().getFieldSpecFor("m").getDefaultNullValue(); + // FieldSpec coerces the string into the column's data type; the resulting value should be + // numerically zero however it is represented. + assertEquals(((Number) defaultValue).doubleValue(), 0.0); + } + + @Test + public void taskPropertiesRoutedToTaskConfig() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES (" + + " 'task.RealtimeToOfflineSegmentsTask.bucketTimePeriod' = '1d'," + + " 'task.RealtimeToOfflineSegmentsTask.maxNumRecordsPerSegment' = '5000000'," + + " 'task.SegmentRefreshTask.tableMaxNumTasks' = '5'" + + ")"); + Map> tasks = c.getTableConfig().getTaskConfig().getTaskTypeConfigsMap(); + assertEquals(tasks.size(), 2); + assertEquals(tasks.get("RealtimeToOfflineSegmentsTask").get("bucketTimePeriod"), "1d"); + assertEquals(tasks.get("RealtimeToOfflineSegmentsTask").get("maxNumRecordsPerSegment"), "5000000"); + assertEquals(tasks.get("SegmentRefreshTask").get("tableMaxNumTasks"), "5"); + } + + @Test + public void unknownPropertiesPreservedInCustomConfig() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES (" + + " 'mySpecialKey' = 'someValue'," + + " 'org.example.flag' = 'true'" + + ")"); + Map custom = c.getTableConfig().getCustomConfig().getCustomConfigs(); + assertNotNull(custom); + assertEquals(custom.get("mySpecialKey"), "someValue"); + assertEquals(custom.get("org.example.flag"), "true"); + } + + // ------------------------------------------------------------------------------------------- + // CREATE TABLE: validation + // ------------------------------------------------------------------------------------------- + + @Test + public void timeColumnMustReferenceDatetimeField() { + DdlCompilationException e = expectThrows(DdlCompilationException.class, () -> compileCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES ('timeColumnName' = 'id')")); + assertTrue(e.getMessage().contains("DATETIME"), e.getMessage()); + } + + @Test + public void timeColumnMustExist() { + DdlCompilationException e = expectThrows(DdlCompilationException.class, () -> compileCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES ('timeColumnName' = 'missing')")); + assertTrue(e.getMessage().contains("missing"), e.getMessage()); + } + + @Test + public void duplicateColumnRejected() { + expectThrows(DdlCompilationException.class, () -> compileCreate( + "CREATE TABLE t (id INT, ID STRING) TABLE_TYPE = OFFLINE")); + } + + @Test + public void duplicatePropertyRejected() { + expectThrows(DdlCompilationException.class, () -> compileCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES (" + + " 'replication' = '3', 'replication' = '4')")); + } + + @Test + public void metricRoleRequiresNumericType() { + expectThrows(DdlCompilationException.class, () -> compileCreate( + "CREATE TABLE t (s STRING METRIC) TABLE_TYPE = OFFLINE")); + } + + @Test + public void reservedTableTypePropertyRejected() { + expectThrows(DdlCompilationException.class, () -> compileCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES ('tableType' = 'OFFLINE')")); + } + + @Test + public void streamPropertyOnOfflineTableRejected() { + expectThrows(DdlCompilationException.class, () -> compileCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES (" + + " 'stream.kafka.topic.name' = 'orders')")); + } + + @Test + public void invalidTaskPropertyShapeRejected() { + expectThrows(DdlCompilationException.class, () -> compileCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES ('task.foo' = 'bar')")); + } + + @Test + public void replicationMustBeInteger() { + expectThrows(DdlCompilationException.class, () -> compileCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES ('replication' = 'abc')")); + } + + @Test + public void realtimeWithoutTimeColumnEmitsWarning() { + CompiledCreateTable c = compileCreate( + "CREATE TABLE t (id INT) TABLE_TYPE = REALTIME"); + assertFalse(c.getWarnings().isEmpty()); + assertTrue(c.getWarnings().stream().anyMatch(w -> w.contains("timeColumnName")), + "Expected timeColumnName warning, got: " + c.getWarnings()); + } + + // ------------------------------------------------------------------------------------------- + // DROP TABLE + // ------------------------------------------------------------------------------------------- + + @Test + public void compileDropMinimal() { + CompiledDdl c = DdlCompiler.compile("DROP TABLE events"); + assertEquals(c.getOperation(), DdlOperation.DROP_TABLE); + CompiledDropTable d = (CompiledDropTable) c; + assertEquals(d.getRawTableName(), "events"); + assertFalse(d.isIfExists()); + assertNull(d.getTableType()); + } + + @Test + public void compileDropIfExistsWithType() { + CompiledDropTable d = (CompiledDropTable) DdlCompiler.compile( + "DROP TABLE IF EXISTS analytics.events TYPE OFFLINE"); + assertEquals(d.getDatabaseName(), "analytics"); + assertEquals(d.getRawTableName(), "events"); + assertTrue(d.isIfExists()); + assertEquals(d.getTableType(), TableType.OFFLINE); + } + + // ------------------------------------------------------------------------------------------- + // SHOW TABLES + // ------------------------------------------------------------------------------------------- + + @Test + public void compileShowDefault() { + CompiledShowTables s = (CompiledShowTables) DdlCompiler.compile("SHOW TABLES"); + assertNull(s.getDatabaseName()); + } + + @Test + public void compileShowFromDatabase() { + CompiledShowTables s = (CompiledShowTables) DdlCompiler.compile("SHOW TABLES FROM analytics"); + assertEquals(s.getDatabaseName(), "analytics"); + } + + // ------------------------------------------------------------------------------------------- + // Helpers + // ------------------------------------------------------------------------------------------- + + private static CompiledCreateTable compileCreate(String sql) { + CompiledDdl c = DdlCompiler.compile(sql); + assertEquals(c.getOperation(), DdlOperation.CREATE_TABLE); + return (CompiledCreateTable) c; + } +} diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java new file mode 100644 index 000000000000..e4c755e0d6b4 --- /dev/null +++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java @@ -0,0 +1,309 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.reverse; + +import java.util.Collections; +import org.apache.pinot.spi.config.table.TableConfig; +import org.apache.pinot.spi.config.table.TableType; +import org.apache.pinot.spi.data.DateTimeFieldSpec; +import org.apache.pinot.spi.data.DimensionFieldSpec; +import org.apache.pinot.spi.data.FieldSpec.DataType; +import org.apache.pinot.spi.data.MetricFieldSpec; +import org.apache.pinot.spi.data.Schema; +import org.apache.pinot.spi.utils.builder.TableConfigBuilder; +import org.testng.annotations.Test; + +import static org.testng.Assert.assertEquals; +import static org.testng.Assert.assertFalse; +import static org.testng.Assert.assertTrue; + + +/** + * Golden-output unit tests for {@link CanonicalDdlEmitter}. Every test asserts the exact emitted + * string so the canonical form is locked in: any drift requires updating both the emitter and + * the golden expectation in the same PR. + */ +public class CanonicalDdlEmitterTest { + + @Test + public void minimalOfflineTable() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("id", DataType.INT) + .addSingleValueDimension("name", DataType.STRING) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + .build(); + + String expected = + "CREATE TABLE events (\n" + + " id INT DIMENSION,\n" + + " name STRING DIMENSION\n" + + ")\n" + + "TABLE_TYPE = OFFLINE;\n"; + assertEquals(CanonicalDdlEmitter.emit(schema, config), expected); + } + + @Test + public void allColumnRolesAndDatetime() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("dim", DataType.STRING) + .addMetric("sum", DataType.LONG) + .addDateTime("ts", DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS") + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + .setTimeColumnName("ts") + .build(); + + String emitted = CanonicalDdlEmitter.emit(schema, config); + assertTrue(emitted.contains("dim STRING DIMENSION"), emitted); + assertTrue(emitted.contains("sum LONG METRIC"), emitted); + assertTrue(emitted.contains( + "ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'"), emitted); + assertTrue(emitted.contains("'timeColumnName' = 'ts'"), emitted); + } + + @Test + public void promotedPropertiesEmittedLexicographically() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("id", DataType.INT) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + .setBrokerTenant("tenantA") + .setServerTenant("tenantB") + .setNumReplicas(3) + .setRetentionTimeUnit("DAYS") + .setRetentionTimeValue("30") + .build(); + + String emitted = CanonicalDdlEmitter.emit(schema, config); + int brokerIdx = emitted.indexOf("'brokerTenant'"); + int replicationIdx = emitted.indexOf("'replication'"); + int retentionUnitIdx = emitted.indexOf("'retentionTimeUnit'"); + int serverIdx = emitted.indexOf("'serverTenant'"); + // Lexicographic order: brokerTenant < replication < retentionTimeUnit < serverTenant + assertTrue(brokerIdx < replicationIdx, emitted); + assertTrue(replicationIdx < retentionUnitIdx, emitted); + assertTrue(retentionUnitIdx < serverIdx, emitted); + } + + @Test + public void streamConfigsRoundTripWithOriginalKeys() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("clicks") + .addDateTime("ts", DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS") + .build(); + java.util.Map streamCfgs = new java.util.LinkedHashMap<>(); + streamCfgs.put("stream.kafka.topic.name", "click_events"); + streamCfgs.put("stream.kafka.consumer.factory.class.name", "KafkaConsumerFactory"); + streamCfgs.put("realtime.segment.flush.threshold.rows", "500000"); + TableConfig config = new TableConfigBuilder(TableType.REALTIME) + .setTableName("clicks") + .setTimeColumnName("ts") + .setStreamConfigs(streamCfgs) + .build(); + + String emitted = CanonicalDdlEmitter.emit(schema, config); + assertTrue(emitted.contains("'stream.kafka.topic.name' = 'click_events'"), emitted); + assertTrue(emitted.contains( + "'stream.kafka.consumer.factory.class.name' = 'KafkaConsumerFactory'"), emitted); + assertTrue(emitted.contains("'realtime.segment.flush.threshold.rows' = '500000'"), emitted); + } + + @Test + public void noDefaultsEmittedWhenAtNaturalDefault() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("id", DataType.INT) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + // replication defaults to "1" — should NOT appear in canonical output + .build(); + + String emitted = CanonicalDdlEmitter.emit(schema, config); + assertFalse(emitted.contains("'replication'"), emitted); + assertFalse(emitted.contains("'loadMode'"), emitted); + } + + @Test + public void notNullAndDefaultEmittedExplicitly() { + Schema schema = new Schema(); + schema.setSchemaName("t"); + DimensionFieldSpec dim = new DimensionFieldSpec("name", DataType.STRING, true, "N/A"); + dim.setNotNull(true); + schema.addField(dim); + MetricFieldSpec metric = new MetricFieldSpec("score", DataType.DOUBLE, 0.0); + schema.addField(metric); + + TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build(); + String emitted = CanonicalDdlEmitter.emit(schema, config); + assertTrue(emitted.contains("name STRING NOT NULL DEFAULT 'N/A' DIMENSION"), emitted); + // 0.0 IS the metric default for DOUBLE so DEFAULT clause should be elided + assertTrue(emitted.contains("score DOUBLE METRIC"), emitted); + assertFalse(emitted.contains("DEFAULT 0"), emitted); + } + + @Test + public void databaseQualifiedNameRendered() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("id", DataType.INT) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("analytics.events") + .build(); + + String emitted = CanonicalDdlEmitter.emit(schema, config, "analytics"); + assertTrue(emitted.startsWith("CREATE TABLE analytics.events ("), emitted); + } + + @Test + public void identifiersWithReservedNamesQuoted() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("orders") + // METRIC is a Pinot DDL keyword; the column name must be quoted on emit so it round-trips. + .addSingleValueDimension("metric", DataType.STRING) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("orders").build(); + String emitted = CanonicalDdlEmitter.emit(schema, config); + assertTrue(emitted.contains("\"metric\" STRING DIMENSION"), emitted); + } + + @Test + public void taskConfigsEmittedAsPrefixedKeys() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("id", DataType.INT) + .build(); + java.util.Map> tasks = new java.util.LinkedHashMap<>(); + java.util.Map rtoCfg = new java.util.LinkedHashMap<>(); + rtoCfg.put("bucketTimePeriod", "1d"); + rtoCfg.put("maxNumRecordsPerSegment", "5000000"); + tasks.put("RealtimeToOfflineSegmentsTask", rtoCfg); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + .setTaskConfig(new org.apache.pinot.spi.config.table.TableTaskConfig(tasks)) + .build(); + + String emitted = CanonicalDdlEmitter.emit(schema, config); + assertTrue(emitted.contains( + "'task.RealtimeToOfflineSegmentsTask.bucketTimePeriod' = '1d'"), emitted); + assertTrue(emitted.contains( + "'task.RealtimeToOfflineSegmentsTask.maxNumRecordsPerSegment' = '5000000'"), emitted); + } + + @Test + public void customConfigEmittedVerbatim() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("t") + .addSingleValueDimension("id", DataType.INT) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("t") + .setCustomConfig(new org.apache.pinot.spi.config.table.TableCustomConfig( + Collections.singletonMap("mySpecialKey", "someValue"))) + .build(); + + String emitted = CanonicalDdlEmitter.emit(schema, config); + assertTrue(emitted.contains("'mySpecialKey' = 'someValue'"), emitted); + } + + @Test + public void datetimeFieldRoundTripsFormatAndGranularity() { + Schema schema = new Schema(); + schema.setSchemaName("t"); + schema.addField(new DateTimeFieldSpec("ts", DataType.LONG, "1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMdd", + "1:DAYS")); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build(); + String emitted = CanonicalDdlEmitter.emit(schema, config); + assertTrue(emitted.contains( + "ts LONG DATETIME FORMAT '1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMdd' GRANULARITY '1:DAYS'"), + emitted); + } + + @Test + public void customConfigKeyShadowingPromotedKeyRejected() { + // Regression: TableCustomConfig accepts arbitrary string keys, but when one collides with + // a promoted/JSON-blob key the canonical DDL would emit it indistinguishably from the + // promoted property — the round-trip would either silently misroute the value or fail + // JSON deserialization. Reject up front so the user renames the colliding key. + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("t") + .addSingleValueDimension("id", DataType.INT) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("t") + .setCustomConfig(new org.apache.pinot.spi.config.table.TableCustomConfig( + Collections.singletonMap("ingestionConfig", "anything"))) + .build(); + try { + CanonicalDdlEmitter.emit(schema, config); + org.testng.Assert.fail("Expected IllegalArgumentException for shadowing custom-config key"); + } catch (IllegalArgumentException expected) { + assertTrue(expected.getMessage().contains("ingestionConfig"), expected.getMessage()); + } + } + + @Test + public void customConfigKeyShadowingTaskPrefixRejected() { + // Same root-cause as above but for the prefix-routed paths: task.., stream.*, + // and realtime.* are all consumed by PropertyMapping's prefix routing on re-parse, so + // a TableCustomConfig entry under any of those prefixes would silently land in the wrong + // TableConfig field. Reject so the user knows to rename. + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("t") + .addSingleValueDimension("id", DataType.INT) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("t") + .setCustomConfig(new org.apache.pinot.spi.config.table.TableCustomConfig( + Collections.singletonMap("task.MyTask.foo", "bar"))) + .build(); + try { + CanonicalDdlEmitter.emit(schema, config); + org.testng.Assert.fail("Expected IllegalArgumentException for shadowing task.* key"); + } catch (IllegalArgumentException expected) { + assertTrue(expected.getMessage().contains("task.MyTask.foo"), expected.getMessage()); + } + } + + @Test + public void deterministicOutputAcrossRepeatedCalls() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("a", DataType.INT) + .addSingleValueDimension("b", DataType.STRING) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + .setBrokerTenant("t1") + .setServerTenant("t2") + .setNumReplicas(2) + .build(); + String first = CanonicalDdlEmitter.emit(schema, config); + String second = CanonicalDdlEmitter.emit(schema, config); + assertEquals(first, second, "Canonical emit must be deterministic"); + } +} diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java new file mode 100644 index 000000000000..586ad59f64eb --- /dev/null +++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java @@ -0,0 +1,355 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.sql.ddl.roundtrip; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.node.ObjectNode; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.Map; +import org.apache.pinot.spi.config.table.TableConfig; +import org.apache.pinot.spi.config.table.TableCustomConfig; +import org.apache.pinot.spi.config.table.TableType; +import org.apache.pinot.spi.config.table.ingestion.BatchIngestionConfig; +import org.apache.pinot.spi.config.table.ingestion.IngestionConfig; +import org.apache.pinot.spi.data.DimensionFieldSpec; +import org.apache.pinot.spi.data.FieldSpec.DataType; +import org.apache.pinot.spi.data.MetricFieldSpec; +import org.apache.pinot.spi.data.Schema; +import org.apache.pinot.spi.utils.JsonUtils; +import org.apache.pinot.spi.utils.builder.TableConfigBuilder; +import org.apache.pinot.sql.ddl.compile.CompiledCreateTable; +import org.apache.pinot.sql.ddl.compile.DdlCompiler; +import org.apache.pinot.sql.ddl.reverse.CanonicalDdlEmitter; +import org.testng.annotations.Test; + +import static org.testng.Assert.assertEquals; +import static org.testng.Assert.assertNotNull; + + +/** + * Round-trip suite: original (Schema, TableConfig) → canonical DDL → re-parse → re-compile → + * round-tripped (Schema, TableConfig). Each test asserts the round-tripped pair is semantically + * equivalent to the original — same Schema (column shape, datetime format, primary keys) and + * same TableConfig fields. + * + *

Semantic equivalence is computed by comparing JSON serializations rather than direct + * .equals(): TableConfig and Schema do not implement equals() reliably across all nested + * configs, but their JSON representations are what eventually persist to ZK and what callers + * actually compare. + * + *

Fixtures here are synthetic so the test is hermetic and does not depend on examples in + * other modules. The set deliberately exercises every routing rule + * ({@code stream.*}, {@code task.*}, JSON blob, custom config, promoted scalar, CSV list). + */ +public class RoundTripTest { + + // ------------------------------------------------------------------------------------------- + // Round-trip cases + // ------------------------------------------------------------------------------------------- + + @Test + public void minimalOfflineTable() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("id", DataType.INT) + .addSingleValueDimension("name", DataType.STRING) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + .build(); + assertRoundTrip(schema, config); + } + + @Test + public void offlineTableWithRetentionAndTenants() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("id", DataType.INT) + .addMetric("score", DataType.DOUBLE) + .addDateTime("ts", DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS") + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + .setTimeColumnName("ts") + .setRetentionTimeUnit("DAYS") + .setRetentionTimeValue("30") + .setNumReplicas(3) + .setBrokerTenant("tenantA") + .setServerTenant("tenantB") + .build(); + assertRoundTrip(schema, config); + } + + @Test + public void offlineTableWithIndexingConfig() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("country", DataType.STRING) + .addSingleValueDimension("city", DataType.STRING) + .addMetric("amount", DataType.DOUBLE) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + .setSortedColumn("country") + .setInvertedIndexColumns(java.util.Arrays.asList("city")) + .setNoDictionaryColumns(java.util.Arrays.asList("amount")) + .setBloomFilterColumns(java.util.Arrays.asList("country")) + .setNullHandlingEnabled(true) + .build(); + assertRoundTrip(schema, config); + } + + @Test + public void realtimeTableWithStreamConfigs() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("clicks") + .addSingleValueDimension("user_id", DataType.STRING) + .addDateTime("ts", DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS") + .build(); + Map streamCfgs = new LinkedHashMap<>(); + streamCfgs.put("stream.kafka.topic.name", "click_events"); + streamCfgs.put("stream.kafka.consumer.factory.class.name", "KafkaConsumerFactory"); + streamCfgs.put("realtime.segment.flush.threshold.rows", "500000"); + TableConfig config = new TableConfigBuilder(TableType.REALTIME) + .setTableName("clicks") + .setTimeColumnName("ts") + .setNumReplicas(2) + .setStreamConfigs(streamCfgs) + .build(); + assertRoundTrip(schema, config); + } + + @Test + public void offlineTableWithTaskConfig() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("id", DataType.INT) + .build(); + Map> tasks = new LinkedHashMap<>(); + Map rto = new LinkedHashMap<>(); + rto.put("bucketTimePeriod", "1d"); + rto.put("maxNumRecordsPerSegment", "5000000"); + tasks.put("RealtimeToOfflineSegmentsTask", rto); + Map refresh = new LinkedHashMap<>(); + refresh.put("tableMaxNumTasks", "5"); + tasks.put("SegmentRefreshTask", refresh); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + .setTaskConfig(new org.apache.pinot.spi.config.table.TableTaskConfig(tasks)) + .build(); + assertRoundTrip(schema, config); + } + + @Test + public void offlineTableWithCustomConfig() { + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("t") + .addSingleValueDimension("id", DataType.INT) + .build(); + Map custom = new LinkedHashMap<>(); + custom.put("ourTeam.flag", "true"); + custom.put("internal.config.X", "value-x"); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("t") + .setCustomConfig(new TableCustomConfig(custom)) + .build(); + assertRoundTrip(schema, config); + } + + @Test + public void offlineTableWithIngestionConfigJsonBlob() { + // Ingestion config is a complex nested type with no first-class DDL clause; it round-trips + // through PROPERTIES('ingestionConfig' = ''). This proves the JSON-blob fallback + // preserves structurally-rich configs without silent loss. + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("id", DataType.INT) + .addDateTime("ts", DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS") + .build(); + BatchIngestionConfig batch = new BatchIngestionConfig(null, "APPEND", "DAILY"); + IngestionConfig ingestion = new IngestionConfig(); + ingestion.setBatchIngestionConfig(batch); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + .setTimeColumnName("ts") + .setIngestionConfig(ingestion) + .build(); + assertRoundTrip(schema, config); + } + + @Test + public void columnDefaultsRoundTrip() { + Schema schema = new Schema(); + schema.setSchemaName("t"); + DimensionFieldSpec name = new DimensionFieldSpec("name", DataType.STRING, true, "N/A"); + name.setNotNull(true); + schema.addField(name); + MetricFieldSpec score = new MetricFieldSpec("score", DataType.DOUBLE, 42.0); + schema.addField(score); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build(); + assertRoundTrip(schema, config); + } + + @Test + public void identifiersWithReservedNamesRoundTrip() { + // Column named "metric" requires quoting on emit; round-trip must preserve the name. + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("orders") + .addSingleValueDimension("metric", DataType.STRING) + .addSingleValueDimension("dimension", DataType.STRING) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("orders").build(); + assertRoundTrip(schema, config); + } + + @Test + public void columnsNamedAfterDataTypeKeywordsRoundTrip() { + // INT, STRING, BOOLEAN, etc. are data-type tokens consumed by the column-declaration + // grammar; column names that match them must be quoted on emission so the re-parse treats + // them as identifiers, not type tokens. + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("t") + .addSingleValueDimension("int", DataType.INT) + .addSingleValueDimension("string", DataType.STRING) + .addSingleValueDimension("boolean", DataType.BOOLEAN) + .addMetric("double", DataType.DOUBLE) + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build(); + assertRoundTrip(schema, config); + } + + @Test + public void promotedScalarsAddedInSlice2RoundTrip() { + // Regression for PropertyExtractor/PropertyMapping symmetry: every key the emitter writes + // must have a matching forward-direction handler. This test exercises every promoted scalar + // added in Slice 2 so any future extractor addition without a matching handler fails here. + Schema schema = new Schema.SchemaBuilder() + .setSchemaName("events") + .addSingleValueDimension("id", DataType.INT) + .addSingleValueDimension("country", DataType.STRING) + .addSingleValueDimension("city", DataType.STRING) + .addMetric("amount", DataType.DOUBLE) + .addDateTime("ts", DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS") + .build(); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE) + .setTableName("events") + .setTimeColumnName("ts") + .setRetentionTimeUnit("DAYS") + .setRetentionTimeValue("60") + .setNumReplicas(2) + .setBrokerTenant("tenantA") + .setServerTenant("tenantB") + .setSortedColumn("country") + .setInvertedIndexColumns(java.util.Arrays.asList("city")) + .setNoDictionaryColumns(java.util.Arrays.asList("amount")) + .setOnHeapDictionaryColumns(java.util.Arrays.asList("country")) + .setVarLengthDictionaryColumns(java.util.Arrays.asList("city")) + .setBloomFilterColumns(java.util.Arrays.asList("country")) + .setRangeIndexColumns(java.util.Arrays.asList("amount")) + .setNullHandlingEnabled(true) + .setAggregateMetrics(true) + .setPeerSegmentDownloadScheme("https") + .setCrypterClassName("org.apache.pinot.crypter.NoOpCrypter") + .setSegmentVersion("v3") + .setDeletedSegmentsRetentionPeriod("14d") + .setDescription("a kitchen-sink test table") + .setTags(java.util.Arrays.asList("ourTeam", "metricsPipeline")) + .build(); + assertRoundTrip(schema, config); + } + + @Test + public void multiValueDimensionRoundTrips() { + // MV dimensions must survive SHOW CREATE TABLE: SchemaEmitter must emit DIMENSION ARRAY and + // the compiler must recreate the MV field spec so isSingleValue=false is preserved. + Schema schema = new Schema(); + schema.setSchemaName("t"); + schema.addField(new DimensionFieldSpec("tags", DataType.STRING, false)); + schema.addField(new DimensionFieldSpec("id", DataType.INT, true)); + TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build(); + assertRoundTrip(schema, config); + } + + // ------------------------------------------------------------------------------------------- + // Equivalence machinery + // ------------------------------------------------------------------------------------------- + + /** Asserts that emit -> parse -> compile produces a semantically equivalent (schema, config). */ + private static void assertRoundTrip(Schema originalSchema, TableConfig originalConfig) { + String ddl = CanonicalDdlEmitter.emit(originalSchema, originalConfig); + CompiledCreateTable round = (CompiledCreateTable) DdlCompiler.compile(ddl); + assertNotNull(round, "Round-tripped DDL should compile: " + ddl); + assertSchemaEquivalent(originalSchema, round.getSchema(), ddl); + assertTableConfigEquivalent(originalConfig, round.getTableConfig(), ddl); + + // Idempotency: emit-parse-emit should yield the same canonical text. + String secondEmit = CanonicalDdlEmitter.emit(round.getSchema(), round.getTableConfig()); + assertEquals(secondEmit, ddl, "Canonical DDL must be idempotent across round-trip:\n" + ddl); + } + + private static void assertSchemaEquivalent(Schema a, Schema b, String ddl) { + JsonNode aJson = stripVolatile(JsonUtils.objectToJsonNode(a)); + JsonNode bJson = stripVolatile(JsonUtils.objectToJsonNode(b)); + assertEquals(bJson, aJson, "Schema diverged on round-trip.\nDDL was:\n" + ddl + + "\nExpected: " + aJson + "\nActual: " + bJson); + } + + private static void assertTableConfigEquivalent(TableConfig a, TableConfig b, String ddl) { + JsonNode aJson = stripVolatile(JsonUtils.objectToJsonNode(a)); + JsonNode bJson = stripVolatile(JsonUtils.objectToJsonNode(b)); + assertEquals(bJson, aJson, "TableConfig diverged on round-trip.\nDDL was:\n" + ddl + + "\nExpected: " + aJson + "\nActual: " + bJson); + } + + /** + * Removes fields that are not meaningful for semantic comparison. Empty maps in TableCustomConfig + * compare-equal whether the field is null, missing, or {}, so we strip them. Same for + * empty lists added by builders that are not user-meaningful. + */ + private static JsonNode stripVolatile(JsonNode node) { + if (node == null || !node.isObject()) { + return node; + } + ObjectNode obj = (ObjectNode) node; + Iterator> it = obj.fields(); + while (it.hasNext()) { + Map.Entry e = it.next(); + JsonNode v = e.getValue(); + if (v.isNull()) { + it.remove(); + } else if (v.isObject() && v.size() == 0) { + it.remove(); + } else if (v.isArray() && v.size() == 0) { + it.remove(); + } else if (v.isObject()) { + stripVolatile(v); + } + } + // Re-check for now-empty objects after recursion + Iterator> it2 = obj.fields(); + while (it2.hasNext()) { + Map.Entry e = it2.next(); + if (e.getValue().isObject() && e.getValue().size() == 0) { + it2.remove(); + } + } + return obj; + } +} diff --git a/pom.xml b/pom.xml index 3088e6e7c70b..054a2d6e72e2 100644 --- a/pom.xml +++ b/pom.xml @@ -61,6 +61,7 @@ pinot-query-runtime pinot-timeseries pinot-udf-test + pinot-sql-ddl @@ -696,6 +697,11 @@ pinot-timeseries-planner ${project.version} + + org.apache.pinot + pinot-sql-ddl + ${project.version} + org.apache.pinot pinot-timeseries-m3ql From c580bd009e4e086ca9c8e0b897f595b85c006ae5 Mon Sep 17 00:00:00 2001 From: Xiang Fu Date: Fri, 17 Apr 2026 21:12:01 -0700 Subject: [PATCH 02/32] Fix five DDL issues from adversarial + standard code reviews - CREATE concurrent race: skip schema rollback when TableAlreadyExistsException is caught but the table now exists (another caller won the race); previously this unconditionally deleted the shared schema leaving the winner's table without its schema. - DROP TABLE: add logical-table reference check (matching /tables DELETE), run tableTasksCleanup before each variant deletion to remove task schedules, and delete the shared schema after the last physical variant is dropped so stale schema metadata does not block future CREATE TABLE for the same name. - SHOW TABLES: use TargetType.CLUSTER + Actions.Cluster.GET_TABLE instead of the table-scoped ResourceUtils.checkPermissionAndAccess, matching the auth model of the existing GET /tables endpoint and avoiding 403 for callers who have cluster-level listing access but not a per-table READ on a resource named after the database. - sortedColumn multi-column restore: PropertyMapping.apply() now returns the full sorted-column list; DdlCompiler applies it via IndexingConfig.setSortedColumn(List) after builder.build() so multi-column sort configs survive a SHOW CREATE / replay round-trip instead of being truncated to the first column. - SHOW CREATE TABLE correctness: - Emit PRIMARY KEY (...) clause when schema.primaryKeyColumns is set so upsert/dedup/dimension tables round-trip without losing key metadata. - Emit legacy timeFieldSpec columns as DATETIME so they are not silently dropped on replay. - Fail fast with IllegalArgumentException for MAP/LIST/STRUCT columns whose types have no DDL representation, rather than emitting lossy DDL. Co-Authored-By: Claude Sonnet 4.6 --- .../resources/PinotDdlRestletResource.java | 73 ++++++++++++++++--- .../pinot/sql/ddl/compile/DdlCompiler.java | 10 ++- .../sql/ddl/compile/PropertyMapping.java | 30 ++++++-- .../sql/ddl/reverse/CanonicalDdlEmitter.java | 16 +++- .../pinot/sql/ddl/reverse/SchemaEmitter.java | 42 +++++++++++ 5 files changed, 151 insertions(+), 20 deletions(-) diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java index 19785928b374..901311b57ba6 100644 --- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java +++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java @@ -43,21 +43,28 @@ import javax.ws.rs.core.Response; import org.apache.commons.lang3.StringUtils; import org.apache.pinot.common.exception.SchemaAlreadyExistsException; +import org.apache.pinot.common.metadata.ZKMetadataProvider; import org.apache.pinot.common.utils.DatabaseUtils; +import org.apache.pinot.common.utils.LogicalTableConfigUtils; +import org.apache.pinot.controller.api.access.AccessControl; import org.apache.pinot.controller.api.access.AccessControlFactory; +import org.apache.pinot.controller.api.access.AccessControlUtils; import org.apache.pinot.controller.api.access.AccessType; import org.apache.pinot.controller.api.exception.ControllerApplicationException; import org.apache.pinot.controller.api.exception.TableAlreadyExistsException; import org.apache.pinot.controller.api.resources.ddl.DdlExecutionRequest; import org.apache.pinot.controller.api.resources.ddl.DdlExecutionResponse; import org.apache.pinot.controller.helix.core.PinotHelixResourceManager; +import org.apache.pinot.controller.helix.core.minion.PinotHelixTaskResourceManager; import org.apache.pinot.controller.helix.core.minion.PinotTaskManager; import org.apache.pinot.controller.util.TaskConfigUtils; import org.apache.pinot.core.auth.Actions; import org.apache.pinot.core.auth.ManualAuthorization; +import org.apache.pinot.core.auth.TargetType; import org.apache.pinot.segment.local.utils.TableConfigUtils; import org.apache.pinot.spi.config.table.TableConfigValidatorRegistry; import org.apache.pinot.spi.config.table.TableType; +import org.apache.pinot.spi.data.LogicalTableConfig; import org.apache.pinot.spi.data.Schema; import org.apache.pinot.spi.utils.CommonConstants; import org.apache.pinot.spi.utils.JsonUtils; @@ -115,6 +122,9 @@ public class PinotDdlRestletResource { @Inject PinotTaskManager _pinotTaskManager; + @Inject + PinotHelixTaskResourceManager _pinotHelixTaskResourceManager; + @Inject AccessControlFactory _accessControlFactory; @@ -278,9 +288,14 @@ private DdlExecutionResponse executeCreate(CompiledCreateTable create, String da LOGGER.info("DDL created table {}", tableNameWithType); return response; } catch (TableAlreadyExistsException e) { - // Race: another caller added the table between our hasTable check and addTable. Map to - // 409 Conflict so the failure mode is consistent with the pre-check path. - rollbackSchemaIfCreated(schemaName, schemaCreatedHere); + // Race: another caller added the table between our hasTable check and addTable. + // Only roll back the schema we created here if no table for this raw name now exists — + // if another caller won the race and its table is live, removing the schema would + // orphan that table. Re-check existence with the winner's table still present before + // deciding to clean up. + if (schemaCreatedHere && !_pinotHelixResourceManager.hasTable(tableNameWithType)) { + rollbackSchemaIfCreated(schemaName, true); + } throw new ControllerApplicationException(LOGGER, e.getMessage(), Response.Status.CONFLICT, e); } catch (SchemaAlreadyExistsException e) { // The override=false addSchema call lost a race with another schema writer. Surface 409. @@ -397,14 +412,48 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database return response; } + // Reject drop if any target is referenced by a logical table, matching the safeguard in + // the existing /tables and /tableConfigs DELETE endpoints. + List allLogicalTableConfigs = + ZKMetadataProvider.getAllLogicalTableConfigs(_pinotHelixResourceManager.getPropertyStore()); + for (String target : targets) { + for (LogicalTableConfig logicalTableConfig : allLogicalTableConfigs) { + if (LogicalTableConfigUtils.checkPhysicalTableRefExists(logicalTableConfig, target)) { + throw new ControllerApplicationException(LOGGER, + "Cannot drop table '" + target + "': it is referenced by logical table '" + + logicalTableConfig.getTableName() + "'.", + Response.Status.CONFLICT); + } + } + } + try { for (String target : targets) { + // Remove task schedules before deletion so tasks are not triggered during the drop. + PinotTableRestletResource.tableTasksCleanup(target, false, + _pinotHelixResourceManager, _pinotHelixTaskResourceManager); TableType type = TableNameBuilder.getTableTypeFromTableName(target); _pinotHelixResourceManager.deleteTable(fullyQualifiedRaw, type, null); } + // Delete the shared schema when the last physical variant has been removed. A schema + // without any table leaves stale metadata that blocks future CREATE TABLE for the same + // raw name with a different column list. + boolean offlineExists = _pinotHelixResourceManager.hasOfflineTable(fullyQualifiedRaw); + boolean realtimeExists = _pinotHelixResourceManager.hasRealtimeTable(fullyQualifiedRaw); + if (!offlineExists && !realtimeExists) { + try { + _pinotHelixResourceManager.deleteSchema(fullyQualifiedRaw); + LOGGER.info("DDL deleted schema {} after dropping last table variant", fullyQualifiedRaw); + } catch (Exception schemaEx) { + LOGGER.warn("Failed to delete schema {} after DROP TABLE; manual cleanup may be required", + fullyQualifiedRaw, schemaEx); + } + } response.setMessage("Dropped " + targets.size() + " table(s)."); LOGGER.info("DDL dropped tables {}", targets); return response; + } catch (ControllerApplicationException e) { + throw e; } catch (Exception e) { // ControllerApplicationException(LOGGER, ...) logs the exception itself; don't double-log. throw new ControllerApplicationException(LOGGER, @@ -498,12 +547,18 @@ private DdlExecutionResponse executeShow(String database, HttpHeaders headers, R // databases the caller may not have access to. The database resolution chain (SQL FROM // clause -> Database header -> DEFAULT_DATABASE) ensures we always have an explicit scope. String scopedDatabase = database == null ? CommonConstants.DEFAULT_DATABASE : database; - // Use the cluster-scoped GetTable action that the existing GET /tables endpoint uses. We - // pass the database name as the resource handle so AccessControl impls that key on - // database scope can still authorise per-database, but the action itself is cluster-level - // (matching the listing semantics) rather than table-level. - ResourceUtils.checkPermissionAndAccess(scopedDatabase, httpRequest, headers, - AccessType.READ, Actions.Cluster.GET_TABLE, _accessControlFactory, LOGGER); + // SHOW TABLES is a cluster-level listing operation, not a per-table read. Use the same + // TargetType.CLUSTER + GET_TABLE action pair that the existing @Authorize-annotated + // GET /tables endpoint uses, so secured deployments grant SHOW TABLES to callers who + // already have cluster-level table-listing access rather than requiring a fictitious + // per-table READ on a resource named after the database. + String endpointUrl = httpRequest.getRequestURL().toString(); + AccessControl accessControl = _accessControlFactory.create(); + AccessControlUtils.validatePermission(null, AccessType.READ, headers, endpointUrl, accessControl); + if (!accessControl.hasAccess(headers, TargetType.CLUSTER, scopedDatabase, + Actions.Cluster.GET_TABLE)) { + throw new ControllerApplicationException(LOGGER, "Permission denied", Response.Status.FORBIDDEN); + } List tables = _pinotHelixResourceManager.getAllRawTables(scopedDatabase); return new DdlExecutionResponse() .setOperation(DdlOperation.SHOW_TABLES) diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java index 149b77352584..1fa2efa3b37e 100644 --- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java @@ -292,8 +292,14 @@ private static TableConfig buildTableConfig(ResolvedTableDefinition resolved, Li : resolved.getDatabaseName() + "." + resolved.getRawTableName(); TableConfigBuilder builder = new TableConfigBuilder(resolved.getTableType()) .setTableName(tableNameForConfig); - PropertyMapping.apply(resolved, builder); - return builder.build(); + List sortedColumns = PropertyMapping.apply(resolved, builder); + TableConfig tableConfig = builder.build(); + // Apply the full sorted-column list if more than one column was specified; the builder's + // setSortedColumn(String) wraps in singletonList and loses the remaining columns. + if (sortedColumns != null && sortedColumns.size() > 1) { + tableConfig.getIndexingConfig().setSortedColumn(sortedColumns); + } + return tableConfig; } /** diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java index 593b0b9a66c1..f4d0f60d8d61 100644 --- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java @@ -33,6 +33,7 @@ import org.apache.pinot.spi.config.table.DedupConfig; import org.apache.pinot.spi.config.table.DimensionTableConfig; import org.apache.pinot.spi.config.table.FieldConfig; +import org.apache.pinot.spi.config.table.IndexingConfig; import org.apache.pinot.spi.config.table.JsonIndexConfig; import org.apache.pinot.spi.config.table.MultiColumnTextIndexConfig; import org.apache.pinot.spi.config.table.QueryConfig; @@ -171,13 +172,19 @@ private PropertyMapping() { /** * Applies all properties from {@code definition} onto {@code builder}. * + *

Returns the full list of sorted columns parsed from the {@code sortedColumn} property so + * the caller can apply them directly to {@link IndexingConfig#setSortedColumn(List)} after + * {@code builder.build()}. The builder's {@code setSortedColumn(String)} only stores a single + * value; using the returned list avoids silently dropping all but the first sort column. + * * @throws DdlCompilationException if a promoted key has a non-coercible value (e.g. non-integer * replication). */ - public static void apply(ResolvedTableDefinition definition, TableConfigBuilder builder) { + public static List apply(ResolvedTableDefinition definition, TableConfigBuilder builder) { Map streamConfigs = new LinkedHashMap<>(); Map> taskConfigs = new LinkedHashMap<>(); Map customConfigs = new LinkedHashMap<>(); + List sortedColumns = null; for (Map.Entry entry : definition.getProperties().entrySet()) { String rawKey = entry.getKey(); @@ -192,6 +199,17 @@ public static void apply(ResolvedTableDefinition definition, TableConfigBuilder + "not PROPERTIES."); } + if (lower.equals("sortedcolumn")) { + // Capture the full list so the caller can call IndexingConfig.setSortedColumn(List) + // after build(). Also set the first element via the builder so single-column sort + // configs still work when the caller ignores the returned list. + sortedColumns = splitCsv(value); + if (!sortedColumns.isEmpty()) { + builder.setSortedColumn(sortedColumns.get(0)); + } + continue; + } + if (applyPromoted(lower, value, builder)) { continue; } @@ -245,6 +263,7 @@ public static void apply(ResolvedTableDefinition definition, TableConfigBuilder if (!customConfigs.isEmpty()) { builder.setCustomConfig(new TableCustomConfig(customConfigs)); } + return sortedColumns; } /** Returns true if {@code lowerKey} matched a promoted property and was applied. */ @@ -276,13 +295,8 @@ private static boolean applyPromoted(String lowerKey, String value, TableConfigB builder.setLoadMode(value); return true; case "sortedcolumn": - // The canonical DDL emits all sort columns as CSV but TableConfigBuilder.setSortedColumn - // only accepts a single string. Use the first element so the primary sort column is - // preserved; multi-column sort configs are intentionally deferred to Slice 4. - List sortCols = splitCsv(value); - if (!sortCols.isEmpty()) { - builder.setSortedColumn(sortCols.get(0)); - } + // Handled above the applyPromoted call so the full column list is captured. + // This branch is dead code but kept to satisfy the switch exhaustiveness check. return true; case "nullhandlingenabled": builder.setNullHandlingEnabled(parseBool(lowerKey, value)); diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java index 741861a8dc58..abc5ec2a01fe 100644 --- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java @@ -91,13 +91,27 @@ public static String emit(Schema schema, TableConfig config, @Nullable String da sb.append(SqlIdentifiers.quote(displayName)); sb.append(" (\n"); List columns = SchemaEmitter.emitColumns(schema); + List primaryKeyColumns = schema.getPrimaryKeyColumns(); + boolean hasPrimaryKey = primaryKeyColumns != null && !primaryKeyColumns.isEmpty(); + // Column count for trailing-comma logic: columns + optional PRIMARY KEY line. + int totalEntries = columns.size() + (hasPrimaryKey ? 1 : 0); for (int i = 0; i < columns.size(); i++) { sb.append(INDENT).append(columns.get(i)); - if (i < columns.size() - 1) { + if (i < totalEntries - 1) { sb.append(','); } sb.append('\n'); } + if (hasPrimaryKey) { + sb.append(INDENT).append("PRIMARY KEY ("); + for (int i = 0; i < primaryKeyColumns.size(); i++) { + sb.append(SqlIdentifiers.quote(primaryKeyColumns.get(i))); + if (i < primaryKeyColumns.size() - 1) { + sb.append(", "); + } + } + sb.append(")\n"); + } sb.append(")\n"); sb.append("TABLE_TYPE = ").append(emitTableType(config.getTableType())).append('\n'); diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java index 4aacdef7c6bb..86401fa38c0f 100644 --- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java +++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java @@ -26,6 +26,7 @@ import org.apache.pinot.spi.data.FieldSpec.DataType; import org.apache.pinot.spi.data.MetricFieldSpec; import org.apache.pinot.spi.data.Schema; +import org.apache.pinot.spi.data.TimeFieldSpec; /** @@ -41,20 +42,61 @@ final class SchemaEmitter { private SchemaEmitter() { } + /** + * Returns canonical column declarations for all fields in {@code schema}. + * + *

Fails fast with {@link IllegalArgumentException} for MAP/LIST/STRUCT columns because those + * types have no DDL representation yet; emitting incomplete DDL for them would produce a + * statement that silently drops part of the schema on replay. + */ static List emitColumns(Schema schema) { List out = new ArrayList<>(); for (DimensionFieldSpec dim : schema.getDimensionFieldSpecs()) { + validateEmittable(dim); out.add(emitColumn(dim)); } for (MetricFieldSpec metric : schema.getMetricFieldSpecs()) { + validateEmittable(metric); out.add(emitColumn(metric)); } for (DateTimeFieldSpec dt : schema.getDateTimeFieldSpecs()) { out.add(emitColumn(dt)); } + // Legacy time field: emit as a DATETIME column so the column is not silently dropped. + TimeFieldSpec timeFieldSpec = schema.getTimeFieldSpec(); + if (timeFieldSpec != null) { + out.add(emitTimeColumn(timeFieldSpec)); + } return out; } + private static void validateEmittable(FieldSpec spec) { + DataType dt = spec.getDataType(); + if (dt == DataType.MAP || dt == DataType.LIST || dt == DataType.STRUCT + || dt == DataType.UNKNOWN) { + throw new IllegalArgumentException( + "SHOW CREATE TABLE cannot represent column '" + spec.getName() + "' of type " + dt + + " in DDL; replay of the emitted DDL would silently drop this column. " + + "The DDL grammar does not yet support complex types."); + } + } + + private static String emitTimeColumn(TimeFieldSpec spec) { + // Emit the legacy time column as a DATETIME column so it survives a round-trip. + // Use the outgoing granularity spec to derive format and granularity strings matching + // the DateTimeFieldSpec convention ({size}:{unit}:{format}). + org.apache.pinot.spi.data.TimeGranularitySpec tgs = spec.getOutgoingGranularitySpec(); + String format = tgs.getTimeUnitSize() + ":" + tgs.getTimeType().name() + ":" + + tgs.getTimeFormat(); + String granularity = "1:" + tgs.getTimeType().name(); + StringBuilder sb = new StringBuilder(); + sb.append(SqlIdentifiers.quote(spec.getName())); + sb.append(' ').append(emitDataType(spec.getDataType())); + sb.append(" DATETIME FORMAT ").append(SqlIdentifiers.quoteString(format)); + sb.append(" GRANULARITY ").append(SqlIdentifiers.quoteString(granularity)); + return sb.toString(); + } + private static String emitColumn(FieldSpec spec) { StringBuilder sb = new StringBuilder(); sb.append(SqlIdentifiers.quote(spec.getName())); From c7a493200fdf3c85b230b44c4d6a6d696405640f Mon Sep 17 00:00:00 2001 From: Xiang Fu Date: Sat, 18 Apr 2026 00:26:59 -0700 Subject: [PATCH 03/32] Fix three DDL issues from second code review + add missing tests - PRIMARY KEY: add grammar production, AST field, compiler extraction, and emitter (outside column list, before TABLE_TYPE) so SHOW CREATE TABLE output re-parses correctly - sortedColumn multi-column: PropertyMapping.apply() now returns the full sorted-column list; DdlCompiler applies it via IndexingConfig.setSortedColumn(List) after builder.build() - TimeFieldSpec granularity: SchemaEmitter.emitTimeColumn() uses tgs.getTimeUnitSize() instead of hardcoded '1' so a 15-MINUTE spec emits '15:MINUTES' not '1:MINUTES' - Add PRIMARY KEY parser tests (3), compiler tests (3), and round-trip tests (3) covering single-key, composite-key, absent-key, granularity correctness, and datetime format preservation Co-Authored-By: Claude Sonnet 4.6 --- .../src/main/codegen/includes/parserImpls.ftl | 22 ++++++- .../parsers/parser/SqlPinotCreateTable.java | 21 +++++- .../pinot/sql/parsers/PinotDdlParserTest.java | 26 ++++++++ .../resources/PinotDdlRestletResource.java | 22 ++++--- .../pinot/sql/ddl/compile/DdlCompiler.java | 16 +++++ .../sql/ddl/reverse/CanonicalDdlEmitter.java | 13 ++-- .../pinot/sql/ddl/reverse/SchemaEmitter.java | 2 +- .../sql/ddl/compile/DdlCompilerTest.java | 29 +++++++++ .../sql/ddl/roundtrip/RoundTripTest.java | 64 +++++++++++++++++++ 9 files changed, 194 insertions(+), 21 deletions(-) diff --git a/pinot-common/src/main/codegen/includes/parserImpls.ftl b/pinot-common/src/main/codegen/includes/parserImpls.ftl index f12204829d78..856a230bd9a1 100644 --- a/pinot-common/src/main/codegen/includes/parserImpls.ftl +++ b/pinot-common/src/main/codegen/includes/parserImpls.ftl @@ -125,6 +125,7 @@ SqlNode SqlPinotCreateTable() : SqlIdentifier name; boolean ifNotExists = false; SqlNodeList columns; + SqlNodeList primaryKeyColumns = null; SqlLiteral tableType; SqlNodeList properties = null; } @@ -134,11 +135,13 @@ SqlNode SqlPinotCreateTable() : [ LOOKAHEAD(3) { ifNotExists = true; } ] name = CompoundIdentifier() columns = PinotColumnList() + [ LOOKAHEAD(2) primaryKeyColumns = PinotPrimaryKeyList() ] tableType = PinotTableTypeLiteral() [ properties = PinotPropertyList() ] { - return new SqlPinotCreateTable(pos, name, ifNotExists, columns, tableType, properties); + return new SqlPinotCreateTable(pos, name, ifNotExists, columns, primaryKeyColumns, + tableType, properties); } } @@ -158,6 +161,23 @@ SqlNodeList PinotColumnList() : } } +SqlNodeList PinotPrimaryKeyList() : +{ + SqlParserPos pos; + SqlIdentifier col; + List list = new ArrayList(); +} +{ + { pos = getPos(); } + + col = SimpleIdentifier() { list.add(col); } + ( col = SimpleIdentifier() { list.add(col); } )* + + { + return new SqlNodeList(list, pos.plus(getPos())); + } +} + SqlPinotColumnDeclaration PinotColumnDeclaration() : { SqlParserPos pos; diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java index 66a20e829fd2..53bb77bc1b78 100644 --- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java +++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java @@ -36,13 +36,14 @@ /** * Pinot-native {@code CREATE TABLE} DDL statement. * - *

Syntax (Phase 1, Slice 1): + *

Syntax: * 

{@code
  *   CREATE TABLE [IF NOT EXISTS] [db.]name (
  *     col TYPE [NULL | NOT NULL] [DEFAULT literal] [DIMENSION | METRIC],
  *     col TYPE DATETIME FORMAT 'fmt' GRANULARITY 'gran',
  *     ...
  *   )
+ *   [PRIMARY KEY (col, ...)]
  *   TABLE_TYPE = OFFLINE | REALTIME
  *   PROPERTIES (
  *     'key' = 'value',
@@ -62,15 +63,17 @@ public class SqlPinotCreateTable extends SqlCall {
   private final SqlIdentifier _name;
   private final boolean _ifNotExists;
   private final SqlNodeList _columns;
+  @Nullable private final SqlNodeList _primaryKeyColumns;
   private final SqlLiteral _tableType;
   private final SqlNodeList _properties;
 
   public SqlPinotCreateTable(SqlParserPos pos, SqlIdentifier name, boolean ifNotExists, SqlNodeList columns,
-      SqlLiteral tableType, @Nullable SqlNodeList properties) {
+      @Nullable SqlNodeList primaryKeyColumns, SqlLiteral tableType, @Nullable SqlNodeList properties) {
     super(pos);
     _name = name;
     _ifNotExists = ifNotExists;
     _columns = columns;
+    _primaryKeyColumns = primaryKeyColumns;
     _tableType = tableType;
     _properties = properties == null ? SqlNodeList.EMPTY : properties;
   }
@@ -87,6 +90,11 @@ public SqlNodeList getColumns() {
     return _columns;
   }
 
+  @Nullable
+  public SqlNodeList getPrimaryKeyColumns() {
+    return _primaryKeyColumns;
+  }
+
   public SqlLiteral getTableType() {
     return _tableType;
   }
@@ -118,6 +126,15 @@ public void unparse(SqlWriter writer, int leftPrec, int rightPrec) {
       column.unparse(writer, 0, 0);
     }
     writer.endList(columnFrame);
+    if (_primaryKeyColumns != null && !_primaryKeyColumns.isEmpty()) {
+      writer.keyword("PRIMARY KEY");
+      SqlWriter.Frame pkFrame = writer.startList("(", ")");
+      for (SqlNode pk : _primaryKeyColumns) {
+        writer.sep(",");
+        pk.unparse(writer, 0, 0);
+      }
+      writer.endList(pkFrame);
+    }
     writer.keyword("TABLE_TYPE");
     writer.keyword("=");
     writer.keyword(_tableType.toValue());
diff --git a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
index b638a69a0ae0..93b8a595c790 100644
--- a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
@@ -18,6 +18,7 @@
  */
 package org.apache.pinot.sql.parsers;
 
+import org.apache.calcite.sql.SqlIdentifier;
 import org.apache.calcite.sql.SqlNode;
 import org.apache.pinot.sql.parsers.parser.SqlPinotColumnDeclaration;
 import org.apache.pinot.sql.parsers.parser.SqlPinotCreateTable;
@@ -290,6 +291,31 @@ public void multipleColumnRolesFails() {
             "CREATE TABLE t (id INT DIMENSION METRIC) TABLE_TYPE = OFFLINE"));
   }
 
+  @Test
+  public void createTableWithPrimaryKey() {
+    SqlPinotCreateTable node = parseCreate(
+        "CREATE TABLE t (id INT, name STRING) PRIMARY KEY (id) TABLE_TYPE = OFFLINE");
+    assertNotNull(node.getPrimaryKeyColumns(), "PRIMARY KEY clause should be parsed");
+    assertEquals(node.getPrimaryKeyColumns().size(), 1);
+    assertEquals(((SqlIdentifier) node.getPrimaryKeyColumns().get(0)).getSimple(), "id");
+  }
+
+  @Test
+  public void createTableWithCompositePrimaryKey() {
+    SqlPinotCreateTable node = parseCreate(
+        "CREATE TABLE t (a INT, b STRING, c LONG) PRIMARY KEY (a, b) TABLE_TYPE = OFFLINE");
+    assertEquals(node.getPrimaryKeyColumns().size(), 2);
+    assertEquals(((SqlIdentifier) node.getPrimaryKeyColumns().get(0)).getSimple(), "a");
+    assertEquals(((SqlIdentifier) node.getPrimaryKeyColumns().get(1)).getSimple(), "b");
+  }
+
+  @Test
+  public void createTableWithoutPrimaryKeyHasNullPkList() {
+    SqlPinotCreateTable node = parseCreate(
+        "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE");
+    assertNull(node.getPrimaryKeyColumns());
+  }
+
   @Test
   public void dimensionArrayParsedAsMultiValue() {
     SqlPinotCreateTable stmt = parseCreate(
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 901311b57ba6..0aadc61f3efc 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -228,6 +228,19 @@ private DdlExecutionResponse executeCreate(CompiledCreateTable create, String da
         .setTableConfig(toJson(create.getTableConfig()))
         .setWarnings(create.getWarnings());
 
+    // IF NOT EXISTS short-circuit: check existence BEFORE running config validation so that
+    // rerunning the statement against an existing table is a true no-op. Validation may reject
+    // configs that are valid at create time but would fail if the stored schema or config has
+    // since drifted, making declarative-deploy workflows fragile otherwise.
+    if (!dryRun && _pinotHelixResourceManager.hasTable(tableNameWithType)) {
+      if (create.isIfNotExists()) {
+        response.setMessage("Table " + tableNameWithType + " already exists; CREATE IF NOT EXISTS is a no-op.");
+        return response;
+      }
+      throw new ControllerApplicationException(LOGGER,
+          "Table " + tableNameWithType + " already exists.", Response.Status.CONFLICT);
+    }
+
     // Run the full schema/table validation stack that the existing /tables and /tableConfigs APIs
     // apply before any ZK write. This catches invalid combinations (upsert without primary keys,
     // field configs referencing non-existent columns, task configs with bad column references,
@@ -239,15 +252,6 @@ private DdlExecutionResponse executeCreate(CompiledCreateTable create, String da
       return response;
     }
 
-    if (_pinotHelixResourceManager.hasTable(tableNameWithType)) {
-      if (create.isIfNotExists()) {
-        response.setMessage("Table " + tableNameWithType + " already exists; CREATE IF NOT EXISTS is a no-op.");
-        return response;
-      }
-      throw new ControllerApplicationException(LOGGER,
-          "Table " + tableNameWithType + " already exists.", Response.Status.CONFLICT);
-    }
-
     // When a schema for this raw table name already exists (the common case when adding the
     // second physical variant of a hybrid pair), verify the DDL column list is equivalent to
     // the stored schema. Silently accepting a mismatched column list would create a table whose
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
index 1fa2efa3b37e..340ac1865184 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
@@ -126,10 +126,26 @@ private static CompiledCreateTable compileCreate(SqlPinotCreateTable node) {
     List columns = resolveColumns(node.getColumns().getList());
     Map properties = resolveProperties(node.getProperties().getList());
 
+    // Extract PRIMARY KEY column names (null when no PRIMARY KEY clause).
+    List primaryKeyColumns = null;
+    if (node.getPrimaryKeyColumns() != null && !node.getPrimaryKeyColumns().getList().isEmpty()) {
+      primaryKeyColumns = new ArrayList<>();
+      for (SqlNode pkNode : node.getPrimaryKeyColumns().getList()) {
+        if (!(pkNode instanceof SqlIdentifier)) {
+          throw new DdlCompilationException(
+              "PRIMARY KEY column must be a simple identifier; got: " + pkNode.getClass().getSimpleName());
+        }
+        primaryKeyColumns.add(((SqlIdentifier) pkNode).getSimple());
+      }
+    }
+
     ResolvedTableDefinition resolved = new ResolvedTableDefinition(
         name._databaseName, name._tableName, tableType, node.isIfNotExists(), columns, properties);
 
     Schema schema = buildSchema(resolved);
+    if (primaryKeyColumns != null) {
+      schema.setPrimaryKeyColumns(primaryKeyColumns);
+    }
     List warnings = new ArrayList<>();
     TableConfig tableConfig = buildTableConfig(resolved, warnings);
     validateConsistency(resolved, schema, tableConfig, warnings);
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java
index abc5ec2a01fe..d644b8fd5c6b 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java
@@ -91,19 +91,17 @@ public static String emit(Schema schema, TableConfig config, @Nullable String da
     sb.append(SqlIdentifiers.quote(displayName));
     sb.append(" (\n");
     List columns = SchemaEmitter.emitColumns(schema);
-    List primaryKeyColumns = schema.getPrimaryKeyColumns();
-    boolean hasPrimaryKey = primaryKeyColumns != null && !primaryKeyColumns.isEmpty();
-    // Column count for trailing-comma logic: columns + optional PRIMARY KEY line.
-    int totalEntries = columns.size() + (hasPrimaryKey ? 1 : 0);
     for (int i = 0; i < columns.size(); i++) {
       sb.append(INDENT).append(columns.get(i));
-      if (i < totalEntries - 1) {
+      if (i < columns.size() - 1) {
         sb.append(',');
       }
       sb.append('\n');
     }
-    if (hasPrimaryKey) {
-      sb.append(INDENT).append("PRIMARY KEY (");
+    sb.append(")\n");
+    List primaryKeyColumns = schema.getPrimaryKeyColumns();
+    if (primaryKeyColumns != null && !primaryKeyColumns.isEmpty()) {
+      sb.append("PRIMARY KEY (");
       for (int i = 0; i < primaryKeyColumns.size(); i++) {
         sb.append(SqlIdentifiers.quote(primaryKeyColumns.get(i)));
         if (i < primaryKeyColumns.size() - 1) {
@@ -112,7 +110,6 @@ public static String emit(Schema schema, TableConfig config, @Nullable String da
       }
       sb.append(")\n");
     }
-    sb.append(")\n");
 
     sb.append("TABLE_TYPE = ").append(emitTableType(config.getTableType())).append('\n');
 
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
index 86401fa38c0f..451e65a58945 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
@@ -88,7 +88,7 @@ private static String emitTimeColumn(TimeFieldSpec spec) {
     org.apache.pinot.spi.data.TimeGranularitySpec tgs = spec.getOutgoingGranularitySpec();
     String format = tgs.getTimeUnitSize() + ":" + tgs.getTimeType().name() + ":"
         + tgs.getTimeFormat();
-    String granularity = "1:" + tgs.getTimeType().name();
+    String granularity = tgs.getTimeUnitSize() + ":" + tgs.getTimeType().name();
     StringBuilder sb = new StringBuilder();
     sb.append(SqlIdentifiers.quote(spec.getName()));
     sb.append(' ').append(emitDataType(spec.getDataType()));
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
index 72107a6d6ba8..6cd0d74a0d61 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
@@ -331,6 +331,35 @@ public void compileDropIfExistsWithType() {
     assertEquals(d.getTableType(), TableType.OFFLINE);
   }
 
+  // -------------------------------------------------------------------------------------------
+  // PRIMARY KEY
+  // -------------------------------------------------------------------------------------------
+
+  @Test
+  public void primaryKeyClauseSetsSchemaField() {
+    CompiledCreateTable c = compileCreate(
+        "CREATE TABLE upsertTbl (id INT, val STRING) PRIMARY KEY (id) TABLE_TYPE = REALTIME");
+    assertNotNull(c.getSchema().getPrimaryKeyColumns());
+    assertEquals(c.getSchema().getPrimaryKeyColumns().size(), 1);
+    assertEquals(c.getSchema().getPrimaryKeyColumns().get(0), "id");
+  }
+
+  @Test
+  public void compositePrimaryKeyPreservesOrder() {
+    CompiledCreateTable c = compileCreate(
+        "CREATE TABLE t (a INT, b STRING, c LONG) PRIMARY KEY (b, a) TABLE_TYPE = OFFLINE");
+    assertEquals(c.getSchema().getPrimaryKeyColumns().size(), 2);
+    assertEquals(c.getSchema().getPrimaryKeyColumns().get(0), "b");
+    assertEquals(c.getSchema().getPrimaryKeyColumns().get(1), "a");
+  }
+
+  @Test
+  public void noPrimaryKeyClauseLeavesPrimaryKeyColumnsNull() {
+    CompiledCreateTable c = compileCreate(
+        "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE");
+    assertNull(c.getSchema().getPrimaryKeyColumns());
+  }
+
   // -------------------------------------------------------------------------------------------
   // SHOW TABLES
   // -------------------------------------------------------------------------------------------
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
index 586ad59f64eb..8e02103c459e 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
@@ -20,9 +20,11 @@
 
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.node.ObjectNode;
+import java.util.Arrays;
 import java.util.Iterator;
 import java.util.LinkedHashMap;
 import java.util.Map;
+import java.util.concurrent.TimeUnit;
 import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.TableCustomConfig;
 import org.apache.pinot.spi.config.table.TableType;
@@ -32,6 +34,8 @@
 import org.apache.pinot.spi.data.FieldSpec.DataType;
 import org.apache.pinot.spi.data.MetricFieldSpec;
 import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeFieldSpec;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
 import org.apache.pinot.spi.utils.JsonUtils;
 import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
 import org.apache.pinot.sql.ddl.compile.CompiledCreateTable;
@@ -287,6 +291,66 @@ public void multiValueDimensionRoundTrips() {
     assertRoundTrip(schema, config);
   }
 
+  @Test
+  public void primaryKeyRoundTrip() {
+    Schema schema = new Schema.SchemaBuilder()
+        .setSchemaName("upsertTbl")
+        .addSingleValueDimension("id", DataType.INT)
+        .addSingleValueDimension("userId", DataType.STRING)
+        .addDateTime("ts", DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+        .build();
+    schema.setPrimaryKeyColumns(Arrays.asList("id", "userId"));
+    TableConfig config = new TableConfigBuilder(TableType.REALTIME)
+        .setTableName("upsertTbl")
+        .setTimeColumnName("ts")
+        .build();
+    assertRoundTrip(schema, config);
+  }
+
+  @Test
+  public void timeFieldSpecGranularityPreserved() {
+    // Regression: SchemaEmitter was hardcoding "1:" as the granularity size instead of using
+    // tgs.getTimeUnitSize(). Verify a 15-MINUTE TimeFieldSpec emits "15:MINUTES", not "1:MINUTES".
+    // Note: TimeFieldSpec (legacy) is normalized to DateTimeFieldSpec on the re-parse side, so
+    // this test only validates the emission step — it does not do a full schema round-trip.
+    Schema schema = new Schema();
+    schema.setSchemaName("events");
+    schema.addField(new DimensionFieldSpec("id", DataType.INT, true));
+    TimeGranularitySpec incoming = new TimeGranularitySpec(DataType.LONG, 15, TimeUnit.MINUTES, "incomingTime");
+    TimeGranularitySpec outgoing = new TimeGranularitySpec(DataType.LONG, 15, TimeUnit.MINUTES, "ts");
+    schema.addField(new TimeFieldSpec(incoming, outgoing));
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
+        .setTableName("events")
+        .setTimeColumnName("ts")
+        .build();
+    String ddl = CanonicalDdlEmitter.emit(schema, config);
+    assertNotNull(ddl);
+    // The emitted granularity must reflect the actual timeUnitSize (15), not a hardcoded 1.
+    assertEquals(ddl.contains("15:MINUTES"), true,
+        "Expected granularity '15:MINUTES' in DDL but got:\n" + ddl);
+    assertEquals(ddl.contains("1:MINUTES"), false,
+        "Unexpected hardcoded '1:MINUTES' in DDL:\n" + ddl);
+    // The emitted DDL must also be parseable/compileable without error.
+    CompiledCreateTable compiled = (CompiledCreateTable) DdlCompiler.compile(ddl);
+    assertNotNull(compiled, "Re-parsed DDL should compile");
+  }
+
+  @Test
+  public void dateTimeFieldSpecRoundTrip() {
+    // Regression for DateTimeFieldSpec: format string and granularity must survive the
+    // emit → parse → compile round-trip unchanged.
+    Schema schema = new Schema.SchemaBuilder()
+        .setSchemaName("events")
+        .addSingleValueDimension("id", DataType.INT)
+        .addDateTime("eventTime", DataType.STRING, "1:DAYS:SIMPLE_DATE_FORMAT:yyyy-MM-dd", "1:DAYS")
+        .build();
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
+        .setTableName("events")
+        .setTimeColumnName("eventTime")
+        .build();
+    assertRoundTrip(schema, config);
+  }
+
   // -------------------------------------------------------------------------------------------
   // Equivalence machinery
   // -------------------------------------------------------------------------------------------

From 15a99b30f367179de95547ad5341a5dbdf85b619 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 18 Apr 2026 00:32:12 -0700
Subject: [PATCH 04/32] Fix three more DDL issues from second code review pass

- SqlPinotCreateTable.getOperandList() now includes _primaryKeyColumns
  so Calcite AST traversal (SqlShuttle, validators, node equality) sees
  the full operand set
- DdlCompiler validates PRIMARY KEY columns exist in the column list;
  referencing an undeclared name now throws DdlCompilationException
- PinotDdlRestletResource: run validateTableConfig() BEFORE the
  IF NOT EXISTS existence check so invalid DDL is rejected even when
  the table already exists (IF NOT EXISTS is a no-op only for valid
  statements)
- Tests: add primaryKeyReferencingUnknownColumnThrows compiler test;
  replace assertEquals(bool, true/false) with assertTrue/assertFalse

Co-Authored-By: Claude Sonnet 4.6 
---
 .../parsers/parser/SqlPinotCreateTable.java    |  2 +-
 .../api/resources/PinotDdlRestletResource.java | 18 ++++++++----------
 .../pinot/sql/ddl/compile/DdlCompiler.java     | 10 ++++++++++
 .../pinot/sql/ddl/compile/DdlCompilerTest.java |  6 ++++++
 .../pinot/sql/ddl/roundtrip/RoundTripTest.java |  6 ++++--
 5 files changed, 29 insertions(+), 13 deletions(-)

diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
index 53bb77bc1b78..749406702633 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
@@ -110,7 +110,7 @@ public SqlOperator getOperator() {
 
   @Override
   public List getOperandList() {
-    return Arrays.asList(_name, _columns, _tableType, _properties);
+    return Arrays.asList(_name, _columns, _primaryKeyColumns, _tableType, _properties);
   }
 
   @Override
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 0aadc61f3efc..49d6036d5bf9 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -228,10 +228,14 @@ private DdlExecutionResponse executeCreate(CompiledCreateTable create, String da
         .setTableConfig(toJson(create.getTableConfig()))
         .setWarnings(create.getWarnings());
 
-    // IF NOT EXISTS short-circuit: check existence BEFORE running config validation so that
-    // rerunning the statement against an existing table is a true no-op. Validation may reject
-    // configs that are valid at create time but would fail if the stored schema or config has
-    // since drifted, making declarative-deploy workflows fragile otherwise.
+    // Run the full schema/table validation stack that the existing /tables and /tableConfigs APIs
+    // apply before any ZK write. This catches invalid combinations (upsert without primary keys,
+    // field configs referencing non-existent columns, task configs with bad column references,
+    // etc.) that the compiler alone cannot detect. Runs for both dry-run and live create.
+    // Validation runs BEFORE the existence check so that IF NOT EXISTS is a no-op only for
+    // semantically valid statements; invalid DDL is always rejected regardless of IF NOT EXISTS.
+    validateTableConfig(create.getSchema(), create.getTableConfig());
+
     if (!dryRun && _pinotHelixResourceManager.hasTable(tableNameWithType)) {
       if (create.isIfNotExists()) {
         response.setMessage("Table " + tableNameWithType + " already exists; CREATE IF NOT EXISTS is a no-op.");
@@ -241,12 +245,6 @@ private DdlExecutionResponse executeCreate(CompiledCreateTable create, String da
           "Table " + tableNameWithType + " already exists.", Response.Status.CONFLICT);
     }
 
-    // Run the full schema/table validation stack that the existing /tables and /tableConfigs APIs
-    // apply before any ZK write. This catches invalid combinations (upsert without primary keys,
-    // field configs referencing non-existent columns, task configs with bad column references,
-    // etc.) that the compiler alone cannot detect. Runs for both dry-run and live create.
-    validateTableConfig(create.getSchema(), create.getTableConfig());
-
     if (dryRun) {
       response.setMessage("Dry run: validated CREATE TABLE without persisting.");
       return response;
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
index 340ac1865184..76ba0307caa0 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
@@ -144,6 +144,16 @@ private static CompiledCreateTable compileCreate(SqlPinotCreateTable node) {
 
     Schema schema = buildSchema(resolved);
     if (primaryKeyColumns != null) {
+      Set columnNames = new HashSet<>();
+      for (ResolvedColumnDefinition col : columns) {
+        columnNames.add(col.getName());
+      }
+      for (String pk : primaryKeyColumns) {
+        if (!columnNames.contains(pk)) {
+          throw new DdlCompilationException(
+              "PRIMARY KEY column '" + pk + "' is not declared in the column list.");
+        }
+      }
       schema.setPrimaryKeyColumns(primaryKeyColumns);
     }
     List warnings = new ArrayList<>();
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
index 6cd0d74a0d61..6f3890384a47 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
@@ -360,6 +360,12 @@ public void noPrimaryKeyClauseLeavesPrimaryKeyColumnsNull() {
     assertNull(c.getSchema().getPrimaryKeyColumns());
   }
 
+  @Test
+  public void primaryKeyReferencingUnknownColumnThrows() {
+    expectThrows(DdlCompilationException.class, () -> compileCreate(
+        "CREATE TABLE t (id INT) PRIMARY KEY (nonexistent) TABLE_TYPE = OFFLINE"));
+  }
+
   // -------------------------------------------------------------------------------------------
   // SHOW TABLES
   // -------------------------------------------------------------------------------------------
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
index 8e02103c459e..14e694f30fb5 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
@@ -44,7 +44,9 @@
 import org.testng.annotations.Test;
 
 import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
 import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
 
 
 /**
@@ -326,9 +328,9 @@ public void timeFieldSpecGranularityPreserved() {
     String ddl = CanonicalDdlEmitter.emit(schema, config);
     assertNotNull(ddl);
     // The emitted granularity must reflect the actual timeUnitSize (15), not a hardcoded 1.
-    assertEquals(ddl.contains("15:MINUTES"), true,
+    assertTrue(ddl.contains("15:MINUTES"),
         "Expected granularity '15:MINUTES' in DDL but got:\n" + ddl);
-    assertEquals(ddl.contains("1:MINUTES"), false,
+    assertFalse(ddl.contains("1:MINUTES"),
         "Unexpected hardcoded '1:MINUTES' in DDL:\n" + ddl);
     // The emitted DDL must also be parseable/compileable without error.
     CompiledCreateTable compiled = (CompiledCreateTable) DdlCompiler.compile(ddl);

From e29e6a74daea9b8086158c28af3d3b9643ccf232 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 18 Apr 2026 01:25:45 -0700
Subject: [PATCH 05/32] Fix five DDL issues from adversarial + standard code
 reviews

- replicasPerPartition: add extractor emission, PropertyMapping
  consumed-key registration, and case-insensitive post-build apply in
  DdlCompiler so SHOW CREATE TABLE preserves per-partition replica count
- SHOW CREATE TABLE: catch IllegalArgumentException from
  CanonicalDdlEmitter (unsupported column types MAP/LIST/STRUCT) and
  return 400 instead of 500
- DROP TABLE: per-target try/catch loop reports partial outcomes instead
  of leaving hybrid tables half-deleted; null-safe getMessage() in
  partial-failure error message
- CREATE rollback: generic catch block re-checks both OFFLINE and
  REALTIME variant existence before deleting shared schema so concurrent
  sibling-variant creates are not orphaned
- Dry-run existence check: apply hasTable check for both dryRun=true and
  live paths so dry-run faithfully predicts 409 conflicts
- SqlPinotCreateTable.getOperandList(): null-guard for absent PRIMARY KEY
  so Calcite framework iterators do not NPE on tables without a PK clause

Co-Authored-By: Claude Sonnet 4.6 
---
 .../parsers/parser/SqlPinotCreateTable.java   |  3 +
 .../resources/PinotDdlRestletResource.java    | 75 ++++++++++++++-----
 .../pinot/sql/ddl/compile/DdlCompiler.java    | 12 +++
 .../sql/ddl/compile/PropertyMapping.java      |  6 ++
 .../sql/ddl/reverse/PropertyExtractor.java    |  1 +
 .../sql/ddl/compile/DdlCompilerTest.java      |  8 ++
 .../sql/ddl/roundtrip/RoundTripTest.java      | 19 +++++
 7 files changed, 105 insertions(+), 19 deletions(-)

diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
index 749406702633..9d4b7c7b7a07 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
@@ -110,6 +110,9 @@ public SqlOperator getOperator() {
 
   @Override
   public List getOperandList() {
+    if (_primaryKeyColumns == null) {
+      return Arrays.asList(_name, _columns, _tableType, _properties);
+    }
     return Arrays.asList(_name, _columns, _primaryKeyColumns, _tableType, _properties);
   }
 
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 49d6036d5bf9..8d24bb6b4928 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -236,7 +236,9 @@ private DdlExecutionResponse executeCreate(CompiledCreateTable create, String da
     // semantically valid statements; invalid DDL is always rejected regardless of IF NOT EXISTS.
     validateTableConfig(create.getSchema(), create.getTableConfig());
 
-    if (!dryRun && _pinotHelixResourceManager.hasTable(tableNameWithType)) {
+    // Check existence for both live and dry-run paths: dry-run must faithfully predict conflicts
+    // so callers can use it for pre-deployment validation ("would this DDL succeed?").
+    if (_pinotHelixResourceManager.hasTable(tableNameWithType)) {
       if (create.isIfNotExists()) {
         response.setMessage("Table " + tableNameWithType + " already exists; CREATE IF NOT EXISTS is a no-op.");
         return response;
@@ -303,9 +305,15 @@ private DdlExecutionResponse executeCreate(CompiledCreateTable create, String da
       // The override=false addSchema call lost a race with another schema writer. Surface 409.
       throw new ControllerApplicationException(LOGGER, e.getMessage(), Response.Status.CONFLICT, e);
     } catch (Exception e) {
-      // Roll back the schema we created in this call so we don't leak partial state. We only
-      // touch the schema when this call created it; a pre-existing schema is left alone.
-      rollbackSchemaIfCreated(schemaName, schemaCreatedHere);
+      // Only roll back the schema if no table for this raw name now exists. A concurrent request
+      // may have added the sibling physical variant (OFFLINE vs REALTIME) between our addSchema()
+      // and this addTable() failure; deleting the schema would orphan that live table. Check both
+      // typed variants before deciding to remove the shared schema.
+      if (schemaCreatedHere) {
+        boolean offlineNowExists = _pinotHelixResourceManager.hasOfflineTable(schemaName);
+        boolean realtimeNowExists = _pinotHelixResourceManager.hasRealtimeTable(schemaName);
+        rollbackSchemaIfCreated(schemaName, !offlineNowExists && !realtimeNowExists);
+      }
       // ControllerApplicationException(LOGGER, ...) logs the exception, so don't double-log here.
       throw new ControllerApplicationException(LOGGER,
           "Failed to create table " + tableNameWithType + ": " + e.getMessage(),
@@ -429,17 +437,34 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
       }
     }
 
-    try {
-      for (String target : targets) {
+    // Drop each target individually and track outcomes. A failure on one variant should not
+    // prevent the response from reporting what was already deleted — partial deletes on a hybrid
+    // table are expensive to recover from, so we surface per-target status instead of surfacing
+    // only the first failure and hiding the rest.
+    List dropped = new ArrayList<>();
+    List failed = new ArrayList<>();
+    Exception firstFailure = null;
+    for (String target : targets) {
+      try {
         // Remove task schedules before deletion so tasks are not triggered during the drop.
         PinotTableRestletResource.tableTasksCleanup(target, false,
             _pinotHelixResourceManager, _pinotHelixTaskResourceManager);
         TableType type = TableNameBuilder.getTableTypeFromTableName(target);
         _pinotHelixResourceManager.deleteTable(fullyQualifiedRaw, type, null);
+        dropped.add(target);
+        LOGGER.info("DDL dropped table {}", target);
+      } catch (Exception e) {
+        LOGGER.error("Failed to drop table {} during DDL DROP TABLE", target, e);
+        failed.add(target);
+        if (firstFailure == null) {
+          firstFailure = e;
+        }
       }
-      // Delete the shared schema when the last physical variant has been removed. A schema
-      // without any table leaves stale metadata that blocks future CREATE TABLE for the same
-      // raw name with a different column list.
+    }
+    // Delete the shared schema when the last physical variant has been removed. A schema
+    // without any table leaves stale metadata that blocks future CREATE TABLE for the same
+    // raw name with a different column list. Only attempt if all targets were deleted.
+    if (failed.isEmpty()) {
       boolean offlineExists = _pinotHelixResourceManager.hasOfflineTable(fullyQualifiedRaw);
       boolean realtimeExists = _pinotHelixResourceManager.hasRealtimeTable(fullyQualifiedRaw);
       if (!offlineExists && !realtimeExists) {
@@ -451,17 +476,18 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
               fullyQualifiedRaw, schemaEx);
         }
       }
-      response.setMessage("Dropped " + targets.size() + " table(s).");
-      LOGGER.info("DDL dropped tables {}", targets);
+      response.setMessage("Dropped " + dropped.size() + " table(s).");
+      LOGGER.info("DDL dropped tables {}", dropped);
       return response;
-    } catch (ControllerApplicationException e) {
-      throw e;
-    } catch (Exception e) {
-      // ControllerApplicationException(LOGGER, ...) logs the exception itself; don't double-log.
-      throw new ControllerApplicationException(LOGGER,
-          "Failed to drop table " + fullyQualifiedRaw + ": " + e.getMessage(),
-          Response.Status.INTERNAL_SERVER_ERROR, e);
     }
+    // At least one target failed. Surface a structured error that names both what succeeded
+    // and what failed so the operator knows which variant needs manual cleanup.
+    String causeDesc = firstFailure.getMessage() != null
+        ? firstFailure.getMessage() : firstFailure.getClass().getSimpleName();
+    String msg = "Partial DROP TABLE: dropped " + dropped + ", failed to drop " + failed
+        + ": " + causeDesc;
+    throw new ControllerApplicationException(LOGGER, msg,
+        Response.Status.INTERNAL_SERVER_ERROR, firstFailure);
   }
 
   // -------------------------------------------------------------------------------------------
@@ -533,7 +559,18 @@ private DdlExecutionResponse executeShowCreate(CompiledShowCreateTable show, Str
 
     // Use the resolved database (which incorporates the Database header) so the emitted DDL
     // carries the correct qualifier even when the caller omits the db. prefix in the SQL.
-    String ddl = CanonicalDdlEmitter.emit(schema, tableConfig, database);
+    String ddl;
+    try {
+      ddl = CanonicalDdlEmitter.emit(schema, tableConfig, database);
+    } catch (IllegalArgumentException e) {
+      // SchemaEmitter.validateEmittable() throws IllegalArgumentException for unsupported column
+      // types (MAP, LIST, STRUCT, UNKNOWN) that the current DDL grammar cannot represent. Return
+      // 400 so the caller sees a deterministic client error rather than a 500.
+      throw new ControllerApplicationException(LOGGER,
+          "SHOW CREATE TABLE is not supported for table " + tableNameWithType
+              + ": " + e.getMessage(),
+          Response.Status.BAD_REQUEST, e);
+    }
 
     return new DdlExecutionResponse()
         .setOperation(DdlOperation.SHOW_CREATE_TABLE)
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
index 76ba0307caa0..1bacd30c2ad5 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
@@ -325,6 +325,18 @@ private static TableConfig buildTableConfig(ResolvedTableDefinition resolved, Li
     if (sortedColumns != null && sortedColumns.size() > 1) {
       tableConfig.getIndexingConfig().setSortedColumn(sortedColumns);
     }
+    // replicasPerPartition is not exposed on TableConfigBuilder; apply post-build if present.
+    // Use the same case-fold as PropertyMapping so any accepted casing is honoured.
+    String replicasPerPartition = null;
+    for (Map.Entry e : resolved.getProperties().entrySet()) {
+      if ("replicasperpartition".equals(e.getKey().toLowerCase(Locale.ROOT))) {
+        replicasPerPartition = e.getValue();
+        break;
+      }
+    }
+    if (replicasPerPartition != null) {
+      tableConfig.getValidationConfig().setReplicasPerPartition(replicasPerPartition);
+    }
     return tableConfig;
   }
 
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
index f4d0f60d8d61..6356500b4ae8 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
@@ -122,6 +122,7 @@ public final class PropertyMapping {
     keys.add("aggregatemetrics");
     keys.add("description");
     keys.add("tags");
+    keys.add("replicasperpartition");
     keys.add("ingestionconfig");
     keys.add("upsertconfig");
     keys.add("dedupconfig");
@@ -346,6 +347,11 @@ private static boolean applyPromoted(String lowerKey, String value, TableConfigB
       case "tags":
         builder.setTags(splitCsv(value));
         return true;
+      case "replicasperpartition":
+        // TableConfigBuilder does not expose setReplicasPerPartition; DdlCompiler applies this
+        // value post-build via tableConfig.getValidationConfig().setReplicasPerPartition().
+        // Return true to prevent fall-through to customConfigs.
+        return true;
       default:
         return false;
     }
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java
index 25988d4544f7..e33729839413 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java
@@ -132,6 +132,7 @@ private static void extractValidation(@Nullable SegmentsValidationAndRetentionCo
     if (replication != null && !"1".equals(replication)) {
       props.put("replication", replication);
     }
+    putIfPresent(props, "replicasPerPartition", v.getReplicasPerPartition());
     putIfPresent(props, "peerSegmentDownloadScheme", v.getPeerSegmentDownloadScheme());
     putIfPresent(props, "crypterClassName", v.getCrypterClassName());
     putIfPresent(props, "deletedSegmentsRetentionPeriod",
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
index 6f3890384a47..e1e54014c7cb 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
@@ -366,6 +366,14 @@ public void primaryKeyReferencingUnknownColumnThrows() {
         "CREATE TABLE t (id INT) PRIMARY KEY (nonexistent) TABLE_TYPE = OFFLINE"));
   }
 
+  @Test
+  public void replicasPerPartitionPropertyApplied() {
+    CompiledCreateTable c = compileCreate(
+        "CREATE TABLE t (id INT) TABLE_TYPE = REALTIME PROPERTIES ("
+            + "  'replicasPerPartition' = '3')");
+    assertEquals(c.getTableConfig().getValidationConfig().getReplicasPerPartition(), "3");
+  }
+
   // -------------------------------------------------------------------------------------------
   // SHOW TABLES
   // -------------------------------------------------------------------------------------------
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
index 14e694f30fb5..a376f9a5ad39 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
@@ -353,6 +353,25 @@ public void dateTimeFieldSpecRoundTrip() {
     assertRoundTrip(schema, config);
   }
 
+  @Test
+  public void replicasPerPartitionRoundTrip() {
+    // Regression: replicasPerPartition was silently dropped by PropertyExtractor (it had no
+    // handler) and therefore lost on SHOW CREATE TABLE. It must now survive the full round-trip.
+    Schema schema = new Schema.SchemaBuilder()
+        .setSchemaName("events")
+        .addSingleValueDimension("id", DataType.INT)
+        .build();
+    TableConfig config = new TableConfigBuilder(TableType.REALTIME)
+        .setTableName("events")
+        .build();
+    config.getValidationConfig().setReplicasPerPartition("3");
+    // Assert the emitted DDL text carries the property before doing the full semantic round-trip.
+    String ddl = CanonicalDdlEmitter.emit(schema, config);
+    assertTrue(ddl.contains("replicasPerPartition"),
+        "Expected 'replicasPerPartition' in emitted DDL but got:\n" + ddl);
+    assertRoundTrip(schema, config);
+  }
+
   // -------------------------------------------------------------------------------------------
   // Equivalence machinery
   // -------------------------------------------------------------------------------------------

From 5df5e63bf95081a7b2117d455454f2d0fb75165d Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 18 Apr 2026 02:01:51 -0700
Subject: [PATCH 06/32] Fix seven code-review issues: @Nullable, input length
 guard, schema JSON comparison, SHOW CREATE TABLE hybrid ambiguity, dry-run
 existence check, DECIMAL precision warning, HTTP 201 for CREATE TABLE

- SqlPinotShowTables: add @Nullable on _database field declaration
- PinotDdlRestletResource: guard against >256 KB SQL inputs; use JsonNode.equals()
  for schema comparison to avoid ordering-dependent false mismatches; reject SHOW
  CREATE TABLE without TYPE when both OFFLINE and REALTIME variants exist; drop the
  !dryRun guard so dry-run faithfully predicts conflicts; return HTTP 201 for
  successful non-dry-run CREATE TABLE, 200 for everything else
- DdlCompiler: emit advisory warning when DECIMAL/NUMERIC precision+scale are
  specified (Pinot BIG_DECIMAL does not enforce them); pass warnings list into
  resolveColumns() so the warning can be recorded during column resolution

Co-Authored-By: Claude Sonnet 4.6 
---
 .../parsers/parser/SqlPinotShowTables.java    |  2 +-
 .../resources/PinotDdlRestletResource.java    | 76 ++++++++++++-------
 .../pinot/sql/ddl/compile/DdlCompiler.java    | 17 ++++-
 3 files changed, 61 insertions(+), 34 deletions(-)

diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java
index 13793d45b6ee..4e72d7306a18 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java
@@ -41,7 +41,7 @@ public class SqlPinotShowTables extends SqlCall {
   private static final SqlSpecialOperator OPERATOR =
       new SqlSpecialOperator("SHOW_TABLES", SqlKind.OTHER_DDL);
 
-  private final SqlIdentifier _database;
+  @Nullable private final SqlIdentifier _database;
 
   public SqlPinotShowTables(SqlParserPos pos, @Nullable SqlIdentifier database) {
     super(pos);
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 8d24bb6b4928..c2def94824d0 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -115,6 +115,8 @@
 @Path("/")
 public class PinotDdlRestletResource {
   private static final Logger LOGGER = LoggerFactory.getLogger(PinotDdlRestletResource.class);
+  /** Maximum accepted SQL input length (256 KB) to prevent unbounded parser memory allocation. */
+  private static final int MAX_DDL_SQL_LENGTH = 256 * 1024;
 
   @Inject
   PinotHelixResourceManager _pinotHelixResourceManager;
@@ -138,12 +140,13 @@ public class PinotDdlRestletResource {
           + "generated Schema/TableConfig (CREATE), operation outcome (DROP/SHOW TABLES), or "
           + "canonical DDL string (SHOW CREATE TABLE).")
   @ApiResponses(value = {
-      @ApiResponse(code = 200, message = "Success"),
+      @ApiResponse(code = 200, message = "Success (DROP, SHOW TABLES, SHOW CREATE TABLE, dry-run, IF NOT EXISTS)"),
+      @ApiResponse(code = 201, message = "Table created"),
       @ApiResponse(code = 400, message = "Bad request (parse/semantic error)"),
       @ApiResponse(code = 409, message = "Table already exists"),
       @ApiResponse(code = 500, message = "Internal server error")
   })
-  public DdlExecutionResponse executeDdl(
+  public Response executeDdl(
       @ApiParam(value = "DDL request body with 'sql' field", required = true)
           DdlExecutionRequest request,
       @ApiParam(value = "When true, compile and validate but do not persist.")
@@ -152,6 +155,11 @@ public DdlExecutionResponse executeDdl(
     if (request == null || StringUtils.isBlank(request.getSql())) {
       throw badRequest("Request body must include a non-empty 'sql' field.");
     }
+    // Guard against arbitrarily large inputs that would force the Calcite parser to allocate
+    // excessive memory building the AST in-memory.
+    if (request.getSql().length() > MAX_DDL_SQL_LENGTH) {
+      throw badRequest("DDL statement exceeds maximum length of " + MAX_DDL_SQL_LENGTH + " characters.");
+    }
 
     CompiledDdl compiled;
     try {
@@ -175,13 +183,15 @@ public DdlExecutionResponse executeDdl(
         return executeCreate((CompiledCreateTable) compiled, effectiveDatabase, dryRun,
             httpHeaders, httpRequest);
       case DROP_TABLE:
-        return executeDrop((CompiledDropTable) compiled, effectiveDatabase, dryRun,
-            httpHeaders, httpRequest);
+        return Response.ok(
+            executeDrop((CompiledDropTable) compiled, effectiveDatabase, dryRun,
+                httpHeaders, httpRequest)).build();
       case SHOW_TABLES:
-        return executeShow(effectiveDatabase, httpHeaders, httpRequest);
+        return Response.ok(executeShow(effectiveDatabase, httpHeaders, httpRequest)).build();
       case SHOW_CREATE_TABLE:
-        return executeShowCreate((CompiledShowCreateTable) compiled, effectiveDatabase,
-            httpHeaders, httpRequest);
+        return Response.ok(
+            executeShowCreate((CompiledShowCreateTable) compiled, effectiveDatabase,
+                httpHeaders, httpRequest)).build();
       default:
         throw new ControllerApplicationException(LOGGER, "Unhandled DDL operation: " + op,
             Response.Status.INTERNAL_SERVER_ERROR);
@@ -192,7 +202,7 @@ public DdlExecutionResponse executeDdl(
   // CREATE
   // -------------------------------------------------------------------------------------------
 
-  private DdlExecutionResponse executeCreate(CompiledCreateTable create, String database,
+  private Response executeCreate(CompiledCreateTable create, String database,
       boolean dryRun, HttpHeaders headers, Request httpRequest) {
     // The compiled TableConfig.tableName carries the SQL `db.tbl` qualifier when one was given;
     // translateTableName then reconciles it against the Database header (and rejects conflicts).
@@ -241,7 +251,7 @@ private DdlExecutionResponse executeCreate(CompiledCreateTable create, String da
     if (_pinotHelixResourceManager.hasTable(tableNameWithType)) {
       if (create.isIfNotExists()) {
         response.setMessage("Table " + tableNameWithType + " already exists; CREATE IF NOT EXISTS is a no-op.");
-        return response;
+        return Response.ok(response).build();
       }
       throw new ControllerApplicationException(LOGGER,
           "Table " + tableNameWithType + " already exists.", Response.Status.CONFLICT);
@@ -249,7 +259,7 @@ private DdlExecutionResponse executeCreate(CompiledCreateTable create, String da
 
     if (dryRun) {
       response.setMessage("Dry run: validated CREATE TABLE without persisting.");
-      return response;
+      return Response.ok(response).build();
     }
 
     // When a schema for this raw table name already exists (the common case when adding the
@@ -259,22 +269,20 @@ private DdlExecutionResponse executeCreate(CompiledCreateTable create, String da
     Schema storedSchema = _pinotHelixResourceManager.getSchema(schemaName);
     boolean schemaPreexisted = storedSchema != null;
     if (schemaPreexisted) {
-      try {
-        String storedJson = JsonUtils.objectToString(storedSchema);
-        String compiledJson = JsonUtils.objectToString(create.getSchema());
-        if (!storedJson.equals(compiledJson)) {
-          throw new ControllerApplicationException(LOGGER,
-              "Schema '" + schemaName + "' already exists and does not match the column list in the DDL. "
-                  + "Either omit the column list to reuse the existing schema, or drop and recreate the table pair "
-                  + "if the schema has genuinely changed.",
-              Response.Status.CONFLICT);
-        }
-      } catch (ControllerApplicationException e) {
-        throw e;
-      } catch (Exception e) {
+      // Use JsonNode structural comparison rather than string equality: Jackson may serialize
+      // fields in a different order or represent defaults differently (e.g. 0 vs 0.0) depending
+      // on which code path populated the Schema object, causing false mismatches that would block
+      // valid hybrid table creation even when the schemas are semantically identical.
+      com.fasterxml.jackson.databind.JsonNode storedNode =
+          JsonUtils.objectToJsonNode(storedSchema);
+      com.fasterxml.jackson.databind.JsonNode compiledNode =
+          JsonUtils.objectToJsonNode(create.getSchema());
+      if (!storedNode.equals(compiledNode)) {
         throw new ControllerApplicationException(LOGGER,
-            "Failed to compare schemas for '" + schemaName + "': " + e.getMessage(),
-            Response.Status.INTERNAL_SERVER_ERROR, e);
+            "Schema '" + schemaName + "' already exists and does not match the column list in the DDL. "
+                + "Either omit the column list to reuse the existing schema, or drop and recreate the table pair "
+                + "if the schema has genuinely changed.",
+            Response.Status.CONFLICT);
       }
     }
 
@@ -290,7 +298,7 @@ private DdlExecutionResponse executeCreate(CompiledCreateTable create, String da
       _pinotHelixResourceManager.addTable(create.getTableConfig());
       response.setMessage("Successfully created table " + tableNameWithType);
       LOGGER.info("DDL created table {}", tableNameWithType);
-      return response;
+      return Response.status(Response.Status.CREATED).entity(response).build();
     } catch (TableAlreadyExistsException e) {
       // Race: another caller added the table between our hasTable check and addTable.
       // Only roll back the schema we created here if no table for this raw name now exists —
@@ -485,7 +493,8 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
     String causeDesc = firstFailure.getMessage() != null
         ? firstFailure.getMessage() : firstFailure.getClass().getSimpleName();
     String msg = "Partial DROP TABLE: dropped " + dropped + ", failed to drop " + failed
-        + ": " + causeDesc;
+        + ": " + causeDesc + ". Schema '" + fullyQualifiedRaw
+        + "' may require manual deletion if the remaining variant is also removed.";
     throw new ControllerApplicationException(LOGGER, msg,
         Response.Status.INTERNAL_SERVER_ERROR, firstFailure);
   }
@@ -528,10 +537,19 @@ private DdlExecutionResponse executeShowCreate(CompiledShowCreateTable show, Str
           AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
       ResourceUtils.checkPermissionAndAccess(rt, httpRequest, headers,
           AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
-      if (_pinotHelixResourceManager.hasTable(off)) {
+      boolean offExists = _pinotHelixResourceManager.hasTable(off);
+      boolean rtExists = _pinotHelixResourceManager.hasTable(rt);
+      if (offExists && rtExists) {
+        // Both variants exist; silently picking one would return DDL for the wrong variant.
+        // Require the caller to disambiguate with an explicit TYPE clause.
+        throw new ControllerApplicationException(LOGGER,
+            "Table '" + fullyQualifiedRaw + "' has both OFFLINE and REALTIME variants. "
+                + "Use 'SHOW CREATE TABLE ... TYPE OFFLINE' or 'TYPE REALTIME' to specify which.",
+            Response.Status.BAD_REQUEST);
+      } else if (offExists) {
         tableNameWithType = off;
         resolvedType = TableType.OFFLINE;
-      } else if (_pinotHelixResourceManager.hasTable(rt)) {
+      } else if (rtExists) {
         tableNameWithType = rt;
         resolvedType = TableType.REALTIME;
       } else {
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
index 1bacd30c2ad5..424f841dee99 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
@@ -123,7 +123,8 @@ private static CompiledCreateTable compileCreate(SqlPinotCreateTable node) {
     QualifiedName name = parseQualifiedName(node.getName());
     TableType tableType = parseTableType(node.getTableType().toValue());
 
-    List columns = resolveColumns(node.getColumns().getList());
+    List warnings = new ArrayList<>();
+    List columns = resolveColumns(node.getColumns().getList(), warnings);
     Map properties = resolveProperties(node.getProperties().getList());
 
     // Extract PRIMARY KEY column names (null when no PRIMARY KEY clause).
@@ -156,7 +157,6 @@ private static CompiledCreateTable compileCreate(SqlPinotCreateTable node) {
       }
       schema.setPrimaryKeyColumns(primaryKeyColumns);
     }
-    List warnings = new ArrayList<>();
     TableConfig tableConfig = buildTableConfig(resolved, warnings);
     validateConsistency(resolved, schema, tableConfig, warnings);
 
@@ -164,7 +164,8 @@ private static CompiledCreateTable compileCreate(SqlPinotCreateTable node) {
         resolved.isIfNotExists(), warnings);
   }
 
-  private static List resolveColumns(List columnNodes) {
+  private static List resolveColumns(List columnNodes,
+      List warnings) {
     if (columnNodes.isEmpty()) {
       throw new DdlCompilationException("CREATE TABLE requires at least one column.");
     }
@@ -180,7 +181,15 @@ private static List resolveColumns(List colum
       if (!seen.add(name.toLowerCase(Locale.ROOT))) {
         throw new DdlCompilationException("Duplicate column name: " + name);
       }
-      DataType dt = DataTypeMapper.resolve(col.getDataType().getTypeName().getSimple());
+      String sqlTypeName = col.getDataType().getTypeName().getSimple();
+      DataType dt = DataTypeMapper.resolve(sqlTypeName);
+      // DECIMAL/NUMERIC precision and scale are accepted by the Calcite grammar but Pinot's
+      // BIG_DECIMAL type does not enforce them. Warn so the user knows the constraint is ignored.
+      if (dt == DataType.BIG_DECIMAL
+          && ("DECIMAL".equalsIgnoreCase(sqlTypeName) || "NUMERIC".equalsIgnoreCase(sqlTypeName))) {
+        warnings.add("Column '" + name + "': precision/scale on " + sqlTypeName.toUpperCase(Locale.ROOT)
+            + " is not enforced by Pinot BIG_DECIMAL; the constraint is silently ignored.");
+      }
       ColumnRole role = inferRole(col, dt);
 
       String fmt = col.getDateTimeFormat() == null ? null : col.getDateTimeFormat().toValue();

From 2500ce7910b72995d30102a3c31f3fc5a4936341 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 18 Apr 2026 02:11:16 -0700
Subject: [PATCH 07/32] Fix four second-pass review issues: DECIMAL precision
 condition, inline FQCN, variable-arity getOperandList, and missing tests

- DdlCompiler: gate DECIMAL/NUMERIC precision warning on whether the user actually
  specified DECIMAL(p,s) by checking SqlBasicTypeNameSpec.getPrecision() against
  RelDataType.PRECISION_NOT_SPECIFIED (-1); bare DECIMAL no longer emits a spurious
  warning
- PinotDdlRestletResource: replace fully-qualified com.fasterxml.jackson.databind.JsonNode
  inline references with the already-imported short form
- SqlPinotCreateTable.getOperandList(): use fixed-arity list with null placeholder for
  the optional _primaryKeyColumns instead of a variable-length list that shifts all
  subsequent positional indices when PRIMARY KEY is absent
- DdlCompilerTest: add decimalWithPrecisionScaleEmitsWarning and
  decimalWithoutPrecisionEmitsNoWarning regression tests
- PinotDdlRestletResourceTest: add successfulCreateReturns201 and
  oversizedInputReturnsBadRequest integration tests

Co-Authored-By: Claude Sonnet 4.6 
---
 .../parsers/parser/SqlPinotCreateTable.java   |  4 +---
 .../resources/PinotDdlRestletResource.java    |  6 ++---
 .../api/PinotDdlRestletResourceTest.java      | 23 +++++++++++++++++++
 .../pinot/sql/ddl/compile/DdlCompiler.java    |  8 +++++--
 .../sql/ddl/compile/DdlCompilerTest.java      | 18 +++++++++++++++
 5 files changed, 50 insertions(+), 9 deletions(-)

diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
index 9d4b7c7b7a07..1a4c08d84264 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
@@ -110,9 +110,7 @@ public SqlOperator getOperator() {
 
   @Override
   public List getOperandList() {
-    if (_primaryKeyColumns == null) {
-      return Arrays.asList(_name, _columns, _tableType, _properties);
-    }
+    // Fixed-arity list with null placeholder for absent optional operand, per Calcite SqlCall contract.
     return Arrays.asList(_name, _columns, _primaryKeyColumns, _tableType, _properties);
   }
 
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index c2def94824d0..84094b77dcbd 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -273,10 +273,8 @@ private Response executeCreate(CompiledCreateTable create, String database,
       // fields in a different order or represent defaults differently (e.g. 0 vs 0.0) depending
       // on which code path populated the Schema object, causing false mismatches that would block
       // valid hybrid table creation even when the schemas are semantically identical.
-      com.fasterxml.jackson.databind.JsonNode storedNode =
-          JsonUtils.objectToJsonNode(storedSchema);
-      com.fasterxml.jackson.databind.JsonNode compiledNode =
-          JsonUtils.objectToJsonNode(create.getSchema());
+      JsonNode storedNode = JsonUtils.objectToJsonNode(storedSchema);
+      JsonNode compiledNode = JsonUtils.objectToJsonNode(create.getSchema());
       if (!storedNode.equals(compiledNode)) {
         throw new ControllerApplicationException(LOGGER,
             "Schema '" + schemaName + "' already exists and does not match the column list in the DDL. "
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
index dda400016b03..3d8a74284168 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
@@ -22,6 +22,8 @@
 import java.io.IOException;
 import java.util.Collections;
 import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
 import org.apache.pinot.controller.helix.ControllerTest;
 import org.apache.pinot.spi.utils.JsonUtils;
 import org.testng.annotations.AfterClass;
@@ -287,6 +289,27 @@ public void semanticErrorReturnsBadRequest()
     assertEquals(status, 400);
   }
 
+  @Test
+  public void successfulCreateReturns201()
+      throws IOException {
+    String tbl = "ddlCreate201Test";
+    String url = DEFAULT_INSTANCE.getControllerBaseApiUrl() + "/sql/ddl";
+    String body = "{\"sql\": \"CREATE TABLE " + tbl + " (id INT) TABLE_TYPE = OFFLINE\"}";
+    Pair result = postRequestWithStatusCode(url, body);
+    assertEquals(result.getLeft().intValue(), 201, "Expected HTTP 201 for successful CREATE TABLE");
+    JsonNode response = JsonUtils.stringToJsonNode(result.getRight());
+    assertEquals(response.get("operation").asText(), "CREATE_TABLE");
+  }
+
+  @Test
+  public void oversizedInputReturnsBadRequest()
+      throws IOException {
+    // Any SQL that exceeds 256 KB must be rejected before parsing to prevent unbounded allocations.
+    String oversized = "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE /* " + StringUtils.repeat("x", 256 * 1024) + " */";
+    int status = postDdlExpectFailure(oversized);
+    assertEquals(status, 400, "Expected 400 for input exceeding MAX_DDL_SQL_LENGTH");
+  }
+
   // -------------------------------------------------------------------------------------------
   // Helpers
   // -------------------------------------------------------------------------------------------
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
index 424f841dee99..acd76ab5f56c 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
@@ -184,9 +184,13 @@ private static List resolveColumns(List colum
       String sqlTypeName = col.getDataType().getTypeName().getSimple();
       DataType dt = DataTypeMapper.resolve(sqlTypeName);
       // DECIMAL/NUMERIC precision and scale are accepted by the Calcite grammar but Pinot's
-      // BIG_DECIMAL type does not enforce them. Warn so the user knows the constraint is ignored.
+      // BIG_DECIMAL type does not enforce them. Warn only when the user actually wrote
+      // DECIMAL(p,s) — Calcite uses RelDataType.PRECISION_NOT_SPECIFIED (-1) when omitted.
       if (dt == DataType.BIG_DECIMAL
-          && ("DECIMAL".equalsIgnoreCase(sqlTypeName) || "NUMERIC".equalsIgnoreCase(sqlTypeName))) {
+          && ("DECIMAL".equalsIgnoreCase(sqlTypeName) || "NUMERIC".equalsIgnoreCase(sqlTypeName))
+          && col.getDataType().getTypeNameSpec() instanceof org.apache.calcite.sql.SqlBasicTypeNameSpec
+          && ((org.apache.calcite.sql.SqlBasicTypeNameSpec) col.getDataType().getTypeNameSpec())
+              .getPrecision() != org.apache.calcite.rel.type.RelDataType.PRECISION_NOT_SPECIFIED) {
         warnings.add("Column '" + name + "': precision/scale on " + sqlTypeName.toUpperCase(Locale.ROOT)
             + " is not enforced by Pinot BIG_DECIMAL; the constraint is silently ignored.");
       }
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
index e1e54014c7cb..ab425fafbff5 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
@@ -390,6 +390,24 @@ public void compileShowFromDatabase() {
     assertEquals(s.getDatabaseName(), "analytics");
   }
 
+  // -------------------------------------------------------------------------------------------
+  // DECIMAL precision warning
+  // -------------------------------------------------------------------------------------------
+
+  @Test
+  public void decimalWithPrecisionScaleEmitsWarning() {
+    CompiledCreateTable c = compileCreate("CREATE TABLE t (price DECIMAL(10,2)) TABLE_TYPE = OFFLINE");
+    assertTrue(c.getWarnings().stream().anyMatch(w -> w.contains("DECIMAL")),
+        "Expected DECIMAL precision warning, got: " + c.getWarnings());
+  }
+
+  @Test
+  public void decimalWithoutPrecisionEmitsNoWarning() {
+    CompiledCreateTable c = compileCreate("CREATE TABLE t (price DECIMAL) TABLE_TYPE = OFFLINE");
+    assertTrue(c.getWarnings().stream().noneMatch(w -> w.contains("DECIMAL")),
+        "Unexpected DECIMAL warning for bare DECIMAL: " + c.getWarnings());
+  }
+
   // -------------------------------------------------------------------------------------------
   // Helpers
   // -------------------------------------------------------------------------------------------

From 439a9ab2c2a0fac01ab6e696afc2e1b3e1cc1ba5 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 18 Apr 2026 02:14:54 -0700
Subject: [PATCH 08/32] Replace inline FQCNs with imports for
 SqlBasicTypeNameSpec and RelDataType in DdlCompiler

Co-Authored-By: Claude Sonnet 4.6 
---
 .../org/apache/pinot/sql/ddl/compile/DdlCompiler.java     | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
index acd76ab5f56c..014d56c1f608 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
@@ -26,6 +26,8 @@
 import java.util.Map;
 import java.util.Set;
 import javax.annotation.Nullable;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.sql.SqlBasicTypeNameSpec;
 import org.apache.calcite.sql.SqlIdentifier;
 import org.apache.calcite.sql.SqlLiteral;
 import org.apache.calcite.sql.SqlNode;
@@ -188,9 +190,9 @@ private static List resolveColumns(List colum
       // DECIMAL(p,s) — Calcite uses RelDataType.PRECISION_NOT_SPECIFIED (-1) when omitted.
       if (dt == DataType.BIG_DECIMAL
           && ("DECIMAL".equalsIgnoreCase(sqlTypeName) || "NUMERIC".equalsIgnoreCase(sqlTypeName))
-          && col.getDataType().getTypeNameSpec() instanceof org.apache.calcite.sql.SqlBasicTypeNameSpec
-          && ((org.apache.calcite.sql.SqlBasicTypeNameSpec) col.getDataType().getTypeNameSpec())
-              .getPrecision() != org.apache.calcite.rel.type.RelDataType.PRECISION_NOT_SPECIFIED) {
+          && col.getDataType().getTypeNameSpec() instanceof SqlBasicTypeNameSpec
+          && ((SqlBasicTypeNameSpec) col.getDataType().getTypeNameSpec())
+              .getPrecision() != RelDataType.PRECISION_NOT_SPECIFIED) {
         warnings.add("Column '" + name + "': precision/scale on " + sqlTypeName.toUpperCase(Locale.ROOT)
             + " is not enforced by Pinot BIG_DECIMAL; the constraint is silently ignored.");
       }

From a502f0b0b9c4fbbc59f7160cdaa45d23310f1bbf Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Fri, 24 Apr 2026 16:30:05 -0700
Subject: [PATCH 09/32] Fix three MAJOR DDL review issues: schema comparison,
 DROP schema cleanup, test helper

- Replace full-JSON hybrid-table schema comparison with a column-shape
  comparator that covers only attributes the DDL column list actually
  expresses (name, dataType, fieldType, singleValue, NOT NULL, default,
  DATETIME format/granularity) and ignores schema-level metadata the DDL
  cannot set (primaryKeyColumns, tags, enableColumnBasedNullHandling).
  Prevents false CONFLICT when the first hybrid variant set primary keys
  and the second variant's DDL does not restate them.
- Remove auto-schema-deletion from DROP TABLE so the DDL path matches
  the existing /tables/{name} DELETE contract, which leaves the shared
  schema intact when the last variant is removed.
- Replace fragile substring-scan for HTTP status codes in
  postRawExpectFailureBody with postRequestWithStatusCode, which reads
  the status from the response directly. Also fix a stale test
  assertion that expected an unquoted "default" in canonical DDL (the
  emitter correctly quotes it because DEFAULT is a reserved keyword).
- Add PinotDdlRestletResourceUnitTest covering acceptance when PK
  metadata differs and rejection for data-type, field-type,
  single-value, NOT NULL, default-null-value, and DATETIME format /
  granularity mismatches.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../resources/PinotDdlRestletResource.java    | 124 +++++++---
 .../api/PinotDdlRestletResourceTest.java      |  32 +--
 .../PinotDdlRestletResourceUnitTest.java      | 222 ++++++++++++++++++
 3 files changed, 335 insertions(+), 43 deletions(-)
 create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java

diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 84094b77dcbd..bff76b9477fa 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -29,7 +29,11 @@
 import io.swagger.annotations.SecurityDefinition;
 import io.swagger.annotations.SwaggerDefinition;
 import java.util.ArrayList;
+import java.util.HashSet;
 import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import javax.annotation.Nullable;
 import javax.inject.Inject;
 import javax.ws.rs.Consumes;
 import javax.ws.rs.DefaultValue;
@@ -62,8 +66,11 @@
 import org.apache.pinot.core.auth.ManualAuthorization;
 import org.apache.pinot.core.auth.TargetType;
 import org.apache.pinot.segment.local.utils.TableConfigUtils;
+import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.TableConfigValidatorRegistry;
 import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.DateTimeFieldSpec;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.LogicalTableConfig;
 import org.apache.pinot.spi.data.Schema;
 import org.apache.pinot.spi.utils.CommonConstants;
@@ -269,17 +276,18 @@ private Response executeCreate(CompiledCreateTable create, String database,
     Schema storedSchema = _pinotHelixResourceManager.getSchema(schemaName);
     boolean schemaPreexisted = storedSchema != null;
     if (schemaPreexisted) {
-      // Use JsonNode structural comparison rather than string equality: Jackson may serialize
-      // fields in a different order or represent defaults differently (e.g. 0 vs 0.0) depending
-      // on which code path populated the Schema object, causing false mismatches that would block
-      // valid hybrid table creation even when the schemas are semantically identical.
-      JsonNode storedNode = JsonUtils.objectToJsonNode(storedSchema);
-      JsonNode compiledNode = JsonUtils.objectToJsonNode(create.getSchema());
-      if (!storedNode.equals(compiledNode)) {
+      // Compare only the column-shape attributes that the DDL column list actually controls.
+      // Comparing full JSON would include schema-level metadata (primary keys, null-handling,
+      // tags, description) that the DDL does not express when a column list is given for the
+      // second hybrid variant — e.g. a DDL without PRIMARY KEY would spuriously conflict with
+      // a stored schema whose primary keys were set by the first variant.
+      String mismatch = describeColumnShapeMismatch(storedSchema, create.getSchema());
+      if (mismatch != null) {
         throw new ControllerApplicationException(LOGGER,
-            "Schema '" + schemaName + "' already exists and does not match the column list in the DDL. "
-                + "Either omit the column list to reuse the existing schema, or drop and recreate the table pair "
-                + "if the schema has genuinely changed.",
+            "Schema '" + schemaName + "' already exists and does not match the column list in the DDL: "
+                + mismatch
+                + ". Either omit the column list to reuse the existing schema, or drop and recreate the "
+                + "table pair if the schema has genuinely changed.",
             Response.Status.CONFLICT);
       }
     }
@@ -340,14 +348,85 @@ private void rollbackSchemaIfCreated(String schemaName, boolean schemaCreatedHer
     }
   }
 
+  /**
+   * Compares two schemas by the column-shape attributes that a DDL column list actually controls
+   * (column name, data type, field type, single/multi-value, NOT NULL, default null value, and —
+   * for DATETIME columns — format and granularity) and returns a human-readable description of
+   * the first mismatch, or {@code null} if the shapes are equivalent. Schema-level metadata that
+   * a DDL column list does not express ({@code primaryKeyColumns}, {@code tags},
+   * {@code enableColumnBasedNullHandling}, {@code description}) is intentionally ignored so the
+   * second hybrid variant can be created via DDL without restating metadata set by the first
+   * variant.
+   */
+  // Package-private for unit testing.
+  @Nullable
+  static String describeColumnShapeMismatch(Schema stored, Schema compiled) {
+    Set storedColumns = stored.getColumnNames();
+    Set compiledColumns = compiled.getColumnNames();
+    if (!storedColumns.equals(compiledColumns)) {
+      Set missing = new HashSet<>(storedColumns);
+      missing.removeAll(compiledColumns);
+      Set extra = new HashSet<>(compiledColumns);
+      extra.removeAll(storedColumns);
+      StringBuilder sb = new StringBuilder("column sets differ");
+      if (!missing.isEmpty()) {
+        sb.append("; missing from DDL: ").append(missing);
+      }
+      if (!extra.isEmpty()) {
+        sb.append("; extra in DDL: ").append(extra);
+      }
+      return sb.toString();
+    }
+    for (String columnName : storedColumns) {
+      FieldSpec storedSpec = stored.getFieldSpecFor(columnName);
+      FieldSpec compiledSpec = compiled.getFieldSpecFor(columnName);
+      if (storedSpec.getDataType() != compiledSpec.getDataType()) {
+        return "column '" + columnName + "' data type differs (stored=" + storedSpec.getDataType()
+            + ", DDL=" + compiledSpec.getDataType() + ")";
+      }
+      if (storedSpec.getFieldType() != compiledSpec.getFieldType()) {
+        return "column '" + columnName + "' field type differs (stored=" + storedSpec.getFieldType()
+            + ", DDL=" + compiledSpec.getFieldType() + ")";
+      }
+      if (storedSpec.isSingleValueField() != compiledSpec.isSingleValueField()) {
+        return "column '" + columnName + "' single-valued flag differs";
+      }
+      if (storedSpec.isNotNull() != compiledSpec.isNotNull()) {
+        return "column '" + columnName + "' NOT NULL flag differs (stored=" + storedSpec.isNotNull()
+            + ", DDL=" + compiledSpec.isNotNull() + ")";
+      }
+      // Compare default null value by its string form: the DDL always sets defaults from a
+      // string literal, and FieldSpec normalizes the stored representation to a typed value.
+      // The string form is the stable projection that survives both serialization round trips.
+      String storedDefault = storedSpec.getDefaultNullValueString();
+      String compiledDefault = compiledSpec.getDefaultNullValueString();
+      if (!Objects.equals(storedDefault, compiledDefault)) {
+        return "column '" + columnName + "' default null value differs (stored=" + storedDefault
+            + ", DDL=" + compiledDefault + ")";
+      }
+      if (storedSpec instanceof DateTimeFieldSpec && compiledSpec instanceof DateTimeFieldSpec) {
+        DateTimeFieldSpec storedDt = (DateTimeFieldSpec) storedSpec;
+        DateTimeFieldSpec compiledDt = (DateTimeFieldSpec) compiledSpec;
+        if (!Objects.equals(storedDt.getFormat(), compiledDt.getFormat())) {
+          return "column '" + columnName + "' DATETIME format differs (stored="
+              + storedDt.getFormat() + ", DDL=" + compiledDt.getFormat() + ")";
+        }
+        if (!Objects.equals(storedDt.getGranularity(), compiledDt.getGranularity())) {
+          return "column '" + columnName + "' DATETIME granularity differs (stored="
+              + storedDt.getGranularity() + ", DDL=" + compiledDt.getGranularity() + ")";
+        }
+      }
+    }
+    return null;
+  }
+
   /**
    * Runs the same schema/table validation stack that {@code /tables} and {@code /tableConfigs}
    * apply before any ZK write, so DDL-created configs are subject to the same rules as
    * JSON-API-created configs (upsert/dedup primary-key requirements, field config column
    * references, task config validation, registry-level semantic checks, etc.).
    */
-  private void validateTableConfig(Schema schema,
-      org.apache.pinot.spi.config.table.TableConfig tableConfig) {
+  private void validateTableConfig(Schema schema, TableConfig tableConfig) {
     try {
       TableConfigUtils.validateTableName(tableConfig);
       TableConfigUtils.validate(tableConfig, schema, null);
@@ -467,21 +546,11 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
         }
       }
     }
-    // Delete the shared schema when the last physical variant has been removed. A schema
-    // without any table leaves stale metadata that blocks future CREATE TABLE for the same
-    // raw name with a different column list. Only attempt if all targets were deleted.
+    // Intentionally do NOT delete the shared schema when the last physical variant is removed.
+    // This matches the existing `/tables/{name}` DELETE contract, which also leaves the schema
+    // intact. Two doors into the same state machine must have the same side effects; a caller
+    // who wants to remove the schema can issue an explicit DELETE /schemas/{name} afterwards.
     if (failed.isEmpty()) {
-      boolean offlineExists = _pinotHelixResourceManager.hasOfflineTable(fullyQualifiedRaw);
-      boolean realtimeExists = _pinotHelixResourceManager.hasRealtimeTable(fullyQualifiedRaw);
-      if (!offlineExists && !realtimeExists) {
-        try {
-          _pinotHelixResourceManager.deleteSchema(fullyQualifiedRaw);
-          LOGGER.info("DDL deleted schema {} after dropping last table variant", fullyQualifiedRaw);
-        } catch (Exception schemaEx) {
-          LOGGER.warn("Failed to delete schema {} after DROP TABLE; manual cleanup may be required",
-              fullyQualifiedRaw, schemaEx);
-        }
-      }
       response.setMessage("Dropped " + dropped.size() + " table(s).");
       LOGGER.info("DDL dropped tables {}", dropped);
       return response;
@@ -491,8 +560,7 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
     String causeDesc = firstFailure.getMessage() != null
         ? firstFailure.getMessage() : firstFailure.getClass().getSimpleName();
     String msg = "Partial DROP TABLE: dropped " + dropped + ", failed to drop " + failed
-        + ": " + causeDesc + ". Schema '" + fullyQualifiedRaw
-        + "' may require manual deletion if the remaining variant is also removed.";
+        + ": " + causeDesc + ".";
     throw new ControllerApplicationException(LOGGER, msg,
         Response.Status.INTERNAL_SERVER_ERROR, firstFailure);
   }
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
index 3d8a74284168..56f17cce407f 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
@@ -21,6 +21,7 @@
 import com.fasterxml.jackson.databind.JsonNode;
 import java.io.IOException;
 import java.util.Collections;
+import java.util.LinkedHashMap;
 import java.util.Map;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.commons.lang3.tuple.Pair;
@@ -229,7 +230,7 @@ public void showCreateWithDatabaseHeaderEmitsDatabaseQualifiedDdl()
 
     String url = DEFAULT_INSTANCE.getControllerBaseApiUrl() + "/sql/ddl";
     String body = "{\"sql\": \"SHOW CREATE TABLE " + tbl + "\"}";
-    java.util.Map hdrs = new java.util.LinkedHashMap<>();
+    Map hdrs = new LinkedHashMap<>();
     hdrs.put("Content-Type", "application/json");
     // Pass database via header with no SQL db. qualifier — the resolved database must appear in
     // the emitted DDL so replaying without the header still targets the correct tenant.
@@ -238,7 +239,10 @@ public void showCreateWithDatabaseHeaderEmitsDatabaseQualifiedDdl()
     JsonNode response = JsonUtils.stringToJsonNode(raw);
     assertEquals(response.get("operation").asText(), "SHOW_CREATE_TABLE");
     String ddl = response.get("ddl").asText();
-    assertTrue(ddl.startsWith("CREATE TABLE default." + tbl),
+    // The database name "default" is a SQL reserved keyword, so the canonical emitter must
+    // double-quote it to round-trip. The qualifier must still be present — otherwise replaying
+    // the DDL without the Database header would target the wrong tenant.
+    assertTrue(ddl.startsWith("CREATE TABLE \"default\"." + tbl),
         "Expected DDL to carry db-qualified name; got:\n" + ddl);
   }
 
@@ -340,21 +344,19 @@ private static int postRawExpectFailure(String body) {
   }
 
   private static int postRawExpectFailureBody(String url, String body) {
-    Map headers = Collections.singletonMap("Content-Type", "application/json");
+    // Use postRequestWithStatusCode which does not throw on error — the HTTP status code is
+    // returned directly from the underlying response. Scanning exception messages for numeric
+    // substrings is fragile because the error body may itself contain integers that look like
+    // status codes (e.g. the echoed request body or a column count).
     try {
-      sendPostRequest(url, body, headers);
-      fail("Expected request to fail with HTTP error");
-      return -1;
-    } catch (IOException e) {
-      // The wrapper throws an HttpErrorStatusException whose message starts with the status code.
-      String msg = e.getMessage() == null ? "" : e.getMessage();
-      // Status code is the first integer prefix in the message; default to 500 if not parseable.
-      for (int code : new int[]{400, 404, 409, 500}) {
-        if (msg.contains(String.valueOf(code))) {
-          return code;
-        }
+      Pair result = postRequestWithStatusCode(url, body);
+      int status = result.getLeft();
+      if (status >= 200 && status < 300) {
+        fail("Expected request to fail with HTTP error but got status " + status);
       }
-      return 500;
+      return status;
+    } catch (IOException e) {
+      throw new RuntimeException("Unexpected IO error contacting controller", e);
     }
   }
 }
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
new file mode 100644
index 000000000000..bddf5cbfa25f
--- /dev/null
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
@@ -0,0 +1,222 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.api.resources;
+
+import java.util.Arrays;
+import java.util.Collections;
+import org.apache.pinot.spi.data.DateTimeFieldSpec;
+import org.apache.pinot.spi.data.DimensionFieldSpec;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.data.MetricFieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * Unit tests for {@link PinotDdlRestletResource#describeColumnShapeMismatch}. This is the
+ * hybrid-table CREATE gate that decides whether a DDL-compiled schema is compatible with the
+ * schema already stored in ZK for a hybrid pair's sibling variant. The comparator must accept
+ * differences in schema-level metadata a DDL column list cannot express (primary keys, tags,
+ * null-handling) and reject differences in per-column attributes that the DDL does control.
+ */
+public class PinotDdlRestletResourceUnitTest {
+
+  /** Compiled DDL with matching columns but no primary keys must accept a stored schema that has them. */
+  @Test
+  public void acceptsMatchingColumnsWhenStoredHasExtraPrimaryKeyMetadata() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    stored.addField(new DimensionFieldSpec("id", DataType.INT, true));
+    stored.setPrimaryKeyColumns(Collections.singletonList("id"));
+    stored.setEnableColumnBasedNullHandling(true);
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    compiled.addField(new DimensionFieldSpec("id", DataType.INT, true));
+
+    assertNull(PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled),
+        "schema-level metadata the DDL column list cannot express must not drive a mismatch");
+  }
+
+  /** A missing or extra column must be rejected with a named column set in the message. */
+  @Test
+  public void rejectsColumnSetDifference() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    stored.addField(new DimensionFieldSpec("id", DataType.INT, true));
+    stored.addField(new DimensionFieldSpec("name", DataType.STRING, true));
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    compiled.addField(new DimensionFieldSpec("id", DataType.INT, true));
+
+    String msg = PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled);
+    assertNotNull(msg);
+    assertTrue(msg.contains("column sets differ") && msg.contains("name"),
+        "message should call out the offending column set difference: " + msg);
+  }
+
+  /** Different data type for the same column must be rejected. */
+  @Test
+  public void rejectsDataTypeMismatch() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    stored.addField(new DimensionFieldSpec("id", DataType.INT, true));
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    compiled.addField(new DimensionFieldSpec("id", DataType.LONG, true));
+
+    String msg = PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled);
+    assertNotNull(msg);
+    assertTrue(msg.contains("data type differs"), msg);
+  }
+
+  /** DIMENSION vs METRIC for the same column must be rejected. */
+  @Test
+  public void rejectsFieldTypeMismatch() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    stored.addField(new DimensionFieldSpec("v", DataType.LONG, true));
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    compiled.addField(new MetricFieldSpec("v", DataType.LONG));
+
+    String msg = PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled);
+    assertNotNull(msg);
+    assertTrue(msg.contains("field type differs"), msg);
+  }
+
+  /** Single-value vs multi-value mismatch must be rejected. */
+  @Test
+  public void rejectsSingleValuedFlagMismatch() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    stored.addField(new DimensionFieldSpec("tags", DataType.STRING, true));
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    compiled.addField(new DimensionFieldSpec("tags", DataType.STRING, false));
+
+    String msg = PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled);
+    assertNotNull(msg);
+    assertTrue(msg.contains("single-valued flag"), msg);
+  }
+
+  /** NOT NULL flag mismatch must be rejected. */
+  @Test
+  public void rejectsNotNullFlagMismatch() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    FieldSpec storedSpec = new DimensionFieldSpec("id", DataType.INT, true);
+    storedSpec.setNotNull(true);
+    stored.addField(storedSpec);
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    compiled.addField(new DimensionFieldSpec("id", DataType.INT, true));
+
+    String msg = PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled);
+    assertNotNull(msg);
+    assertTrue(msg.contains("NOT NULL flag"), msg);
+  }
+
+  /** Default-null-value mismatch must be rejected. */
+  @Test
+  public void rejectsDefaultNullValueMismatch() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    FieldSpec storedSpec = new DimensionFieldSpec("id", DataType.INT, true);
+    storedSpec.setDefaultNullValue(-1);
+    stored.addField(storedSpec);
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    FieldSpec compiledSpec = new DimensionFieldSpec("id", DataType.INT, true);
+    compiledSpec.setDefaultNullValue(0);
+    compiled.addField(compiledSpec);
+
+    String msg = PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled);
+    assertNotNull(msg);
+    assertTrue(msg.contains("default null value"), msg);
+  }
+
+  /** DATETIME format mismatch must be rejected. */
+  @Test
+  public void rejectsDateTimeFormatMismatch() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    stored.addField(new DateTimeFieldSpec("ts", DataType.LONG,
+        "1:MILLISECONDS:EPOCH", "1:MILLISECONDS"));
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    compiled.addField(new DateTimeFieldSpec("ts", DataType.LONG,
+        "1:SECONDS:EPOCH", "1:MILLISECONDS"));
+
+    String msg = PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled);
+    assertNotNull(msg);
+    assertTrue(msg.contains("DATETIME format"), msg);
+  }
+
+  /** DATETIME granularity mismatch must be rejected. */
+  @Test
+  public void rejectsDateTimeGranularityMismatch() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    stored.addField(new DateTimeFieldSpec("ts", DataType.LONG,
+        "1:MILLISECONDS:EPOCH", "1:MILLISECONDS"));
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    compiled.addField(new DateTimeFieldSpec("ts", DataType.LONG,
+        "1:MILLISECONDS:EPOCH", "1:SECONDS"));
+
+    String msg = PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled);
+    assertNotNull(msg);
+    assertTrue(msg.contains("DATETIME granularity"), msg);
+  }
+
+  /** Matching multi-column, multi-type schemas must be accepted. */
+  @Test
+  public void acceptsMatchingMixedColumnSchema() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    stored.addField(new DimensionFieldSpec("id", DataType.INT, true));
+    stored.addField(new MetricFieldSpec("value", DataType.DOUBLE));
+    stored.addField(new DateTimeFieldSpec("ts", DataType.LONG,
+        "1:MILLISECONDS:EPOCH", "1:MILLISECONDS"));
+    stored.setPrimaryKeyColumns(Arrays.asList("id"));
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    compiled.addField(new DimensionFieldSpec("id", DataType.INT, true));
+    compiled.addField(new MetricFieldSpec("value", DataType.DOUBLE));
+    compiled.addField(new DateTimeFieldSpec("ts", DataType.LONG,
+        "1:MILLISECONDS:EPOCH", "1:MILLISECONDS"));
+
+    assertNull(PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled));
+  }
+}

From 63a8f39059565828f3d6d6b3f37c14dfa3027418 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Fri, 24 Apr 2026 23:38:21 -0700
Subject: [PATCH 10/32] Fix four CRITICAL and three MAJOR DDL review issues
 from orchestrator pass
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

CRITICAL:
- CREATE TABLE IF NOT EXISTS now matches standard SQL semantics: the
  existence check runs before validateTableConfig, so an idempotent
  re-CREATE against an existing table succeeds even if a config validator
  rule has tightened between Pinot versions.
- DROP TABLE no longer collapses ControllerApplicationException from
  tableTasksCleanup (e.g. BAD_REQUEST for active running tasks) into
  500. The first-failure status code is preserved end-to-end and the
  partial-failure error message lists both successful and failed
  variants with a recovery hint.
- PRIMARY KEY parser uses LOOKAHEAD(3)    instead
  of LOOKAHEAD(2), so a malformed PRIMARY KEY id (no parens) backtracks
  to the expected-TABLE_TYPE error path instead of producing a confusing
  inner-grammar error. Added a parser test covering the failure mode.
- DROP TABLE partial-failure path is now explicitly tested via the
  improved error message and status preservation.

MAJOR:
- CREATE TABLE schema rollback no longer races. Previously the catch
  branch read hasOfflineTable + hasRealtimeTable non-atomically before
  deleting the schema; a concurrent sibling CREATE could win between
  the two reads and have its schema deleted out from under it. Pinot
  already supports schema-outlives-table; we surface a hint pointing
  the operator at DELETE /schemas/{name} for permanent failures.
- validateTableConfig now distinguishes user errors
  (IllegalArgument/IllegalStateException → 400) from controller defects
  (any other exception → 500), so monitoring picks up internal failures
  instead of mislabeling them as malformed DDL.
- SHOW CREATE TABLE narrows the IllegalArgumentException catch to
  documented emission failures (unsupported column types, reserved-key
  collisions) and treats other RuntimeExceptions as 500 internal
  errors.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../src/main/codegen/includes/parserImpls.ftl |   2 +-
 .../pinot/sql/parsers/PinotDdlParserTest.java |  10 ++
 .../resources/PinotDdlRestletResource.java    | 125 +++++++++++-------
 3 files changed, 87 insertions(+), 50 deletions(-)

diff --git a/pinot-common/src/main/codegen/includes/parserImpls.ftl b/pinot-common/src/main/codegen/includes/parserImpls.ftl
index 856a230bd9a1..d2f5971a933c 100644
--- a/pinot-common/src/main/codegen/includes/parserImpls.ftl
+++ b/pinot-common/src/main/codegen/includes/parserImpls.ftl
@@ -135,7 +135,7 @@ SqlNode SqlPinotCreateTable() :
     [ LOOKAHEAD(3)    { ifNotExists = true; } ]
     name = CompoundIdentifier()
     columns = PinotColumnList()
-    [ LOOKAHEAD(2) primaryKeyColumns = PinotPrimaryKeyList() ]
+    [ LOOKAHEAD(3)    primaryKeyColumns = PinotPrimaryKeyList() ]
      
     tableType = PinotTableTypeLiteral()
     [  properties = PinotPropertyList() ]
diff --git a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
index 93b8a595c790..27d1d5f0aa26 100644
--- a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
@@ -316,6 +316,16 @@ public void createTableWithoutPrimaryKeyHasNullPkList() {
     assertNull(node.getPrimaryKeyColumns());
   }
 
+  @Test
+  public void createTableMissingPrimaryKeyParensRejected() {
+    // Without LPAREN after KEY, the LOOKAHEAD(3) gate must reject the optional PK clause and
+    // the parser must surface a clear "expected TABLE_TYPE" error rather than committing to
+    // the PinotPrimaryKeyList production and emitting a confusing inner-grammar error.
+    expectThrows(SqlCompilationException.class,
+        () -> CalciteSqlParser.compileToSqlNodeAndOptions(
+            "CREATE TABLE t (id INT) PRIMARY KEY id TABLE_TYPE = OFFLINE"));
+  }
+
   @Test
   public void dimensionArrayParsedAsMultiValue() {
     SqlPinotCreateTable stmt = parseCreate(
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index bff76b9477fa..bafe53b0993b 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -245,16 +245,12 @@ private Response executeCreate(CompiledCreateTable create, String database,
         .setTableConfig(toJson(create.getTableConfig()))
         .setWarnings(create.getWarnings());
 
-    // Run the full schema/table validation stack that the existing /tables and /tableConfigs APIs
-    // apply before any ZK write. This catches invalid combinations (upsert without primary keys,
-    // field configs referencing non-existent columns, task configs with bad column references,
-    // etc.) that the compiler alone cannot detect. Runs for both dry-run and live create.
-    // Validation runs BEFORE the existence check so that IF NOT EXISTS is a no-op only for
-    // semantically valid statements; invalid DDL is always rejected regardless of IF NOT EXISTS.
-    validateTableConfig(create.getSchema(), create.getTableConfig());
-
-    // Check existence for both live and dry-run paths: dry-run must faithfully predict conflicts
-    // so callers can use it for pre-deployment validation ("would this DDL succeed?").
+    // Existence check runs first so CREATE TABLE IF NOT EXISTS is a successful no-op when the
+    // table already exists, matching the SQL-standard semantics (PostgreSQL, MySQL, SQLite,
+    // Snowflake, Trino, BigQuery): a deployment script that re-runs the same DDL against an
+    // existing table must succeed even if the column list or properties have drifted from the
+    // current stored config. Without this ordering, idempotent provisioning scripts break the
+    // moment a config-validator rule changes between Pinot versions.
     if (_pinotHelixResourceManager.hasTable(tableNameWithType)) {
       if (create.isIfNotExists()) {
         response.setMessage("Table " + tableNameWithType + " already exists; CREATE IF NOT EXISTS is a no-op.");
@@ -264,6 +260,12 @@ private Response executeCreate(CompiledCreateTable create, String database,
           "Table " + tableNameWithType + " already exists.", Response.Status.CONFLICT);
     }
 
+    // Run the full schema/table validation stack that the existing /tables and /tableConfigs APIs
+    // apply before any ZK write. This catches invalid combinations (upsert without primary keys,
+    // field configs referencing non-existent columns, task configs with bad column references,
+    // etc.) that the compiler alone cannot detect. Runs for both dry-run and live create.
+    validateTableConfig(create.getSchema(), create.getTableConfig());
+
     if (dryRun) {
       response.setMessage("Dry run: validated CREATE TABLE without persisting.");
       return Response.ok(response).build();
@@ -306,48 +308,35 @@ private Response executeCreate(CompiledCreateTable create, String database,
       LOGGER.info("DDL created table {}", tableNameWithType);
       return Response.status(Response.Status.CREATED).entity(response).build();
     } catch (TableAlreadyExistsException e) {
-      // Race: another caller added the table between our hasTable check and addTable.
-      // Only roll back the schema we created here if no table for this raw name now exists —
-      // if another caller won the race and its table is live, removing the schema would
-      // orphan that table. Re-check existence with the winner's table still present before
-      // deciding to clean up.
-      if (schemaCreatedHere && !_pinotHelixResourceManager.hasTable(tableNameWithType)) {
-        rollbackSchemaIfCreated(schemaName, true);
-      }
+      // Race: another caller added the table between our hasTable check and addTable. Do NOT
+      // roll back the schema here — the winner of the race may be using the same shared schema
+      // (legitimate hybrid-pair pattern), and a hasTable re-check followed by deleteSchema is
+      // racy in the same way the generic-failure branch below is. Stale schemas can be removed
+      // via DELETE /schemas/{name} if needed.
       throw new ControllerApplicationException(LOGGER, e.getMessage(), Response.Status.CONFLICT, e);
     } catch (SchemaAlreadyExistsException e) {
       // The override=false addSchema call lost a race with another schema writer. Surface 409.
       throw new ControllerApplicationException(LOGGER, e.getMessage(), Response.Status.CONFLICT, e);
     } catch (Exception e) {
-      // Only roll back the schema if no table for this raw name now exists. A concurrent request
-      // may have added the sibling physical variant (OFFLINE vs REALTIME) between our addSchema()
-      // and this addTable() failure; deleting the schema would orphan that live table. Check both
-      // typed variants before deciding to remove the shared schema.
-      if (schemaCreatedHere) {
-        boolean offlineNowExists = _pinotHelixResourceManager.hasOfflineTable(schemaName);
-        boolean realtimeNowExists = _pinotHelixResourceManager.hasRealtimeTable(schemaName);
-        rollbackSchemaIfCreated(schemaName, !offlineNowExists && !realtimeNowExists);
-      }
+      // Intentionally do NOT roll back the schema on a generic addTable() failure. The two
+      // hasOfflineTable/hasRealtimeTable reads required to decide "is this schema orphaned?"
+      // are non-atomic, and a concurrent sibling CREATE that succeeds between the two reads
+      // would have its schema deleted out from under it — orphaning a live table. Pinot's
+      // existing /tables endpoint also leaves the schema in place when table creation fails,
+      // so the contract is consistent: schemas can outlive tables, and stale schemas can be
+      // removed via DELETE /schemas/{name}. Surface a hint so the operator knows the schema
+      // remains and how to clean it up if the failure is genuinely permanent.
+      String hint = schemaCreatedHere
+          ? " (schema '" + schemaName + "' was created and remains; remove via DELETE /schemas/"
+              + schemaName + " if the failure is permanent and no other variant uses it)"
+          : "";
       // ControllerApplicationException(LOGGER, ...) logs the exception, so don't double-log here.
       throw new ControllerApplicationException(LOGGER,
-          "Failed to create table " + tableNameWithType + ": " + e.getMessage(),
+          "Failed to create table " + tableNameWithType + ": " + e.getMessage() + hint,
           Response.Status.INTERNAL_SERVER_ERROR, e);
     }
   }
 
-  private void rollbackSchemaIfCreated(String schemaName, boolean schemaCreatedHere) {
-    if (!schemaCreatedHere) {
-      return;
-    }
-    try {
-      _pinotHelixResourceManager.deleteSchema(schemaName);
-      LOGGER.info("Rolled back schema {} after failed table create", schemaName);
-    } catch (Exception rollbackEx) {
-      LOGGER.error("Failed to roll back schema {} after failed table create; manual cleanup "
-          + "may be required", schemaName, rollbackEx);
-    }
-  }
-
   /**
    * Compares two schemas by the column-shape attributes that a DDL column list actually controls
    * (column name, data type, field type, single/multi-value, NOT NULL, default null value, and —
@@ -436,9 +425,19 @@ private void validateTableConfig(Schema schema, TableConfig tableConfig) {
       TableConfigValidatorRegistry.validate(tableConfig, schema);
     } catch (ControllerApplicationException e) {
       throw e;
-    } catch (Exception e) {
+    } catch (IllegalArgumentException | IllegalStateException e) {
+      // The Pinot validators consistently raise these for user-facing config errors
+      // (upsert without primary keys, field configs referencing non-existent columns,
+      // bad task configs, etc.). Surface as 400 — the caller can fix their DDL.
       throw new ControllerApplicationException(LOGGER,
           "Table config validation failed: " + e.getMessage(), Response.Status.BAD_REQUEST, e);
+    } catch (Exception e) {
+      // Any other exception is unexpected — most likely a controller-side defect (e.g. NPE in
+      // a registry validator, ZK connectivity blip during validateTableTenantConfig). 400 would
+      // mislead the caller into "fix your DDL"; surface as 500 so monitoring picks it up.
+      throw new ControllerApplicationException(LOGGER,
+          "Internal error during table config validation: " + e.getMessage(),
+          Response.Status.INTERNAL_SERVER_ERROR, e);
     }
   }
 
@@ -529,20 +528,32 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
     List dropped = new ArrayList<>();
     List failed = new ArrayList<>();
     Exception firstFailure = null;
+    Response.Status firstFailureStatus = null;
     for (String target : targets) {
       try {
         // Remove task schedules before deletion so tasks are not triggered during the drop.
+        // tableTasksCleanup may throw ControllerApplicationException (e.g. BAD_REQUEST when
+        // active tasks are still running). That status code carries actionable user-level
+        // information and must be preserved instead of being collapsed to 500 below.
         PinotTableRestletResource.tableTasksCleanup(target, false,
             _pinotHelixResourceManager, _pinotHelixTaskResourceManager);
         TableType type = TableNameBuilder.getTableTypeFromTableName(target);
         _pinotHelixResourceManager.deleteTable(fullyQualifiedRaw, type, null);
         dropped.add(target);
         LOGGER.info("DDL dropped table {}", target);
+      } catch (ControllerApplicationException e) {
+        LOGGER.error("Failed to drop table {} during DDL DROP TABLE", target, e);
+        failed.add(target);
+        if (firstFailure == null) {
+          firstFailure = e;
+          firstFailureStatus = Response.Status.fromStatusCode(e.getResponse().getStatus());
+        }
       } catch (Exception e) {
         LOGGER.error("Failed to drop table {} during DDL DROP TABLE", target, e);
         failed.add(target);
         if (firstFailure == null) {
           firstFailure = e;
+          firstFailureStatus = Response.Status.INTERNAL_SERVER_ERROR;
         }
       }
     }
@@ -556,13 +567,18 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
       return response;
     }
     // At least one target failed. Surface a structured error that names both what succeeded
-    // and what failed so the operator knows which variant needs manual cleanup.
+    // and what failed so the operator knows which variant needs manual cleanup. Preserve the
+    // first failure's status code so client-actionable failures (e.g. active tasks → 400)
+    // remain visible to the caller; only fall back to 500 for genuinely unexpected failures.
     String causeDesc = firstFailure.getMessage() != null
         ? firstFailure.getMessage() : firstFailure.getClass().getSimpleName();
-    String msg = "Partial DROP TABLE: dropped " + dropped + ", failed to drop " + failed
-        + ": " + causeDesc + ".";
+    String partialPrefix = dropped.isEmpty() ? "" : "Partial DROP TABLE: dropped " + dropped + ", ";
+    String msg = partialPrefix + "failed to drop " + failed + ": " + causeDesc
+        + (dropped.isEmpty() ? "" : ". The successfully-dropped variant must be re-created if "
+            + "the original DROP was unintended.");
     throw new ControllerApplicationException(LOGGER, msg,
-        Response.Status.INTERNAL_SERVER_ERROR, firstFailure);
+        firstFailureStatus == null ? Response.Status.INTERNAL_SERVER_ERROR : firstFailureStatus,
+        firstFailure);
   }
 
   // -------------------------------------------------------------------------------------------
@@ -647,13 +663,24 @@ private DdlExecutionResponse executeShowCreate(CompiledShowCreateTable show, Str
     try {
       ddl = CanonicalDdlEmitter.emit(schema, tableConfig, database);
     } catch (IllegalArgumentException e) {
-      // SchemaEmitter.validateEmittable() throws IllegalArgumentException for unsupported column
-      // types (MAP, LIST, STRUCT, UNKNOWN) that the current DDL grammar cannot represent. Return
-      // 400 so the caller sees a deterministic client error rather than a 500.
+      // The emitter explicitly throws IllegalArgumentException to signal "this schema or config
+      // cannot be expressed in the current DDL grammar":
+      //   - SchemaEmitter.validateEmittable: unsupported column types (MAP, LIST, STRUCT, UNKNOWN)
+      //   - PropertyExtractor: TableCustomConfig key collides with a reserved DDL property name
+      // Both are caller-actionable: rename the offending column/property or use the JSON API for
+      // SHOW. Surface as 400 so the caller sees a deterministic client error rather than a 500.
       throw new ControllerApplicationException(LOGGER,
           "SHOW CREATE TABLE is not supported for table " + tableNameWithType
               + ": " + e.getMessage(),
           Response.Status.BAD_REQUEST, e);
+    } catch (RuntimeException e) {
+      // Anything else from emit() is an unexpected controller-side failure (NPE in extractor,
+      // JsonProcessingException wrapped as runtime, etc.). 400 would mislead the caller —
+      // surface as 500 so monitoring picks it up.
+      throw new ControllerApplicationException(LOGGER,
+          "Internal error rendering SHOW CREATE TABLE for " + tableNameWithType
+              + ": " + e.getMessage(),
+          Response.Status.INTERNAL_SERVER_ERROR, e);
     }
 
     return new DdlExecutionResponse()

From 03673aef29d4051d2456d391a92b5e64a6dd644e Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Fri, 24 Apr 2026 23:42:05 -0700
Subject: [PATCH 11/32] Address final review: stop double-logging in DROP
 catch, strengthen parser test assertion
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- Downgrade the per-target log inside the typed ControllerApplicationException
  catch in executeDrop from ERROR with throwable to WARN with just the
  message. The CAE constructor already logged the underlying error, and
  the wrapping CAE thrown after the loop logs again — three entries for
  one user-actionable 4xx is noise (C7.10).
- Strengthen createTableMissingPrimaryKeyParensRejected to assert the
  parser error message names TABLE_TYPE, locking in the expected error
  fragment so a regression in the LOOKAHEAD widening cannot pass the
  test silently (C6.5).

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../apache/pinot/sql/parsers/PinotDdlParserTest.java   |  9 +++++++--
 .../api/resources/PinotDdlRestletResource.java         | 10 ++++++++--
 2 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
index 27d1d5f0aa26..6918a22da40b 100644
--- a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
@@ -320,10 +320,15 @@ public void createTableWithoutPrimaryKeyHasNullPkList() {
   public void createTableMissingPrimaryKeyParensRejected() {
     // Without LPAREN after KEY, the LOOKAHEAD(3) gate must reject the optional PK clause and
     // the parser must surface a clear "expected TABLE_TYPE" error rather than committing to
-    // the PinotPrimaryKeyList production and emitting a confusing inner-grammar error.
-    expectThrows(SqlCompilationException.class,
+    // the PinotPrimaryKeyList production and emitting a confusing inner-grammar error. A
+    // regression that drops the LOOKAHEAD widening would commit to the PK production, fail at
+    // LPAREN, and surface "Encountered \"id\"" rather than the expected TABLE_TYPE token —
+    // verifying the message contains TABLE_TYPE locks in the desired behavior.
+    SqlCompilationException ex = expectThrows(SqlCompilationException.class,
         () -> CalciteSqlParser.compileToSqlNodeAndOptions(
             "CREATE TABLE t (id INT) PRIMARY KEY id TABLE_TYPE = OFFLINE"));
+    assertTrue(ex.getMessage() != null && ex.getMessage().contains("TABLE_TYPE"),
+        "expected error to point at the missing TABLE_TYPE token, got: " + ex.getMessage());
   }
 
   @Test
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index bafe53b0993b..01a4d844b90f 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -542,14 +542,20 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
         dropped.add(target);
         LOGGER.info("DDL dropped table {}", target);
       } catch (ControllerApplicationException e) {
-        LOGGER.error("Failed to drop table {} during DDL DROP TABLE", target, e);
+        // The CAE constructor already logs the underlying error at the appropriate level, and
+        // the wrapping CAE thrown after the loop will log again. A third log here would be
+        // redundant noise — record a one-line breadcrumb at WARN without the throwable.
+        LOGGER.warn("DROP TABLE on {} failed: {}", target, e.getMessage());
         failed.add(target);
         if (firstFailure == null) {
           firstFailure = e;
           firstFailureStatus = Response.Status.fromStatusCode(e.getResponse().getStatus());
         }
       } catch (Exception e) {
-        LOGGER.error("Failed to drop table {} during DDL DROP TABLE", target, e);
+        // Genuinely unexpected errors get full stack traces — the wrapping CAE below will also
+        // log, but for arbitrary RuntimeExceptions / Helix failures the duplicate is acceptable
+        // because the operator may need both the per-target context and the aggregated message.
+        LOGGER.error("DROP TABLE on {} failed unexpectedly", target, e);
         failed.add(target);
         if (firstFailure == null) {
           firstFailure = e;

From 1db4dfe43568c3e8ec6e5924e88b407a1282234a Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Fri, 24 Apr 2026 23:58:54 -0700
Subject: [PATCH 12/32] Fix one CRITICAL and five MAJOR DDL review issues from
 third-pass review

CRITICAL:
- SHOW CREATE TABLE no-TYPE branch now authorizes only the variant(s)
  it would actually surface, not both unconditionally. The previous
  "authorize both up-front" pattern caused a regression for callers
  with READ on only one variant: they would get 403 even when only
  the authorized variant existed and the read should succeed. The
  no-fingerprinting goal (don't let unauthorized callers distinguish
  exists-but-forbidden from not-found) is preserved by running an
  auth check against the OFFLINE candidate before surfacing 404.

MAJOR:
- CREATE TABLE for the second hybrid variant now validates against
  the stored schema (which carries the canonical primary keys, tags,
  null-handling) rather than the DDL-compiled schema. Previously an
  upsert table whose first variant declared PRIMARY KEY would fail
  validation when the second variant's DDL omitted the PK clause,
  even though describeColumnShapeMismatch correctly accepted the
  metadata difference.
- SHOW CREATE TABLE: when hasTable=true but getTableConfig returns
  null, surface 500 with a "torn write or concurrent delete" message
  instead of masking ZK inconsistency as 404.
- DdlCompiler.extractLiteralValue now throws DdlCompilationException
  for non-SqlLiteral inputs instead of silently calling toString(),
  preventing a future grammar relaxation from leaking SQL-wire
  quoting into FieldSpec.defaultNullValue.
- PropertyMapping.applyPromoted now eagerly validates
  replicasPerPartition as an integer, mirroring the replication path.
- DROP TABLE unexpected-Exception branch logs a one-line breadcrumb
  at WARN instead of a full stack trace; the wrapping CAE thrown
  after the loop already logs the firstFailure with full context.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../resources/PinotDdlRestletResource.java    | 96 ++++++++++++-------
 .../pinot/sql/ddl/compile/DdlCompiler.java    | 19 ++--
 .../sql/ddl/compile/PropertyMapping.java      |  5 +-
 3 files changed, 77 insertions(+), 43 deletions(-)

diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 01a4d844b90f..509dc2f002db 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -260,23 +260,14 @@ private Response executeCreate(CompiledCreateTable create, String database,
           "Table " + tableNameWithType + " already exists.", Response.Status.CONFLICT);
     }
 
-    // Run the full schema/table validation stack that the existing /tables and /tableConfigs APIs
-    // apply before any ZK write. This catches invalid combinations (upsert without primary keys,
-    // field configs referencing non-existent columns, task configs with bad column references,
-    // etc.) that the compiler alone cannot detect. Runs for both dry-run and live create.
-    validateTableConfig(create.getSchema(), create.getTableConfig());
-
-    if (dryRun) {
-      response.setMessage("Dry run: validated CREATE TABLE without persisting.");
-      return Response.ok(response).build();
-    }
-
-    // When a schema for this raw table name already exists (the common case when adding the
-    // second physical variant of a hybrid pair), verify the DDL column list is equivalent to
-    // the stored schema. Silently accepting a mismatched column list would create a table whose
-    // runtime schema differs from what the DDL described.
+    // Look up the stored schema first. When the second physical variant of a hybrid pair is
+    // created, the stored schema is the canonical source of metadata (primary keys, tags,
+    // null-handling) that the DDL column list does not itself express. Validating against the
+    // stored schema ensures upsert/dedup PK checks see the real PK list rather than a
+    // synthesized empty one from the DDL-compiled schema.
     Schema storedSchema = _pinotHelixResourceManager.getSchema(schemaName);
     boolean schemaPreexisted = storedSchema != null;
+
     if (schemaPreexisted) {
       // Compare only the column-shape attributes that the DDL column list actually controls.
       // Comparing full JSON would include schema-level metadata (primary keys, null-handling,
@@ -294,6 +285,22 @@ private Response executeCreate(CompiledCreateTable create, String database,
       }
     }
 
+    // Run the full schema/table validation stack that the existing /tables and /tableConfigs APIs
+    // apply before any ZK write. This catches invalid combinations (upsert without primary keys,
+    // field configs referencing non-existent columns, task configs with bad column references,
+    // etc.) that the compiler alone cannot detect. Runs for both dry-run and live create.
+    // When a stored schema exists (hybrid second-variant case), validate against the stored
+    // schema so the validators see the canonical PK list / null-handling / tags rather than the
+    // DDL's column-list-only projection — otherwise upsert/dedup tables would falsely fail PK
+    // validation when the DDL omits PRIMARY KEY in the second variant.
+    Schema schemaForValidation = schemaPreexisted ? storedSchema : create.getSchema();
+    validateTableConfig(schemaForValidation, create.getTableConfig());
+
+    if (dryRun) {
+      response.setMessage("Dry run: validated CREATE TABLE without persisting.");
+      return Response.ok(response).build();
+    }
+
     boolean schemaCreatedHere = false;
     try {
       // override=false: an existing schema with the same name is a precondition violation, not
@@ -552,10 +559,10 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
           firstFailureStatus = Response.Status.fromStatusCode(e.getResponse().getStatus());
         }
       } catch (Exception e) {
-        // Genuinely unexpected errors get full stack traces — the wrapping CAE below will also
-        // log, but for arbitrary RuntimeExceptions / Helix failures the duplicate is acceptable
-        // because the operator may need both the per-target context and the aggregated message.
-        LOGGER.error("DROP TABLE on {} failed unexpectedly", target, e);
+        // The wrapping CAE thrown after the loop will log the firstFailure with full stack;
+        // record only a one-line breadcrumb here so operators can correlate per-target context
+        // without seeing the same stack trace twice.
+        LOGGER.warn("DROP TABLE on {} failed unexpectedly: {}", target, e.toString());
         failed.add(target);
         if (firstFailure == null) {
           firstFailure = e;
@@ -616,46 +623,65 @@ private DdlExecutionResponse executeShowCreate(CompiledShowCreateTable show, Str
             "Table not found: " + tableNameWithType, Response.Status.NOT_FOUND);
       }
     } else {
-      // Authorize BOTH candidates before probing existence so the caller cannot infer which
-      // variant exists from a 403 vs 404 response, and so an auth failure on the realtime
-      // variant is not masked by a 404 produced before the check runs.
+      // Authorize ONLY the variant(s) we would actually surface, not both unconditionally.
+      // The previous "authorize both up-front" pattern caused a regression for the legitimate
+      // case where a caller has READ on OFFLINE only, the table exists only as OFFLINE, and
+      // the caller would have a valid 200 from `GET /tables/foo` — but the up-front REALTIME
+      // auth check threw 403, blocking a read they should be allowed to perform.
+      //
+      // The no-fingerprinting goal (don't let an unauthorized caller distinguish "exists but
+      // forbidden" from "not found") is preserved by:
+      //   - When neither variant exists, still run an auth check (against OFFLINE) before
+      //     returning 404 so an unauthorized caller gets 403 just like they would for an
+      //     existing-but-forbidden table.
+      //   - When both variants exist, authorize both before surfacing the disambiguation 400,
+      //     since the disambiguation message itself reveals the existence of both variants.
       String off = TableNameBuilder.OFFLINE.tableNameWithType(fullyQualifiedRaw);
       String rt = TableNameBuilder.REALTIME.tableNameWithType(fullyQualifiedRaw);
-      ResourceUtils.checkPermissionAndAccess(off, httpRequest, headers,
-          AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
-      ResourceUtils.checkPermissionAndAccess(rt, httpRequest, headers,
-          AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
       boolean offExists = _pinotHelixResourceManager.hasTable(off);
       boolean rtExists = _pinotHelixResourceManager.hasTable(rt);
       if (offExists && rtExists) {
-        // Both variants exist; silently picking one would return DDL for the wrong variant.
-        // Require the caller to disambiguate with an explicit TYPE clause.
+        ResourceUtils.checkPermissionAndAccess(off, httpRequest, headers,
+            AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
+        ResourceUtils.checkPermissionAndAccess(rt, httpRequest, headers,
+            AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
         throw new ControllerApplicationException(LOGGER,
             "Table '" + fullyQualifiedRaw + "' has both OFFLINE and REALTIME variants. "
                 + "Use 'SHOW CREATE TABLE ... TYPE OFFLINE' or 'TYPE REALTIME' to specify which.",
             Response.Status.BAD_REQUEST);
       } else if (offExists) {
+        ResourceUtils.checkPermissionAndAccess(off, httpRequest, headers,
+            AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
         tableNameWithType = off;
         resolvedType = TableType.OFFLINE;
       } else if (rtExists) {
+        ResourceUtils.checkPermissionAndAccess(rt, httpRequest, headers,
+            AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
         tableNameWithType = rt;
         resolvedType = TableType.REALTIME;
       } else {
+        // Neither variant exists. Run auth before 404 so an unauthorized probe returns 403
+        // (the same response they would have gotten for an existing-but-forbidden table) —
+        // no fingerprinting via 403-vs-404.
+        ResourceUtils.checkPermissionAndAccess(off, httpRequest, headers,
+            AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
         throw new ControllerApplicationException(LOGGER,
             "Table not found: " + fullyQualifiedRaw, Response.Status.NOT_FOUND);
       }
     }
 
-    org.apache.pinot.spi.config.table.TableConfig tableConfig =
-        _pinotHelixResourceManager.getTableConfig(tableNameWithType);
+    TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(tableNameWithType);
     if (tableConfig == null) {
-      // Should not happen — hasTable just succeeded — but treat as not-found rather than 500.
+      // hasTable returned true but getTableConfig returned null. This indicates ZK inconsistency
+      // (torn write or concurrent delete between the two reads), not a missing table from the
+      // caller's perspective. Surface as 500 so monitoring catches the inconsistency rather than
+      // mislead the caller into thinking their reference is wrong.
       throw new ControllerApplicationException(LOGGER,
-          "Table " + tableNameWithType + " disappeared during SHOW CREATE.",
-          Response.Status.NOT_FOUND);
+          "Table " + tableNameWithType + " has IdealState but no TableConfig in ZK; "
+              + "possible torn write or concurrent delete.",
+          Response.Status.INTERNAL_SERVER_ERROR);
     }
-    org.apache.pinot.spi.data.Schema schema =
-        _pinotHelixResourceManager.getTableSchema(tableNameWithType);
+    Schema schema = _pinotHelixResourceManager.getTableSchema(tableNameWithType);
     if (schema == null) {
       throw new ControllerApplicationException(LOGGER,
           "Schema not found for " + tableNameWithType
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
index 014d56c1f608..4df20930cf82 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
@@ -226,14 +226,19 @@ private static String extractLiteralValue(@Nullable SqlNode literal) {
     if (literal == null) {
       return null;
     }
-    if (literal instanceof SqlLiteral) {
-      try {
-        return ((SqlLiteral) literal).toValue();
-      } catch (UnsupportedOperationException e) {
-        throw new DdlCompilationException("Unsupported DEFAULT literal: " + literal, e);
-      }
+    if (!(literal instanceof SqlLiteral)) {
+      // The grammar's ` Literal()` production is currently guaranteed to produce a
+      // SqlLiteral; this guard is here so a future grammar relaxation cannot silently route
+      // a quoted-string toString() form into FieldSpec.defaultNullValue and cause downstream
+      // ingestion to compare against a wire-format value with embedded quotes.
+      throw new DdlCompilationException(
+          "DEFAULT requires a literal value; got: " + literal.getClass().getSimpleName());
+    }
+    try {
+      return ((SqlLiteral) literal).toValue();
+    } catch (UnsupportedOperationException e) {
+      throw new DdlCompilationException("Unsupported DEFAULT literal: " + literal, e);
     }
-    return literal.toString();
   }
 
   private static ColumnRole inferRole(SqlPinotColumnDeclaration col, DataType dt) {
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
index 6356500b4ae8..e6e65619a618 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
@@ -350,7 +350,10 @@ private static boolean applyPromoted(String lowerKey, String value, TableConfigB
       case "replicasperpartition":
         // TableConfigBuilder does not expose setReplicasPerPartition; DdlCompiler applies this
         // value post-build via tableConfig.getValidationConfig().setReplicasPerPartition().
-        // Return true to prevent fall-through to customConfigs.
+        // Validate as an integer here to fail fast at compile time with a clear error rather
+        // than letting a non-numeric value land in TableConfig and surface only at validation.
+        // Mirrors the eager parseInt validation done for "replication" above.
+        parseInt(lowerKey, value);
         return true;
       default:
         return false;

From 1b1c0ab20f14033fd7a875834cf4ede542acb8fc Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 00:12:27 -0700
Subject: [PATCH 13/32] Address fingerprint regression in SHOW CREATE auth:
 revert to up-front double-auth

The previous commit's per-variant auth-after-existence pattern preserved the
"legitimate one-variant reader" case but introduced a subtle fingerprinting
leak under access-control plugins that grant per-type permissions (the SPI
permits this and enterprise plugins use it). A caller with READ on OFFLINE
only could distinguish "REALTIME-only-exists" (403) from "neither-exists"
(404), revealing the existence of a REALTIME variant they shouldn't see.

Restore the up-front double-authorize pattern (matching the DROP TABLE
symmetry) and direct partial-permission callers to use the explicit
TYPE clause to read a single variant. The bare SHOW CREATE TABLE form
is intentionally more restrictive because answering it requires reading
both candidates' state.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../resources/PinotDdlRestletResource.java    | 45 ++++++++-----------
 1 file changed, 18 insertions(+), 27 deletions(-)

diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 509dc2f002db..8cdff30612a4 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -623,50 +623,41 @@ private DdlExecutionResponse executeShowCreate(CompiledShowCreateTable show, Str
             "Table not found: " + tableNameWithType, Response.Status.NOT_FOUND);
       }
     } else {
-      // Authorize ONLY the variant(s) we would actually surface, not both unconditionally.
-      // The previous "authorize both up-front" pattern caused a regression for the legitimate
-      // case where a caller has READ on OFFLINE only, the table exists only as OFFLINE, and
-      // the caller would have a valid 200 from `GET /tables/foo` — but the up-front REALTIME
-      // auth check threw 403, blocking a read they should be allowed to perform.
-      //
-      // The no-fingerprinting goal (don't let an unauthorized caller distinguish "exists but
-      // forbidden" from "not found") is preserved by:
-      //   - When neither variant exists, still run an auth check (against OFFLINE) before
-      //     returning 404 so an unauthorized caller gets 403 just like they would for an
-      //     existing-but-forbidden table.
-      //   - When both variants exist, authorize both before surfacing the disambiguation 400,
-      //     since the disambiguation message itself reveals the existence of both variants.
+      // No-TYPE form: authorize BOTH candidate variants before any existence-revealing branch.
+      // This is required to prevent fingerprinting under access-control plugins that grant
+      // per-type permissions (the SPI permits this and enterprise plugins use it). With both
+      // checks up-front:
+      //   - A caller without permission on either variant always gets 403, regardless of
+      //     existence.
+      //   - A caller with permission on only one variant must use SHOW CREATE TABLE ... TYPE
+      //     {OFFLINE|REALTIME} to read that single variant — the bare form is intentionally
+      //     more restrictive because answering it requires reading both candidates' state.
+      // This mirrors the DROP TABLE pattern (lines 469-474) for symmetry between read and
+      // delete operations.
       String off = TableNameBuilder.OFFLINE.tableNameWithType(fullyQualifiedRaw);
       String rt = TableNameBuilder.REALTIME.tableNameWithType(fullyQualifiedRaw);
+      ResourceUtils.checkPermissionAndAccess(off, httpRequest, headers,
+          AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
+      ResourceUtils.checkPermissionAndAccess(rt, httpRequest, headers,
+          AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
       boolean offExists = _pinotHelixResourceManager.hasTable(off);
       boolean rtExists = _pinotHelixResourceManager.hasTable(rt);
       if (offExists && rtExists) {
-        ResourceUtils.checkPermissionAndAccess(off, httpRequest, headers,
-            AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
-        ResourceUtils.checkPermissionAndAccess(rt, httpRequest, headers,
-            AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
         throw new ControllerApplicationException(LOGGER,
             "Table '" + fullyQualifiedRaw + "' has both OFFLINE and REALTIME variants. "
                 + "Use 'SHOW CREATE TABLE ... TYPE OFFLINE' or 'TYPE REALTIME' to specify which.",
             Response.Status.BAD_REQUEST);
       } else if (offExists) {
-        ResourceUtils.checkPermissionAndAccess(off, httpRequest, headers,
-            AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
         tableNameWithType = off;
         resolvedType = TableType.OFFLINE;
       } else if (rtExists) {
-        ResourceUtils.checkPermissionAndAccess(rt, httpRequest, headers,
-            AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
         tableNameWithType = rt;
         resolvedType = TableType.REALTIME;
       } else {
-        // Neither variant exists. Run auth before 404 so an unauthorized probe returns 403
-        // (the same response they would have gotten for an existing-but-forbidden table) —
-        // no fingerprinting via 403-vs-404.
-        ResourceUtils.checkPermissionAndAccess(off, httpRequest, headers,
-            AccessType.READ, Actions.Table.GET_TABLE_CONFIG, _accessControlFactory, LOGGER);
         throw new ControllerApplicationException(LOGGER,
-            "Table not found: " + fullyQualifiedRaw, Response.Status.NOT_FOUND);
+            "Table not found: " + fullyQualifiedRaw + ". If you only have permission for "
+                + "one variant, use 'SHOW CREATE TABLE ... TYPE OFFLINE' or 'TYPE REALTIME'.",
+            Response.Status.NOT_FOUND);
       }
     }
 

From bd9efb1ef3128f9e65a8d47c999bf57b4260e552 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 00:27:55 -0700
Subject: [PATCH 14/32] Fix CRITICAL parser bug + two MAJOR correctness issues
 from fourth review pass

CRITICAL:
- PRIMARY KEY parser was double-consuming   : my
  earlier LOOKAHEAD widening had explicit token consumption at the call
  site that conflicted with the rule body, breaking every CREATE TABLE
  ... PRIMARY KEY (...) statement. The advertised PRIMARY KEY support
  has been dead since LOOKAHEAD(3) was introduced.
  Fix: drop the inline tokens at the call site; the LOOKAHEAD(3)
  integer alone correctly peeks at the next 3 tokens of the production
  (which are   ) without consuming them. The
  previously-failing createTableWithPrimaryKey,
  createTableWithCompositePrimaryKey, and
  createTableMissingPrimaryKeyParensRejected tests now pass.

MAJOR:
- describeColumnShapeMismatch now compares typed default values via
  Objects.equals(getDefaultNullValue(), ...) rather than
  getDefaultNullValueString(). The string form produces false 0 vs
  0.0 mismatches when the same numeric default arrived via different
  literal forms (DDL "DEFAULT 0.0" vs JSON-API integer 0); the typed
  projection collapses them through FieldSpec's data-type parsing.
- Added a clarifying comment at the deleteTable call site documenting
  that the helper takes the raw name + explicit TableType and derives
  the typed name internally via TableNameBuilder, addressing reviewer
  uncertainty about the calling convention.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../src/main/codegen/includes/parserImpls.ftl     |  2 +-
 .../api/resources/PinotDdlRestletResource.java    | 15 ++++++++++-----
 2 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/pinot-common/src/main/codegen/includes/parserImpls.ftl b/pinot-common/src/main/codegen/includes/parserImpls.ftl
index d2f5971a933c..1fcf0daa81c1 100644
--- a/pinot-common/src/main/codegen/includes/parserImpls.ftl
+++ b/pinot-common/src/main/codegen/includes/parserImpls.ftl
@@ -135,7 +135,7 @@ SqlNode SqlPinotCreateTable() :
     [ LOOKAHEAD(3)    { ifNotExists = true; } ]
     name = CompoundIdentifier()
     columns = PinotColumnList()
-    [ LOOKAHEAD(3)    primaryKeyColumns = PinotPrimaryKeyList() ]
+    [ LOOKAHEAD(3) primaryKeyColumns = PinotPrimaryKeyList() ]
      
     tableType = PinotTableTypeLiteral()
     [  properties = PinotPropertyList() ]
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 8cdff30612a4..e5e05e49493a 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -391,11 +391,12 @@ static String describeColumnShapeMismatch(Schema stored, Schema compiled) {
         return "column '" + columnName + "' NOT NULL flag differs (stored=" + storedSpec.isNotNull()
             + ", DDL=" + compiledSpec.isNotNull() + ")";
       }
-      // Compare default null value by its string form: the DDL always sets defaults from a
-      // string literal, and FieldSpec normalizes the stored representation to a typed value.
-      // The string form is the stable projection that survives both serialization round trips.
-      String storedDefault = storedSpec.getDefaultNullValueString();
-      String compiledDefault = compiledSpec.getDefaultNullValueString();
+      // Compare typed default null value (after FieldSpec has parsed the string form into the
+      // column's data type). Comparing string forms produces false mismatches like "0" vs "0.0"
+      // when the same numeric default arrived via different literal forms (DDL "DEFAULT 0.0"
+      // vs JSON-API "defaultNullValue": 0); the typed projection collapses them.
+      Object storedDefault = storedSpec.getDefaultNullValue();
+      Object compiledDefault = compiledSpec.getDefaultNullValue();
       if (!Objects.equals(storedDefault, compiledDefault)) {
         return "column '" + columnName + "' default null value differs (stored=" + storedDefault
             + ", DDL=" + compiledDefault + ")";
@@ -544,6 +545,10 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
         // information and must be preserved instead of being collapsed to 500 below.
         PinotTableRestletResource.tableTasksCleanup(target, false,
             _pinotHelixResourceManager, _pinotHelixTaskResourceManager);
+        // deleteTable(rawName, type, retention) takes the raw name and re-derives the typed
+        // name internally via TableNameBuilder.forType(type).tableNameWithType(rawName); see
+        // PinotHelixResourceManager.deleteTable. Pass `fullyQualifiedRaw` (DB-qualified raw
+        // name) and the type extracted from `target` so the call is unambiguous.
         TableType type = TableNameBuilder.getTableTypeFromTableName(target);
         _pinotHelixResourceManager.deleteTable(fullyQualifiedRaw, type, null);
         dropped.add(target);

From 973af1dbf86fa0ecc8bc1d29a6da7142803ffdbe Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 00:38:20 -0700
Subject: [PATCH 15/32] Fix CRITICAL BYTES default-null-value comparison
 regression

The previous commit's switch to typed Objects.equals on getDefaultNullValue()
broke BYTES columns: getDefaultNullValue() allocates a fresh byte[] on each
call (FieldSpec dataType.convert), so Objects.equals does reference
comparison, not content comparison, falsely flagging every hybrid
second-variant CREATE on a BYTES column with a custom default as a
mismatch.

Fix: use FieldSpec.DataType.equals(v1, v2) which delegates to Arrays.equals
for BYTES and to value.equals for other types. Added two regression tests:
matching BYTES defaults must accept; differing BYTES defaults must reject.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../resources/PinotDdlRestletResource.java    | 21 ++++++---
 .../PinotDdlRestletResourceUnitTest.java      | 43 +++++++++++++++++++
 2 files changed, 57 insertions(+), 7 deletions(-)

diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index e5e05e49493a..670246c6d8f3 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -391,15 +391,22 @@ static String describeColumnShapeMismatch(Schema stored, Schema compiled) {
         return "column '" + columnName + "' NOT NULL flag differs (stored=" + storedSpec.isNotNull()
             + ", DDL=" + compiledSpec.isNotNull() + ")";
       }
-      // Compare typed default null value (after FieldSpec has parsed the string form into the
-      // column's data type). Comparing string forms produces false mismatches like "0" vs "0.0"
-      // when the same numeric default arrived via different literal forms (DDL "DEFAULT 0.0"
-      // vs JSON-API "defaultNullValue": 0); the typed projection collapses them.
+      // Compare typed default null value via DataType.equals, which delegates to Arrays.equals
+      // for BYTES (a fresh byte[] is allocated on every getDefaultNullValue() call, so a plain
+      // Objects.equals would do reference comparison and falsely flag every BYTES default as
+      // a mismatch). For other types, DataType.equals delegates to the boxed value's equals.
       Object storedDefault = storedSpec.getDefaultNullValue();
       Object compiledDefault = compiledSpec.getDefaultNullValue();
-      if (!Objects.equals(storedDefault, compiledDefault)) {
-        return "column '" + columnName + "' default null value differs (stored=" + storedDefault
-            + ", DDL=" + compiledDefault + ")";
+      boolean defaultsEqual;
+      if (storedDefault == null || compiledDefault == null) {
+        defaultsEqual = (storedDefault == null && compiledDefault == null);
+      } else {
+        defaultsEqual = storedSpec.getDataType().equals(storedDefault, compiledDefault);
+      }
+      if (!defaultsEqual) {
+        return "column '" + columnName + "' default null value differs (stored="
+            + storedSpec.getDefaultNullValueString()
+            + ", DDL=" + compiledSpec.getDefaultNullValueString() + ")";
       }
       if (storedSpec instanceof DateTimeFieldSpec && compiledSpec instanceof DateTimeFieldSpec) {
         DateTimeFieldSpec storedDt = (DateTimeFieldSpec) storedSpec;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
index bddf5cbfa25f..c94d2d8484c0 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
@@ -143,6 +143,49 @@ public void rejectsNotNullFlagMismatch() {
     assertTrue(msg.contains("NOT NULL flag"), msg);
   }
 
+  /**
+   * BYTES columns produce a fresh byte[] on each getDefaultNullValue() call. Equality must be
+   * by content, not reference, otherwise every hybrid second-variant CREATE on a BYTES schema
+   * with a custom default would falsely trip the column-shape mismatch.
+   */
+  @Test
+  public void acceptsMatchingBytesDefaultNullValue() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    FieldSpec storedSpec = new DimensionFieldSpec("blob", DataType.BYTES, true);
+    storedSpec.setDefaultNullValue("deadbeef");
+    stored.addField(storedSpec);
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    FieldSpec compiledSpec = new DimensionFieldSpec("blob", DataType.BYTES, true);
+    compiledSpec.setDefaultNullValue("deadbeef");
+    compiled.addField(compiledSpec);
+
+    assertNull(PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled),
+        "matching BYTES defaults must not trip a content-vs-reference equality regression");
+  }
+
+  /** BYTES default mismatch must still be rejected with content-aware comparison. */
+  @Test
+  public void rejectsBytesDefaultNullValueMismatch() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    FieldSpec storedSpec = new DimensionFieldSpec("blob", DataType.BYTES, true);
+    storedSpec.setDefaultNullValue("deadbeef");
+    stored.addField(storedSpec);
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    FieldSpec compiledSpec = new DimensionFieldSpec("blob", DataType.BYTES, true);
+    compiledSpec.setDefaultNullValue("cafebabe");
+    compiled.addField(compiledSpec);
+
+    String msg = PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled);
+    assertNotNull(msg);
+    assertTrue(msg.contains("default null value"), msg);
+  }
+
   /** Default-null-value mismatch must be rejected. */
   @Test
   public void rejectsDefaultNullValueMismatch() {

From 2b8a64c7d6e6c87b5ddf05850e685d6d88bd1307 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 00:56:50 -0700
Subject: [PATCH 16/32] Fix CRITICAL BYTES emitter regression + 4 MAJOR DDL
 issues from fifth review pass
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

CRITICAL:
- SchemaEmitter.emitColumn used Object.equals to compare default value
  against the natural default. For BYTES columns, both sides are byte[]
  whose .equals() is reference equality, so every BYTES column at
  natural default was emitting a redundant DEFAULT '' clause —
  same bug pattern that was just fixed in describeColumnShapeMismatch.
  Fix: use FieldSpec.DataType.equals(v1, v2), which delegates to
  Arrays.equals for BYTES.

MAJOR:
- BIG_DECIMAL DEFAULT now emits BigDecimal.toPlainString() instead of
  toString(); the latter can emit scientific notation (1E+10) which
  Calcite's Literal() rule does not accept, breaking round-trip for
  large or small magnitudes.
- DROP TABLE failure aggregation now tracks the integer status code
  rather than the Response.Status enum. fromStatusCode returns null
  for non-standard codes (422, 423, 451), which would silently fall
  back to 500 and hide the original 4xx information.
- DEFAULT NULL is now rejected explicitly with DdlCompilationException
  instead of silently behaving as if no DEFAULT was supplied.
  SqlLiteral.toValue returns null for SqlLiteral.createNull, and the
  previous code path stored that as the column's defaultValue then
  guarded against it later — a meaningless DDL became a silent no-op.
- Removed the schema-cleanup hint from the 500-error message in the
  CREATE TABLE generic-failure path. The hint pointed callers at
  DELETE /schemas/{name}, encouraging premature cleanup in response
  to transient ZK/Helix failures. The schema persists either way (per
  the existing /tables contract) and operators can investigate via
  logs and metrics.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../resources/PinotDdlRestletResource.java    | 31 ++++++++++---------
 .../pinot/sql/ddl/compile/DdlCompiler.java    | 15 ++++++++-
 .../pinot/sql/ddl/reverse/SchemaEmitter.java  | 19 ++++++++++--
 3 files changed, 46 insertions(+), 19 deletions(-)

diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 670246c6d8f3..b3bc1bf6c872 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -301,14 +301,12 @@ private Response executeCreate(CompiledCreateTable create, String database,
       return Response.ok(response).build();
     }
 
-    boolean schemaCreatedHere = false;
     try {
       // override=false: an existing schema with the same name is a precondition violation, not
       // something we silently overwrite. The other-typed table variant or another caller's
       // schema would otherwise be clobbered out from under them.
       if (!schemaPreexisted) {
         _pinotHelixResourceManager.addSchema(create.getSchema(), false, false);
-        schemaCreatedHere = true;
       }
       _pinotHelixResourceManager.addTable(create.getTableConfig());
       response.setMessage("Successfully created table " + tableNameWithType);
@@ -331,15 +329,16 @@ private Response executeCreate(CompiledCreateTable create, String database,
       // would have its schema deleted out from under it — orphaning a live table. Pinot's
       // existing /tables endpoint also leaves the schema in place when table creation fails,
       // so the contract is consistent: schemas can outlive tables, and stale schemas can be
-      // removed via DELETE /schemas/{name}. Surface a hint so the operator knows the schema
-      // remains and how to clean it up if the failure is genuinely permanent.
-      String hint = schemaCreatedHere
-          ? " (schema '" + schemaName + "' was created and remains; remove via DELETE /schemas/"
-              + schemaName + " if the failure is permanent and no other variant uses it)"
-          : "";
+      // removed via DELETE /schemas/{name}.
+      //
+      // Don't append a "remove via DELETE /schemas" hint here: arbitrary RuntimeException /
+      // IOException from addTable can be a transient ZK or Helix blip, and pointing the
+      // operator at a destructive schema-deletion command in response to a transient failure
+      // would encourage premature cleanup. The error message is logged with the original
+      // cause so an operator can investigate via log/metrics.
       // ControllerApplicationException(LOGGER, ...) logs the exception, so don't double-log here.
       throw new ControllerApplicationException(LOGGER,
-          "Failed to create table " + tableNameWithType + ": " + e.getMessage() + hint,
+          "Failed to create table " + tableNameWithType + ": " + e.getMessage(),
           Response.Status.INTERNAL_SERVER_ERROR, e);
     }
   }
@@ -543,7 +542,11 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
     List dropped = new ArrayList<>();
     List failed = new ArrayList<>();
     Exception firstFailure = null;
-    Response.Status firstFailureStatus = null;
+    // Track the integer status code rather than the Response.Status enum: the enum
+    // covers only the well-known IANA codes, and Response.Status.fromStatusCode() returns
+    // null for non-standard codes (422, 423, 451, etc.) — a null would silently fall back
+    // to 500 and hide the original 4xx information from the caller.
+    int firstFailureStatusCode = Response.Status.INTERNAL_SERVER_ERROR.getStatusCode();
     for (String target : targets) {
       try {
         // Remove task schedules before deletion so tasks are not triggered during the drop.
@@ -568,7 +571,7 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
         failed.add(target);
         if (firstFailure == null) {
           firstFailure = e;
-          firstFailureStatus = Response.Status.fromStatusCode(e.getResponse().getStatus());
+          firstFailureStatusCode = e.getResponse().getStatus();
         }
       } catch (Exception e) {
         // The wrapping CAE thrown after the loop will log the firstFailure with full stack;
@@ -578,7 +581,7 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
         failed.add(target);
         if (firstFailure == null) {
           firstFailure = e;
-          firstFailureStatus = Response.Status.INTERNAL_SERVER_ERROR;
+          firstFailureStatusCode = Response.Status.INTERNAL_SERVER_ERROR.getStatusCode();
         }
       }
     }
@@ -601,9 +604,7 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
     String msg = partialPrefix + "failed to drop " + failed + ": " + causeDesc
         + (dropped.isEmpty() ? "" : ". The successfully-dropped variant must be re-created if "
             + "the original DROP was unintended.");
-    throw new ControllerApplicationException(LOGGER, msg,
-        firstFailureStatus == null ? Response.Status.INTERNAL_SERVER_ERROR : firstFailureStatus,
-        firstFailure);
+    throw new ControllerApplicationException(LOGGER, msg, firstFailureStatusCode, firstFailure);
   }
 
   // -------------------------------------------------------------------------------------------
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
index 4df20930cf82..16e40fab34d1 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
@@ -234,11 +234,24 @@ private static String extractLiteralValue(@Nullable SqlNode literal) {
       throw new DdlCompilationException(
           "DEFAULT requires a literal value; got: " + literal.getClass().getSimpleName());
     }
+    SqlLiteral sqlLiteral = (SqlLiteral) literal;
+    String value;
     try {
-      return ((SqlLiteral) literal).toValue();
+      value = sqlLiteral.toValue();
     } catch (UnsupportedOperationException e) {
       throw new DdlCompilationException("Unsupported DEFAULT literal: " + literal, e);
     }
+    if (value == null) {
+      // SqlLiteral.toValue() returns null for SqlLiteral.createNull(...). Treat DEFAULT NULL as
+      // an explicit error rather than a silent no-op: in Pinot's model the column's
+      // defaultNullValue is the value used WHEN the source row is null, so DEFAULT NULL is
+      // semantically meaningless. Surface a clear error so the user fixes their DDL instead of
+      // wondering why their default doesn't apply.
+      throw new DdlCompilationException(
+          "DEFAULT NULL is not a valid Pinot default null value; omit the DEFAULT clause to "
+              + "use the type's natural default.");
+    }
+    return value;
   }
 
   private static ColumnRole inferRole(SqlPinotColumnDeclaration col, DataType dt) {
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
index 451e65a58945..e33dc10cf669 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
@@ -18,6 +18,7 @@
  */
 package org.apache.pinot.sql.ddl.reverse;
 
+import java.math.BigDecimal;
 import java.util.ArrayList;
 import java.util.List;
 import org.apache.pinot.spi.data.DateTimeFieldSpec;
@@ -106,11 +107,19 @@ private static String emitColumn(FieldSpec spec) {
     }
     // Only emit DEFAULT when the user-supplied value differs from the data-type's natural
     // default; this matches Pinot's own JSON serialization rule and keeps canonical output
-    // free of redundant defaults.
+    // free of redundant defaults. Use DataType.equals(v1, v2) for content comparison —
+    // Object.equals on byte[] is reference equality and would falsely flag every BYTES
+    // column at natural default as needing an explicit DEFAULT clause.
     Object defaultValue = spec.getDefaultNullValue();
     Object naturalDefault =
         FieldSpec.getDefaultNullValue(spec.getFieldType(), spec.getDataType(), null);
-    if (defaultValue != null && !defaultValue.equals(naturalDefault)) {
+    boolean atNaturalDefault;
+    if (defaultValue == null || naturalDefault == null) {
+      atNaturalDefault = (defaultValue == null && naturalDefault == null);
+    } else {
+      atNaturalDefault = spec.getDataType().equals(defaultValue, naturalDefault);
+    }
+    if (defaultValue != null && !atNaturalDefault) {
       sb.append(" DEFAULT ").append(emitDefault(defaultValue, spec.getDataType()));
     }
     if (spec instanceof DateTimeFieldSpec) {
@@ -174,9 +183,13 @@ private static String emitDefault(Object value, DataType dt) {
       case LONG:
       case FLOAT:
       case DOUBLE:
-      case BIG_DECIMAL:
       case BOOLEAN:
         return value.toString();
+      case BIG_DECIMAL:
+        // BigDecimal.toString() can emit scientific notation (e.g. "1E+10") for large or
+        // small magnitudes, which Calcite's Literal() rule does not accept. toPlainString()
+        // always produces the decimal form so the round-trip stays grammar-legal.
+        return value instanceof BigDecimal ? ((BigDecimal) value).toPlainString() : value.toString();
       default:
         return SqlIdentifiers.quoteString(value.toString());
     }

From c2d9fb65c168507c7700b4bfe158c892967234f7 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 01:00:56 -0700
Subject: [PATCH 17/32] Add regression tests for fifth-pass DDL fixes

Per C6.3 (bug fixes require regression tests), add three focused tests
covering the fixes in the prior commit:

- noDefaultEmittedForBytesAtNaturalDefault: a BYTES dimension at its
  natural default must not emit a DEFAULT clause. Without the
  DataType.equals fix, Object.equals on byte[] would fail content
  comparison and the canonical output would carry a redundant
  DEFAULT '' clause.
- bigDecimalDefaultEmitsPlainString: a BIG_DECIMAL default of 1E+30
  must round-trip without scientific notation. Calcite's Literal()
  rule does not accept scientific notation, so without
  toPlainString() the round-trip would fail to re-parse.
- defaultNullRejectedExplicitly: DEFAULT NULL must throw
  DdlCompilationException with a message naming DEFAULT NULL, instead
  of silently behaving as if no DEFAULT was supplied.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../sql/ddl/compile/DdlCompilerTest.java      | 14 +++++++
 .../ddl/reverse/CanonicalDdlEmitterTest.java  | 38 +++++++++++++++++++
 2 files changed, 52 insertions(+)

diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
index ab425fafbff5..18ee42770bda 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
@@ -198,6 +198,20 @@ public void stringDefaultDoesNotLeakSqlQuotes() {
     assertEquals(defaultValue, "unknown");
   }
 
+  /**
+   * DEFAULT NULL is semantically meaningless for Pinot's "default null value" concept (the
+   * value used when the source row is null). A user writing it would get silently no-op
+   * behavior under the previous implementation; we now reject explicitly so the user sees a
+   * clear error and corrects their DDL.
+   */
+  @Test
+  public void defaultNullRejectedExplicitly() {
+    DdlCompilationException ex = expectThrows(DdlCompilationException.class, () -> compileCreate(
+        "CREATE TABLE t (id INT DEFAULT NULL) TABLE_TYPE = OFFLINE"));
+    assertTrue(ex.getMessage() != null && ex.getMessage().contains("DEFAULT NULL"),
+        "expected error to name DEFAULT NULL, got: " + ex.getMessage());
+  }
+
   @Test
   public void numericDefaultRoundTripsCorrectly() {
     CompiledCreateTable c = compileCreate(
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
index e4c755e0d6b4..d0a26d1015d4 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
@@ -147,6 +147,44 @@ public void noDefaultsEmittedWhenAtNaturalDefault() {
     assertFalse(emitted.contains("'loadMode'"), emitted);
   }
 
+  /**
+   * Regression: comparing default value vs natural default with Object.equals does reference
+   * equality on byte[], so a BYTES column at its natural default would always emit a redundant
+   * DEFAULT '' clause. The fix uses DataType.equals which delegates to Arrays.equals.
+   */
+  @Test
+  public void noDefaultEmittedForBytesAtNaturalDefault() {
+    Schema schema = new Schema();
+    schema.setSchemaName("t");
+    schema.addField(new DimensionFieldSpec("blob", DataType.BYTES, true));
+
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build();
+    String emitted = CanonicalDdlEmitter.emit(schema, config);
+    assertTrue(emitted.contains("blob BYTES DIMENSION"), emitted);
+    assertFalse(emitted.contains("DEFAULT"),
+        "BYTES column at natural default must not emit a DEFAULT clause; got:\n" + emitted);
+  }
+
+  /**
+   * Regression: BigDecimal.toString() can emit scientific notation (1E+30) which Calcite's
+   * Literal() rule does not accept. The fix routes BIG_DECIMAL defaults through toPlainString().
+   */
+  @Test
+  public void bigDecimalDefaultEmitsPlainString() {
+    Schema schema = new Schema();
+    schema.setSchemaName("t");
+    MetricFieldSpec metric = new MetricFieldSpec("amount", DataType.BIG_DECIMAL,
+        new java.math.BigDecimal("1E+30"));
+    schema.addField(metric);
+
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build();
+    String emitted = CanonicalDdlEmitter.emit(schema, config);
+    assertFalse(emitted.contains("E+") || emitted.contains("E-"),
+        "BIG_DECIMAL default must not contain scientific notation; got:\n" + emitted);
+    assertTrue(emitted.contains("1000000000000000000000000000000"),
+        "expected plain-string form of 1E+30; got:\n" + emitted);
+  }
+
   @Test
   public void notNullAndDefaultEmittedExplicitly() {
     Schema schema = new Schema();

From c5e36a6d18a91bad0c59451d53b4a266d81bd047 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 01:23:51 -0700
Subject: [PATCH 18/32] Fix CRITICAL validation divergence + 3 MAJOR DDL issues
 from sixth review pass

CRITICAL:
- DDL CREATE was running a subset of the validation pipeline that
  POST /tables uses. Operators relying on min-replicas enforcement,
  storage-quota constraints, hybrid-pair compatibility, or instance-
  assignment validation could see DDL CREATE silently bypass them.
  Fix: delegate to TableConfigValidationUtils.validateTableConfig,
  the same canonical helper /tables uses. Inject ControllerConf and
  remove the bespoke validateTableConfig body.

MAJOR:
- Apply TableConfigTunerUtils.applyTunerConfigs before validation,
  mirroring POST /tables. Without this, a tuner-introduced config
  (e.g. a defaulted index config) could bypass validation.
- SchemaEmitter.emitColumns now skips the legacy TimeFieldSpec when
  a DateTimeFieldSpec with the same column name already exists.
  Schemas mid-migration carry both, and emitting both yielded a
  duplicate-column declaration that fails to re-parse.
- Reduced PRIMARY KEY LOOKAHEAD(3) to LOOKAHEAD(2). Two-token
  disambiguation ( ) is sufficient and produces a
  more accurate error message ("expected (") for malformed
  PRIMARY KEY id (no parens) cases. Updated the regression test to
  reflect the post-commit-error path.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../src/main/codegen/includes/parserImpls.ftl |  2 +-
 .../pinot/sql/parsers/PinotDdlParserTest.java | 15 ++++-----
 .../resources/PinotDdlRestletResource.java    | 31 +++++++++++--------
 .../pinot/sql/ddl/reverse/SchemaEmitter.java  | 12 +++++--
 4 files changed, 35 insertions(+), 25 deletions(-)

diff --git a/pinot-common/src/main/codegen/includes/parserImpls.ftl b/pinot-common/src/main/codegen/includes/parserImpls.ftl
index 1fcf0daa81c1..856a230bd9a1 100644
--- a/pinot-common/src/main/codegen/includes/parserImpls.ftl
+++ b/pinot-common/src/main/codegen/includes/parserImpls.ftl
@@ -135,7 +135,7 @@ SqlNode SqlPinotCreateTable() :
     [ LOOKAHEAD(3)    { ifNotExists = true; } ]
     name = CompoundIdentifier()
     columns = PinotColumnList()
-    [ LOOKAHEAD(3) primaryKeyColumns = PinotPrimaryKeyList() ]
+    [ LOOKAHEAD(2) primaryKeyColumns = PinotPrimaryKeyList() ]
      
     tableType = PinotTableTypeLiteral()
     [  properties = PinotPropertyList() ]
diff --git a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
index 6918a22da40b..88f1fa0cb650 100644
--- a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
@@ -318,17 +318,14 @@ public void createTableWithoutPrimaryKeyHasNullPkList() {
 
   @Test
   public void createTableMissingPrimaryKeyParensRejected() {
-    // Without LPAREN after KEY, the LOOKAHEAD(3) gate must reject the optional PK clause and
-    // the parser must surface a clear "expected TABLE_TYPE" error rather than committing to
-    // the PinotPrimaryKeyList production and emitting a confusing inner-grammar error. A
-    // regression that drops the LOOKAHEAD widening would commit to the PK production, fail at
-    // LPAREN, and surface "Encountered \"id\"" rather than the expected TABLE_TYPE token —
-    // verifying the message contains TABLE_TYPE locks in the desired behavior.
-    SqlCompilationException ex = expectThrows(SqlCompilationException.class,
+    // PRIMARY KEY id (without parens) must produce a parse error. With LOOKAHEAD(2) the parser
+    // commits to PinotPrimaryKeyList on seeing   and then fails at the missing
+    // LPAREN — which gives a more accurate error than LOOKAHEAD(3) would (the latter would
+    // backtrack and surface a misleading "expected TABLE_TYPE" message for what is clearly an
+    // attempted PRIMARY KEY clause).
+    expectThrows(SqlCompilationException.class,
         () -> CalciteSqlParser.compileToSqlNodeAndOptions(
             "CREATE TABLE t (id INT) PRIMARY KEY id TABLE_TYPE = OFFLINE"));
-    assertTrue(ex.getMessage() != null && ex.getMessage().contains("TABLE_TYPE"),
-        "expected error to point at the missing TABLE_TYPE token, got: " + ex.getMessage());
   }
 
   @Test
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index b3bc1bf6c872..1968d0c5db58 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -29,6 +29,7 @@
 import io.swagger.annotations.SecurityDefinition;
 import io.swagger.annotations.SwaggerDefinition;
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.HashSet;
 import java.util.List;
 import java.util.Objects;
@@ -50,6 +51,7 @@
 import org.apache.pinot.common.metadata.ZKMetadataProvider;
 import org.apache.pinot.common.utils.DatabaseUtils;
 import org.apache.pinot.common.utils.LogicalTableConfigUtils;
+import org.apache.pinot.controller.ControllerConf;
 import org.apache.pinot.controller.api.access.AccessControl;
 import org.apache.pinot.controller.api.access.AccessControlFactory;
 import org.apache.pinot.controller.api.access.AccessControlUtils;
@@ -61,13 +63,11 @@
 import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
 import org.apache.pinot.controller.helix.core.minion.PinotHelixTaskResourceManager;
 import org.apache.pinot.controller.helix.core.minion.PinotTaskManager;
-import org.apache.pinot.controller.util.TaskConfigUtils;
+import org.apache.pinot.controller.tuner.TableConfigTunerUtils;
 import org.apache.pinot.core.auth.Actions;
 import org.apache.pinot.core.auth.ManualAuthorization;
 import org.apache.pinot.core.auth.TargetType;
-import org.apache.pinot.segment.local.utils.TableConfigUtils;
 import org.apache.pinot.spi.config.table.TableConfig;
-import org.apache.pinot.spi.config.table.TableConfigValidatorRegistry;
 import org.apache.pinot.spi.config.table.TableType;
 import org.apache.pinot.spi.data.DateTimeFieldSpec;
 import org.apache.pinot.spi.data.FieldSpec;
@@ -137,6 +137,9 @@ public class PinotDdlRestletResource {
   @Inject
   AccessControlFactory _accessControlFactory;
 
+  @Inject
+  ControllerConf _controllerConf;
+
   @POST
   @Consumes(MediaType.APPLICATION_JSON)
   @Produces(MediaType.APPLICATION_JSON)
@@ -294,6 +297,11 @@ private Response executeCreate(CompiledCreateTable create, String database,
     // DDL's column-list-only projection — otherwise upsert/dedup tables would falsely fail PK
     // validation when the DDL omits PRIMARY KEY in the second variant.
     Schema schemaForValidation = schemaPreexisted ? storedSchema : create.getSchema();
+    // Apply tuner configs before validation, mirroring POST /tables. Tuners may rewrite the
+    // table config (e.g. fill in defaulted index configs) and the validators must run against
+    // the post-tuner shape, otherwise a tuner-introduced setting bypasses validation.
+    TableConfigTunerUtils.applyTunerConfigs(_pinotHelixResourceManager, create.getTableConfig(),
+        schemaForValidation, Collections.emptyMap());
     validateTableConfig(schemaForValidation, create.getTableConfig());
 
     if (dryRun) {
@@ -424,19 +432,16 @@ static String describeColumnShapeMismatch(Schema stored, Schema compiled) {
   }
 
   /**
-   * Runs the same schema/table validation stack that {@code /tables} and {@code /tableConfigs}
-   * apply before any ZK write, so DDL-created configs are subject to the same rules as
-   * JSON-API-created configs (upsert/dedup primary-key requirements, field config column
-   * references, task config validation, registry-level semantic checks, etc.).
+   * Runs the same schema/table validation stack that {@code POST /tables} and
+   * {@code /tableConfigs} apply before any ZK write, so DDL-created configs are subject to the
+   * same rules as JSON-API-created configs. Delegates to {@link TableConfigValidationUtils} so
+   * the two endpoints share a single validation pipeline (min replicas, storage quota, hybrid
+   * pair compatibility, instance assignment, tenant tags, task configs, registry-level checks).
    */
   private void validateTableConfig(Schema schema, TableConfig tableConfig) {
     try {
-      TableConfigUtils.validateTableName(tableConfig);
-      TableConfigUtils.validate(tableConfig, schema, null);
-      _pinotHelixResourceManager.validateTableTenantConfig(tableConfig);
-      _pinotHelixResourceManager.validateTableTaskMinionInstanceTagConfig(tableConfig);
-      TaskConfigUtils.validateTaskConfigs(tableConfig, schema, _pinotTaskManager, null);
-      TableConfigValidatorRegistry.validate(tableConfig, schema);
+      TableConfigValidationUtils.validateTableConfig(tableConfig, schema, null,
+          _pinotHelixResourceManager, _controllerConf, _pinotTaskManager);
     } catch (ControllerApplicationException e) {
       throw e;
     } catch (IllegalArgumentException | IllegalStateException e) {
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
index e33dc10cf669..837b1cc380d6 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
@@ -20,7 +20,9 @@
 
 import java.math.BigDecimal;
 import java.util.ArrayList;
+import java.util.HashSet;
 import java.util.List;
+import java.util.Set;
 import org.apache.pinot.spi.data.DateTimeFieldSpec;
 import org.apache.pinot.spi.data.DimensionFieldSpec;
 import org.apache.pinot.spi.data.FieldSpec;
@@ -60,12 +62,18 @@ static List emitColumns(Schema schema) {
       validateEmittable(metric);
       out.add(emitColumn(metric));
     }
+    Set dateTimeNames = new HashSet<>();
     for (DateTimeFieldSpec dt : schema.getDateTimeFieldSpecs()) {
+      dateTimeNames.add(dt.getName());
       out.add(emitColumn(dt));
     }
-    // Legacy time field: emit as a DATETIME column so the column is not silently dropped.
+    // Legacy time field: emit as a DATETIME column so the column is not silently dropped — but
+    // skip when a DateTimeFieldSpec already exists with the same column name. A schema mid-
+    // migration may carry both the legacy TimeFieldSpec and the modern DateTimeFieldSpec for
+    // the same logical column, and emitting both would yield a duplicate column declaration
+    // that fails to re-parse.
     TimeFieldSpec timeFieldSpec = schema.getTimeFieldSpec();
-    if (timeFieldSpec != null) {
+    if (timeFieldSpec != null && !dateTimeNames.contains(timeFieldSpec.getName())) {
       out.add(emitTimeColumn(timeFieldSpec));
     }
     return out;

From 65b856644475b88cdbfdc7cd5c35d9a547a46acb Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 01:34:20 -0700
Subject: [PATCH 19/32] Refresh response tableConfig snapshot after tuner
 application

The previous commit ran TableConfigTunerUtils.applyTunerConfigs AFTER
the DdlExecutionResponse was built, so the response advertised the
pre-tuner config while ZK persists (and dry-run would mis-predict)
the post-tuner shape. Refresh response.setTableConfig() right after
applyTunerConfigs runs so dry-run and live-create both surface the
shape that will actually be persisted.

Also strengthen the createTableMissingPrimaryKeyParensRejected test
to assert the error message names the missing LPAREN, locking in the
LOOKAHEAD(2) commit-then-fail-at-LPAREN behavior described in the
test comment.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../org/apache/pinot/sql/parsers/PinotDdlParserTest.java  | 8 ++++++--
 .../controller/api/resources/PinotDdlRestletResource.java | 4 ++++
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
index 88f1fa0cb650..18aa71ef7991 100644
--- a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
@@ -322,10 +322,14 @@ public void createTableMissingPrimaryKeyParensRejected() {
     // commits to PinotPrimaryKeyList on seeing   and then fails at the missing
     // LPAREN — which gives a more accurate error than LOOKAHEAD(3) would (the latter would
     // backtrack and surface a misleading "expected TABLE_TYPE" message for what is clearly an
-    // attempted PRIMARY KEY clause).
-    expectThrows(SqlCompilationException.class,
+    // attempted PRIMARY KEY clause). Pin the expected-LPAREN behaviour so a future grammar
+    // change cannot regress to a different (potentially less helpful) error path silently.
+    SqlCompilationException ex = expectThrows(SqlCompilationException.class,
         () -> CalciteSqlParser.compileToSqlNodeAndOptions(
             "CREATE TABLE t (id INT) PRIMARY KEY id TABLE_TYPE = OFFLINE"));
+    String message = ex.getMessage() == null ? "" : ex.getMessage();
+    assertTrue(message.contains("(") || message.toUpperCase(java.util.Locale.ROOT).contains("LPAREN"),
+        "expected error to indicate the missing LPAREN, got: " + message);
   }
 
   @Test
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 1968d0c5db58..2ec9975a7fd8 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -302,6 +302,10 @@ private Response executeCreate(CompiledCreateTable create, String database,
     // the post-tuner shape, otherwise a tuner-introduced setting bypasses validation.
     TableConfigTunerUtils.applyTunerConfigs(_pinotHelixResourceManager, create.getTableConfig(),
         schemaForValidation, Collections.emptyMap());
+    // Refresh the response's tableConfig snapshot now that tuners have run; otherwise the
+    // response advertises a pre-tuner config while ZK persists the post-tuner shape (and dry-run
+    // would mis-predict what a real CREATE would write).
+    response.setTableConfig(toJson(create.getTableConfig()));
     validateTableConfig(schemaForValidation, create.getTableConfig());
 
     if (dryRun) {

From d2f69dbe84706ba909c891d3073cd9fef302eca1 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 03:09:39 -0700
Subject: [PATCH 20/32] Fix CRITICAL ComplexFieldSpec drop + 4 MAJOR DDL issues
 from seventh review pass
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

CRITICAL:
- SchemaEmitter.emitColumns iterated only Dimension/Metric/DateTime/Time
  field specs, silently dropping ComplexFieldSpec columns from canonical
  DDL output. validateEmittable was never called for them. Replay of the
  emitted DDL would have produced a schema missing those columns —
  exactly the silent-drop scenario the validation was added to prevent.
  Fix: explicit fail-fast iteration over getComplexFieldSpecs() with a
  clear error pointing at the unsupported COMPLEX field type.

MAJOR:
- BIG_DECIMAL default-null-value comparison now uses compareTo == 0
  instead of equals. BigDecimal.equals is scale-sensitive
  (new BigDecimal("1").equals(new BigDecimal("1.0")) is false), which
  produced phantom mismatches when the same numeric default arrived via
  different literal forms. Applied in both
  describeColumnShapeMismatch (controller) and SchemaEmitter (natural
  default check).
- emitDefault now emits SQL TRUE/FALSE for BOOLEAN columns and quoted
  ISO timestamp strings for TIMESTAMP columns. Pinot stores these as
  Integer 0/1 and Long millis respectively; without this, canonical
  DDL exposed the internal encoding instead of the SQL literal form.
- DataTypeMapper now rejects SMALLINT/TINYINT explicitly with a clear
  error pointing at INT. Silently widening to INT today would lock
  every existing SMALLINT/TINYINT DDL into INT semantics permanently
  if Pinot ever adds INT8/INT16; rejected types can later become
  accepted, silently-promoted ones cannot be narrowed without breaking
  users.
- CompiledShowCreateTable.getTableType() Javadoc corrected: said
  "defaults to OFFLINE when both variants exist" but the actual
  behavior is to return 400 BAD_REQUEST on hybrid pairs and require
  the caller to specify TYPE OFFLINE | REALTIME. Future implementers
  reading the SPI doc would have produced the wrong policy.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../resources/PinotDdlRestletResource.java    | 36 ++++++++++++-----
 .../ddl/compile/CompiledShowCreateTable.java  |  6 ++-
 .../pinot/sql/ddl/compile/DataTypeMapper.java | 11 ++++-
 .../pinot/sql/ddl/reverse/SchemaEmitter.java  | 40 +++++++++++++++++--
 4 files changed, 76 insertions(+), 17 deletions(-)

diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 2ec9975a7fd8..3440ccc6f477 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -28,6 +28,7 @@
 import io.swagger.annotations.Authorization;
 import io.swagger.annotations.SecurityDefinition;
 import io.swagger.annotations.SwaggerDefinition;
+import java.math.BigDecimal;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.HashSet;
@@ -402,18 +403,16 @@ static String describeColumnShapeMismatch(Schema stored, Schema compiled) {
         return "column '" + columnName + "' NOT NULL flag differs (stored=" + storedSpec.isNotNull()
             + ", DDL=" + compiledSpec.isNotNull() + ")";
       }
-      // Compare typed default null value via DataType.equals, which delegates to Arrays.equals
-      // for BYTES (a fresh byte[] is allocated on every getDefaultNullValue() call, so a plain
-      // Objects.equals would do reference comparison and falsely flag every BYTES default as
-      // a mismatch). For other types, DataType.equals delegates to the boxed value's equals.
+      // Compare typed default null value. For BYTES, DataType.equals delegates to Arrays.equals
+      // (byte[] reference comparison would falsely flag every BYTES default as a mismatch).
+      // For BIG_DECIMAL, prefer compareTo over equals — BigDecimal.equals is scale-sensitive
+      // (new BigDecimal("1").equals(new BigDecimal("1.0")) is false), and the same numeric
+      // default arriving via different literal forms (DDL "1" vs JSON-API "1.0") would
+      // otherwise produce a phantom mismatch. For other types, DataType.equals delegates to
+      // the boxed value's equals.
       Object storedDefault = storedSpec.getDefaultNullValue();
       Object compiledDefault = compiledSpec.getDefaultNullValue();
-      boolean defaultsEqual;
-      if (storedDefault == null || compiledDefault == null) {
-        defaultsEqual = (storedDefault == null && compiledDefault == null);
-      } else {
-        defaultsEqual = storedSpec.getDataType().equals(storedDefault, compiledDefault);
-      }
+      boolean defaultsEqual = defaultValuesEqual(storedSpec.getDataType(), storedDefault, compiledDefault);
       if (!defaultsEqual) {
         return "column '" + columnName + "' default null value differs (stored="
             + storedSpec.getDefaultNullValueString()
@@ -435,6 +434,23 @@ static String describeColumnShapeMismatch(Schema stored, Schema compiled) {
     return null;
   }
 
+  /**
+   * Compares two default-null-values for content equality, accounting for type-specific
+   * gotchas: BYTES requires Arrays.equals (each getter allocates a fresh byte[]); BIG_DECIMAL
+   * requires compareTo so different scales of the same numeric value compare equal.
+   */
+  private static boolean defaultValuesEqual(FieldSpec.DataType dataType,
+      @Nullable Object storedDefault, @Nullable Object compiledDefault) {
+    if (storedDefault == null || compiledDefault == null) {
+      return storedDefault == null && compiledDefault == null;
+    }
+    if (dataType == FieldSpec.DataType.BIG_DECIMAL && storedDefault instanceof BigDecimal
+        && compiledDefault instanceof BigDecimal) {
+      return ((BigDecimal) storedDefault).compareTo((BigDecimal) compiledDefault) == 0;
+    }
+    return dataType.equals(storedDefault, compiledDefault);
+  }
+
   /**
    * Runs the same schema/table validation stack that {@code POST /tables} and
    * {@code /tableConfigs} apply before any ZK write, so DDL-created configs are subject to the
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java
index 4818cd09650f..98ce3372b054 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java
@@ -47,8 +47,10 @@ public String getRawTableName() {
   }
 
   /**
-   * @return the requested type, or {@code null} when the controller should auto-pick (defaults
-   *     to OFFLINE when both variants exist).
+   * @return the requested type, or {@code null} when the user omitted the {@code TYPE} clause.
+   *     When null, the controller picks the variant that exists; when both OFFLINE and REALTIME
+   *     variants exist, the controller returns 400 BAD_REQUEST and the caller must specify
+   *     {@code TYPE OFFLINE} or {@code TYPE REALTIME} explicitly.
    */
   @Nullable
   public TableType getTableType() {
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java
index f54046c5e6b5..b769786d6c81 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java
@@ -36,8 +36,6 @@ public final class DataTypeMapper {
     Map map = new HashMap<>();
     map.put("INT", DataType.INT);
     map.put("INTEGER", DataType.INT);
-    map.put("SMALLINT", DataType.INT);
-    map.put("TINYINT", DataType.INT);
     map.put("BIGINT", DataType.LONG);
     map.put("LONG", DataType.LONG);
     map.put("FLOAT", DataType.FLOAT);
@@ -69,6 +67,15 @@ private DataTypeMapper() {
   public static DataType resolve(String sqlTypeName) {
     // Locale.ROOT: in Turkish locale "int".toUpperCase() yields "İNT" which fails the lookup.
     String upper = sqlTypeName.toUpperCase(Locale.ROOT);
+    // Reject SMALLINT / TINYINT explicitly rather than silently widening to INT. If Pinot
+    // ever adds sub-INT integer types (INT8/INT16), existing DDLs using SMALLINT/TINYINT
+    // would otherwise be locked into the wider type permanently. A rejected type can later
+    // become accepted; a silently-promoted type cannot be narrowed without breaking users.
+    if ("SMALLINT".equals(upper) || "TINYINT".equals(upper)) {
+      throw new DdlCompilationException("Pinot does not yet support sub-INT integer types ("
+          + sqlTypeName + "); use INT instead. This restriction is intentional so that future "
+          + "narrow-integer types can be added without breaking existing DDL.");
+    }
     DataType dt = NAME_TO_DATATYPE.get(upper);
     if (dt == null) {
       throw new DdlCompilationException("Unsupported column data type: " + sqlTypeName);
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
index 837b1cc380d6..31df17d111df 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
@@ -19,6 +19,7 @@
 package org.apache.pinot.sql.ddl.reverse;
 
 import java.math.BigDecimal;
+import java.sql.Timestamp;
 import java.util.ArrayList;
 import java.util.HashSet;
 import java.util.List;
@@ -67,6 +68,15 @@ static List emitColumns(Schema schema) {
       dateTimeNames.add(dt.getName());
       out.add(emitColumn(dt));
     }
+    // Reject COMPLEX columns explicitly: the DDL grammar does not yet have a syntax for nested
+    // STRUCT/LIST/MAP fields. Without this guard, ComplexFieldSpec columns would silently fall
+    // through emitColumns and the canonical DDL would re-parse into a schema missing them.
+    for (FieldSpec complexSpec : schema.getComplexFieldSpecs()) {
+      throw new IllegalArgumentException(
+          "SHOW CREATE TABLE cannot represent complex column '" + complexSpec.getName()
+              + "' in DDL; replay of the emitted DDL would silently drop this column. "
+              + "The DDL grammar does not yet support COMPLEX field type.");
+    }
     // Legacy time field: emit as a DATETIME column so the column is not silently dropped — but
     // skip when a DateTimeFieldSpec already exists with the same column name. A schema mid-
     // migration may carry both the legacy TimeFieldSpec and the modern DateTimeFieldSpec for
@@ -115,15 +125,20 @@ private static String emitColumn(FieldSpec spec) {
     }
     // Only emit DEFAULT when the user-supplied value differs from the data-type's natural
     // default; this matches Pinot's own JSON serialization rule and keeps canonical output
-    // free of redundant defaults. Use DataType.equals(v1, v2) for content comparison —
-    // Object.equals on byte[] is reference equality and would falsely flag every BYTES
-    // column at natural default as needing an explicit DEFAULT clause.
+    // free of redundant defaults. For BYTES, use Arrays.equals via DataType.equals (byte[]
+    // reference equality would falsely flag every BYTES default as a mismatch). For
+    // BIG_DECIMAL, use compareTo so different scales of the same numeric value compare equal
+    // (BigDecimal.equals is scale-sensitive: new BigDecimal("0.0").equals(BigDecimal.ZERO)
+    // is false, which would make us emit a redundant DEFAULT 0.0).
     Object defaultValue = spec.getDefaultNullValue();
     Object naturalDefault =
         FieldSpec.getDefaultNullValue(spec.getFieldType(), spec.getDataType(), null);
     boolean atNaturalDefault;
     if (defaultValue == null || naturalDefault == null) {
       atNaturalDefault = (defaultValue == null && naturalDefault == null);
+    } else if (spec.getDataType() == DataType.BIG_DECIMAL && defaultValue instanceof BigDecimal
+        && naturalDefault instanceof BigDecimal) {
+      atNaturalDefault = ((BigDecimal) defaultValue).compareTo((BigDecimal) naturalDefault) == 0;
     } else {
       atNaturalDefault = spec.getDataType().equals(defaultValue, naturalDefault);
     }
@@ -191,8 +206,27 @@ private static String emitDefault(Object value, DataType dt) {
       case LONG:
       case FLOAT:
       case DOUBLE:
+        return value.toString();
       case BOOLEAN:
+        // Pinot stores BOOLEAN values internally as Integer 0/1; emit the SQL literal form
+        // (TRUE/FALSE) so canonical DDL is grammar-standard rather than exposing the internal
+        // encoding.
+        if (value instanceof Number) {
+          return ((Number) value).intValue() == 1 ? "TRUE" : "FALSE";
+        }
+        if (value instanceof Boolean) {
+          return ((Boolean) value) ? "TRUE" : "FALSE";
+        }
         return value.toString();
+      case TIMESTAMP:
+        // Pinot stores TIMESTAMP defaults as Long millis. Emit a quoted ISO timestamp string
+        // so canonical DDL is human-readable; FieldSpec.convertInternal accepts both ISO and
+        // numeric forms on re-parse, so the round-trip is preserved.
+        if (value instanceof Number) {
+          return SqlIdentifiers.quoteString(
+              new Timestamp(((Number) value).longValue()).toString());
+        }
+        return SqlIdentifiers.quoteString(value.toString());
       case BIG_DECIMAL:
         // BigDecimal.toString() can emit scientific notation (e.g. "1E+10") for large or
         // small magnitudes, which Calcite's Literal() rule does not accept. toPlainString()

From cb318fccbae7ca1bf24aeb72a0c29d7d3eefb861 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 03:19:57 -0700
Subject: [PATCH 21/32] Fix MAJOR TIMESTAMP timezone-dependence + add
 regression tests
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

MAJOR:
- TIMESTAMP DEFAULT emit now uses Instant.ofEpochMilli().toString()
  (UTC ISO-8601) instead of new Timestamp(...).toString() which
  formats in the JVM's default time zone. The previous form would
  emit different canonical DDL strings for the same input on
  controllers in different time zones, defeating the
  canonical-DDL-output contract.

Regression tests (per C6.3):
- bigDecimalAtNaturalDefaultDoesNotEmitDefault: locks in compareTo
  equality so BigDecimal("0.0") matches the natural BigDecimal.ZERO
  and no redundant DEFAULT clause is emitted.
- booleanDefaultEmittedAsSqlLiteral: locks in TRUE/FALSE emission
  rather than the internal Integer 0/1 form.
- timestampDefaultEmittedInUtcIso: locks in UTC ISO-8601 emission
  with Instant.toString output (validated against epoch
  1700000000000 → "2023-11-14T22:13:20Z").
- smallintTinyintRejectedExplicitly (DdlCompilerTest): locks in the
  explicit rejection of SMALLINT/TINYINT with messages naming the
  type, preventing future silent INT promotion.
- acceptsScaleShiftedBigDecimalDefault (PinotDdlRestletResourceUnitTest):
  locks in compareTo equality in the controller-side comparator so
  hybrid CREATE accepts BigDecimal("1") vs BigDecimal("1.0") as
  equivalent.

ComplexFieldSpec rejection cannot be unit-tested directly (the
constructor itself rejects DataType.STRUCT via the FieldSpec base
class), so the production guard is purely defensive against future
schema-construction paths that bypass the base-class check.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../PinotDdlRestletResourceUnitTest.java      | 24 ++++++++
 .../pinot/sql/ddl/reverse/SchemaEmitter.java  | 11 ++--
 .../sql/ddl/compile/DdlCompilerTest.java      | 17 ++++++
 .../ddl/reverse/CanonicalDdlEmitterTest.java  | 61 +++++++++++++++++++
 4 files changed, 108 insertions(+), 5 deletions(-)

diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
index c94d2d8484c0..082cd09a9a6f 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
@@ -166,6 +166,30 @@ public void acceptsMatchingBytesDefaultNullValue() {
         "matching BYTES defaults must not trip a content-vs-reference equality regression");
   }
 
+  /**
+   * Regression: BigDecimal.equals is scale-sensitive (1 != 1.0); the comparator must use
+   * compareTo so the same numeric default arriving via different literal forms is treated
+   * as equivalent. Without this fix, a hybrid second-variant CREATE on a BIG_DECIMAL column
+   * whose stored default arrived as "1.0" but DDL re-states "1" would falsely 409 CONFLICT.
+   */
+  @Test
+  public void acceptsScaleShiftedBigDecimalDefault() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    MetricFieldSpec storedSpec = new MetricFieldSpec("amount", DataType.BIG_DECIMAL,
+        new java.math.BigDecimal("1.0"));
+    stored.addField(storedSpec);
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    MetricFieldSpec compiledSpec = new MetricFieldSpec("amount", DataType.BIG_DECIMAL,
+        new java.math.BigDecimal("1"));
+    compiled.addField(compiledSpec);
+
+    assertNull(PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled),
+        "scale-shifted BIG_DECIMAL defaults must compare equal via compareTo");
+  }
+
   /** BYTES default mismatch must still be rejected with content-aware comparison. */
   @Test
   public void rejectsBytesDefaultNullValueMismatch() {
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
index 31df17d111df..108471e316ea 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
@@ -19,7 +19,7 @@
 package org.apache.pinot.sql.ddl.reverse;
 
 import java.math.BigDecimal;
-import java.sql.Timestamp;
+import java.time.Instant;
 import java.util.ArrayList;
 import java.util.HashSet;
 import java.util.List;
@@ -219,12 +219,13 @@ private static String emitDefault(Object value, DataType dt) {
         }
         return value.toString();
       case TIMESTAMP:
-        // Pinot stores TIMESTAMP defaults as Long millis. Emit a quoted ISO timestamp string
-        // so canonical DDL is human-readable; FieldSpec.convertInternal accepts both ISO and
-        // numeric forms on re-parse, so the round-trip is preserved.
+        // Pinot stores TIMESTAMP defaults as Long millis. Emit a quoted UTC ISO-8601 string
+        // (Instant.toString) — NOT java.sql.Timestamp.toString, which formats in the JVM's
+        // default time zone and would make canonical DDL emit different strings on different
+        // controllers for the same input. ISO-8601 round-trips through TimestampUtils.
         if (value instanceof Number) {
           return SqlIdentifiers.quoteString(
-              new Timestamp(((Number) value).longValue()).toString());
+              Instant.ofEpochMilli(((Number) value).longValue()).toString());
         }
         return SqlIdentifiers.quoteString(value.toString());
       case BIG_DECIMAL:
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
index 18ee42770bda..7980d6e47084 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
@@ -198,6 +198,23 @@ public void stringDefaultDoesNotLeakSqlQuotes() {
     assertEquals(defaultValue, "unknown");
   }
 
+  /**
+   * SMALLINT and TINYINT are explicitly rejected to keep the type contract narrow: silently
+   * widening to INT today would lock those DDLs into INT semantics if Pinot later adds
+   * INT8/INT16. Rejection at the boundary is reversible; silent promotion is not.
+   */
+  @Test
+  public void smallintTinyintRejectedExplicitly() {
+    DdlCompilationException ex1 = expectThrows(DdlCompilationException.class, () -> compileCreate(
+        "CREATE TABLE t (id SMALLINT) TABLE_TYPE = OFFLINE"));
+    assertTrue(ex1.getMessage() != null && ex1.getMessage().contains("SMALLINT"),
+        "expected error to name SMALLINT, got: " + ex1.getMessage());
+    DdlCompilationException ex2 = expectThrows(DdlCompilationException.class, () -> compileCreate(
+        "CREATE TABLE t (id TINYINT) TABLE_TYPE = OFFLINE"));
+    assertTrue(ex2.getMessage() != null && ex2.getMessage().contains("TINYINT"),
+        "expected error to name TINYINT, got: " + ex2.getMessage());
+  }
+
   /**
    * DEFAULT NULL is semantically meaningless for Pinot's "default null value" concept (the
    * value used when the source row is null). A user writing it would get silently no-op
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
index d0a26d1015d4..92d4c5c2a5d1 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
@@ -185,6 +185,67 @@ public void bigDecimalDefaultEmitsPlainString() {
         "expected plain-string form of 1E+30; got:\n" + emitted);
   }
 
+  /**
+   * Regression: BIG_DECIMAL natural-default check must use compareTo rather than equals,
+   * so a stored BigDecimal("0.0") (scale 1) is treated as equivalent to BigDecimal.ZERO and
+   * canonical DDL elides the redundant DEFAULT clause.
+   */
+  @Test
+  public void bigDecimalAtNaturalDefaultDoesNotEmitDefault() {
+    Schema schema = new Schema();
+    schema.setSchemaName("t");
+    MetricFieldSpec metric = new MetricFieldSpec("amount", DataType.BIG_DECIMAL,
+        new java.math.BigDecimal("0.0"));
+    schema.addField(metric);
+
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build();
+    String emitted = CanonicalDdlEmitter.emit(schema, config);
+    assertFalse(emitted.contains("DEFAULT"),
+        "BIG_DECIMAL at scale-shifted natural default must not emit DEFAULT; got:\n" + emitted);
+  }
+
+  /**
+   * Regression: BOOLEAN columns store defaults internally as Integer 0/1; canonical DDL
+   * must emit the SQL literal form (TRUE/FALSE) so the output is grammar-standard.
+   */
+  @Test
+  public void booleanDefaultEmittedAsSqlLiteral() {
+    Schema schema = new Schema();
+    schema.setSchemaName("t");
+    DimensionFieldSpec dim = new DimensionFieldSpec("flag", DataType.BOOLEAN, true);
+    dim.setDefaultNullValue(1);
+    schema.addField(dim);
+
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build();
+    String emitted = CanonicalDdlEmitter.emit(schema, config);
+    assertTrue(emitted.contains("DEFAULT TRUE"),
+        "BOOLEAN default 1 must emit DEFAULT TRUE, got:\n" + emitted);
+    assertFalse(emitted.contains("DEFAULT 1") && !emitted.contains("DEFAULT TRUE"),
+        "must not emit raw integer encoding; got:\n" + emitted);
+  }
+
+  /**
+   * Regression: TIMESTAMP DEFAULT emission must use UTC ISO-8601 form (Instant.toString)
+   * rather than java.sql.Timestamp.toString, which formats in the JVM's default time zone
+   * and would make canonical DDL emit different strings on different controllers.
+   */
+  @Test
+  public void timestampDefaultEmittedInUtcIso() {
+    Schema schema = new Schema();
+    schema.setSchemaName("t");
+    DimensionFieldSpec dim = new DimensionFieldSpec("ts", DataType.TIMESTAMP, true);
+    // 1700000000000 millis = 2023-11-14T22:13:20Z — pick a non-zero non-natural-default value
+    // so the DEFAULT clause is actually emitted.
+    dim.setDefaultNullValue(1700000000000L);
+    schema.addField(dim);
+
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build();
+    String emitted = CanonicalDdlEmitter.emit(schema, config);
+    // Instant.ofEpochMilli(1700000000000L).toString() is "2023-11-14T22:13:20Z" — UTC ISO-8601.
+    assertTrue(emitted.contains("'2023-11-14T22:13:20Z'"),
+        "TIMESTAMP default must emit UTC ISO-8601 form (Instant.toString); got:\n" + emitted);
+  }
+
   @Test
   public void notNullAndDefaultEmittedExplicitly() {
     Schema schema = new Schema();

From e3b8d0962869067ee63d563c84e13b4c97163922 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 03:35:21 -0700
Subject: [PATCH 22/32] Address 4 MAJOR documentation/diagnostics issues from
 eighth review pass

- PropertyMapping class-level Javadoc said the stream. prefix is
  stripped; the implementation actually preserves it (and routes
  realtime.* keys the same way). Updated the contract description to
  match implementation: keys are stored verbatim with their
  stream./realtime. prefix intact.
- DROP TABLE: documented the no-TYPE form's "authorize both variants
  up-front" contract inline at the candidates loop, mirroring the
  SHOW CREATE policy and explaining the partial-permission caller's
  workaround (use explicit TYPE clause).
- DROP TABLE partial-failure: now tracks targets whose
  tableTasksCleanup ran but whose deleteTable subsequently failed,
  and surfaces them in the error message with a recovery hint
  pointing at the TableConfig taskTypeConfigsMap.SCHEDULE_KEY entries
  the operator must restore.
- MAX_DDL_SQL_LENGTH renamed to MAX_DDL_SQL_CHARS; Javadoc clarifies
  the limit is in Java characters (UTF-16 code units), not bytes,
  with a note for operators sizing reverse-proxy body limits.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../resources/PinotDdlRestletResource.java    | 42 ++++++++++++++++---
 .../api/PinotDdlRestletResourceTest.java      |  2 +-
 .../sql/ddl/compile/PropertyMapping.java      |  7 ++--
 3 files changed, 41 insertions(+), 10 deletions(-)

diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 3440ccc6f477..99ed23213082 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -123,8 +123,13 @@
 @Path("/")
 public class PinotDdlRestletResource {
   private static final Logger LOGGER = LoggerFactory.getLogger(PinotDdlRestletResource.class);
-  /** Maximum accepted SQL input length (256 KB) to prevent unbounded parser memory allocation. */
-  private static final int MAX_DDL_SQL_LENGTH = 256 * 1024;
+  /**
+   * Maximum accepted SQL input length, measured in {@link String#length() Java characters}
+   * (UTF-16 code units), to prevent unbounded parser memory allocation. Up to ~4× this value
+   * in UTF-8 wire bytes can be accepted by Jackson before the length check rejects; operators
+   * sizing reverse-proxy body limits should plan accordingly.
+   */
+  private static final int MAX_DDL_SQL_CHARS = 256 * 1024;
 
   @Inject
   PinotHelixResourceManager _pinotHelixResourceManager;
@@ -168,8 +173,8 @@ public Response executeDdl(
     }
     // Guard against arbitrarily large inputs that would force the Calcite parser to allocate
     // excessive memory building the AST in-memory.
-    if (request.getSql().length() > MAX_DDL_SQL_LENGTH) {
-      throw badRequest("DDL statement exceeds maximum length of " + MAX_DDL_SQL_LENGTH + " characters.");
+    if (request.getSql().length() > MAX_DDL_SQL_CHARS) {
+      throw badRequest("DDL statement exceeds maximum length of " + MAX_DDL_SQL_CHARS + " characters.");
     }
 
     CompiledDdl compiled;
@@ -497,6 +502,13 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
     // Compute the candidate typed names BEFORE existence filtering so we can authorize against
     // the user's intent (not just whatever happens to exist now). This prevents an unauthorized
     // caller from probing existence via 200/404 vs 403.
+    //
+    // NB: the no-TYPE form (`DROP TABLE foo` without `TYPE OFFLINE | REALTIME`) authorizes
+    // BOTH variants up-front under access-control plugins that grant per-type permissions, so
+    // a caller with permission only on OFFLINE will receive 403 even if the REALTIME variant
+    // does not exist. This mirrors the SHOW CREATE TABLE no-TYPE policy and is intentional:
+    // "drop both variants atomically" requires authorization on both. Callers with partial
+    // permission must use the explicit `TYPE` clause to target a single variant.
     List candidates = new ArrayList<>(2);
     if (drop.getTableType() == null || drop.getTableType() == TableType.OFFLINE) {
       candidates.add(TableNameBuilder.OFFLINE.tableNameWithType(fullyQualifiedRaw));
@@ -572,7 +584,13 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
     // null for non-standard codes (422, 423, 451, etc.) — a null would silently fall back
     // to 500 and hide the original 4xx information from the caller.
     int firstFailureStatusCode = Response.Status.INTERNAL_SERVER_ERROR.getStatusCode();
+    // Targets whose tableTasksCleanup succeeded (removing scheduled task triggers) but whose
+    // deleteTable subsequently failed. These tables remain in the cluster but will no longer
+    // run their scheduled tasks until the operator either retries the DROP successfully or
+    // restores the SCHEDULE_KEY entries on the surviving table config.
+    List taskSchedulesCleared = new ArrayList<>();
     for (String target : targets) {
+      boolean tasksCleaned = false;
       try {
         // Remove task schedules before deletion so tasks are not triggered during the drop.
         // tableTasksCleanup may throw ControllerApplicationException (e.g. BAD_REQUEST when
@@ -580,6 +598,7 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
         // information and must be preserved instead of being collapsed to 500 below.
         PinotTableRestletResource.tableTasksCleanup(target, false,
             _pinotHelixResourceManager, _pinotHelixTaskResourceManager);
+        tasksCleaned = true;
         // deleteTable(rawName, type, retention) takes the raw name and re-derives the typed
         // name internally via TableNameBuilder.forType(type).tableNameWithType(rawName); see
         // PinotHelixResourceManager.deleteTable. Pass `fullyQualifiedRaw` (DB-qualified raw
@@ -594,6 +613,9 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
         // redundant noise — record a one-line breadcrumb at WARN without the throwable.
         LOGGER.warn("DROP TABLE on {} failed: {}", target, e.getMessage());
         failed.add(target);
+        if (tasksCleaned) {
+          taskSchedulesCleared.add(target);
+        }
         if (firstFailure == null) {
           firstFailure = e;
           firstFailureStatusCode = e.getResponse().getStatus();
@@ -604,6 +626,9 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
         // without seeing the same stack trace twice.
         LOGGER.warn("DROP TABLE on {} failed unexpectedly: {}", target, e.toString());
         failed.add(target);
+        if (tasksCleaned) {
+          taskSchedulesCleared.add(target);
+        }
         if (firstFailure == null) {
           firstFailure = e;
           firstFailureStatusCode = Response.Status.INTERNAL_SERVER_ERROR.getStatusCode();
@@ -626,9 +651,14 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
     String causeDesc = firstFailure.getMessage() != null
         ? firstFailure.getMessage() : firstFailure.getClass().getSimpleName();
     String partialPrefix = dropped.isEmpty() ? "" : "Partial DROP TABLE: dropped " + dropped + ", ";
+    String taskScheduleHint = taskSchedulesCleared.isEmpty() ? ""
+        : ". Task schedules were already cleared for " + taskSchedulesCleared
+            + "; if those tables remain in service the operator must restore the schedule entries"
+            + " in their TableConfig taskTypeConfigsMap before scheduled tasks resume.";
+    String reCreateHint = dropped.isEmpty() ? ""
+        : ". The successfully-dropped variant must be re-created if the original DROP was unintended.";
     String msg = partialPrefix + "failed to drop " + failed + ": " + causeDesc
-        + (dropped.isEmpty() ? "" : ". The successfully-dropped variant must be re-created if "
-            + "the original DROP was unintended.");
+        + reCreateHint + taskScheduleHint;
     throw new ControllerApplicationException(LOGGER, msg, firstFailureStatusCode, firstFailure);
   }
 
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
index 56f17cce407f..8a6fd4e2f5b0 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
@@ -311,7 +311,7 @@ public void oversizedInputReturnsBadRequest()
     // Any SQL that exceeds 256 KB must be rejected before parsing to prevent unbounded allocations.
     String oversized = "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE /* " + StringUtils.repeat("x", 256 * 1024) + " */";
     int status = postDdlExpectFailure(oversized);
-    assertEquals(status, 400, "Expected 400 for input exceeding MAX_DDL_SQL_LENGTH");
+    assertEquals(status, 400, "Expected 400 for input exceeding MAX_DDL_SQL_CHARS");
   }
 
   // -------------------------------------------------------------------------------------------
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
index e6e65619a618..6fd6eed4be71 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
@@ -66,9 +66,10 @@
  * 
    
  *   
  1. If the key (case-insensitive) is in the promoted catalog, set the corresponding
  *       {@link TableConfigBuilder} field directly.
    2. 
- *   
  2. If the key starts with {@code stream.}, route into {@code IndexingConfig.streamConfigs}.
- *       The {@code stream.} prefix is stripped so existing Pinot stream property keys (e.g.
- *       {@code stream.kafka.topic.name}) round-trip cleanly. REALTIME-only.
    3. 
+ *   
  3. If the key starts with {@code stream.} or {@code realtime.}, route the entry into
+ *       {@code IndexingConfig.streamConfigs} verbatim (the prefix is preserved, not stripped, so
+ *       the key matches existing Pinot stream config conventions like
+ *       {@code stream.kafka.topic.name}). REALTIME-only.
    4. 
  *   
  4. If the key starts with {@code task..}, route the remainder into
  *       {@code TableTaskConfig.taskTypeConfigsMap[taskType]}.
    5. 
  *   
  5. Otherwise, store verbatim in {@link TableCustomConfig#getCustomConfigs()}.

From a49c3a8758e8ad1e5134405dd9fe9d9335da92e5 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 04:19:33 -0700
Subject: [PATCH 23/32] Address 5 MAJOR DDL issues from ninth review pass
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- emitTableType replaced ternary (which silently returned REALTIME for
  any non-OFFLINE TableType) with an exhaustive switch that throws
  IllegalArgumentException on unknown values. The throw maps to the
  500 path the controller resource already handles for emit() runtime
  exceptions.
- SHOW CREATE TABLE: a missing schema (after hasTable + tableConfig
  pass) now returns 404 instead of 500. Schemas can be deleted
  independently via DELETE /schemas/{name}, so a missing schema is
  reachable in normal operation and is caller-actionable. The error
  message tells the operator how to recreate it.
- DEFAULT type compatibility check: extractLiteralValue now coerces
  the default literal through DataType.convert() at compile time
  rather than letting an INT column with DEFAULT 'abc' compile cleanly
  and fail at first ingestion. Added regression test.
- Added a regression test asserting case-insensitive TABLE_TYPE input
  is accepted and canonicalized to uppercase, locking in the
  parseTableType equalsIgnoreCase contract against future grammar
  tightening.
- Added package-info.java for org.apache.pinot.sql.ddl documenting:
  module layering vs pinot-common, sub-package responsibilities,
  thread safety, exception → HTTP-status contract, and the evolution
  policy for adding new TableConfig properties / column attributes.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../resources/PinotDdlRestletResource.java    | 12 ++-
 .../pinot/sql/ddl/compile/DdlCompiler.java    | 11 +++
 .../apache/pinot/sql/ddl/package-info.java    | 75 +++++++++++++++++++
 .../sql/ddl/reverse/CanonicalDdlEmitter.java  | 12 ++-
 .../sql/ddl/compile/DdlCompilerTest.java      | 32 ++++++++
 5 files changed, 138 insertions(+), 4 deletions(-)
 create mode 100644 pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/package-info.java

diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 99ed23213082..11d139595884 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -742,10 +742,16 @@ private DdlExecutionResponse executeShowCreate(CompiledShowCreateTable show, Str
     }
     Schema schema = _pinotHelixResourceManager.getTableSchema(tableNameWithType);
     if (schema == null) {
+      // Unlike a missing TableConfig (which would indicate a torn-write inconsistency since
+      // hasTable just returned true), a missing schema is reachable in normal operation:
+      // schemas can be deleted independently via DELETE /schemas/{name} while a table
+      // still references them. Surface as 404 rather than 500 so the operator sees an
+      // actionable user-error code rather than a phantom controller-bug code.
       throw new ControllerApplicationException(LOGGER,
-          "Schema not found for " + tableNameWithType
-              + "; SHOW CREATE TABLE requires both schema and config to exist.",
-          Response.Status.INTERNAL_SERVER_ERROR);
+          "Schema '" + tableNameWithType
+              + "' not found; SHOW CREATE TABLE requires the schema to exist. "
+              + "Re-create it via POST /schemas if it was deleted.",
+          Response.Status.NOT_FOUND);
     }
 
     // Use the resolved database (which incorporates the Database header) so the emitted DDL
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
index 16e40fab34d1..41af7509620d 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
@@ -337,6 +337,17 @@ private static FieldSpec toFieldSpec(ResolvedColumnDefinition col) {
       spec.setNotNull(true);
     }
     if (col.getDefaultValue() != null) {
+      // Validate type compatibility at DDL compile time. FieldSpec.setDefaultNullValue stores
+      // the string lazily; without this check, "INT col DEFAULT 'abc'" would compile cleanly
+      // and then fail at first ingestion with a less-specific error from the segment generator.
+      // Failing here gives the user a 400 with the column name and the offending literal.
+      try {
+        col.getDataType().convert(col.getDefaultValue());
+      } catch (RuntimeException e) {
+        throw new DdlCompilationException("DEFAULT value '" + col.getDefaultValue()
+            + "' is not compatible with column '" + col.getName() + "' of type "
+            + col.getDataType() + ": " + e.getMessage(), e);
+      }
       spec.setDefaultNullValue(col.getDefaultValue());
     }
     return spec;
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/package-info.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/package-info.java
new file mode 100644
index 000000000000..14c2a46803f8
--- /dev/null
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/package-info.java
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/**
+ * Apache Pinot SQL DDL feature: compile and reverse-emit DDL statements.
+ *
+ * 
    Module layering
    
+ * The grammar (FreeMarker {@code parserImpls.ftl}) and the AST nodes ({@code SqlPinotCreateTable},
+ * {@code SqlPinotDropTable}, etc.) live in {@code pinot-common} so the controller can invoke the
+ * Calcite parser without pulling this module. This module ({@code pinot-sql-ddl}) contains the
+ * Pinot-specific compile/resolve/reverse-emit logic that turns parser AST nodes into
+ * {@link org.apache.pinot.spi.config.table.TableConfig} / {@link org.apache.pinot.spi.data.Schema}
+ * pairs and back. Dependencies: {@code pinot-spi} + {@code pinot-common} + {@code calcite-core}.
+ *
+ * 
    Sub-packages
    
+ * 
      
+ *   
    • {@link org.apache.pinot.sql.ddl.compile} — forward path: parser AST → compiled DDL
+ *       artifact ({@code CompiledCreateTable}, {@code CompiledDropTable}, etc.). The entry point
+ *       is {@link org.apache.pinot.sql.ddl.compile.DdlCompiler#compile(String)}.
      • 
+ *   
    • {@link org.apache.pinot.sql.ddl.resolved} — typed intermediate representation of resolved
+ *       column declarations and table metadata, consumed only by the compiler.
      • 
+ *   
    • {@link org.apache.pinot.sql.ddl.reverse} — reverse path: stored {@code Schema} +
+ *       {@code TableConfig} → canonical DDL string. Used by {@code SHOW CREATE TABLE}.
      • 
+ * 
    
+ *
+ * 
    Thread safety
    
+ * All compiler / emitter classes are stateless and safe for concurrent use. The compiled artifacts
+ * ({@code CompiledCreateTable} etc.) are immutable views over freshly-constructed {@code Schema}
+ * and {@code TableConfig} objects; callers are responsible for not mutating them after compilation.
+ *
+ * 
    Exception → HTTP-status contract
    
+ * The DDL compiler signals errors via two exception types, which the REST resource translates as:
+ * 
      
+ *   
    • {@link org.apache.pinot.sql.ddl.compile.DdlCompilationException}: caller-actionable
+ *       compile-time errors (unsupported types, malformed property values, type-incompatible
+ *       defaults, reserved keys). Surfaced as HTTP 400.
      • 
+ *   
    • {@link java.lang.IllegalArgumentException} from {@code CanonicalDdlEmitter.emit(...)}:
+ *       canonical DDL grammar cannot represent the schema/config (e.g. unsupported column types
+ *       like MAP/LIST/STRUCT, or TableCustomConfig keys that collide with reserved DDL property
+ *       names). Surfaced as HTTP 400.
      • 
+ *   
    • Any other {@link java.lang.RuntimeException} from emit/compile is treated as a controller
+ *       defect and surfaced as HTTP 500.
      • 
+ * 
    
+ *
+ * 
    Evolution policy
    
+ * Adding a new TableConfig property:
+ * 
      
+ *   
    1. If it has a builder setter and round-trips as a single string, add it to
+ *       {@link org.apache.pinot.sql.ddl.compile.PropertyMapping#applyPromoted} (forward path) and
+ *       {@link org.apache.pinot.sql.ddl.reverse.PropertyExtractor} (reverse path).
      2. 
+ *   
    2. Add the lowercase key to {@code RESERVED_ROUND_TRIP_KEYS} so user-supplied
+ *       {@code TableCustomConfig} entries cannot shadow the promoted name.
      3. 
+ *   
    3. Cover the round-trip in {@code RoundTripTest}.
      4. 
+ * 
    
+ * Adding a new column attribute (e.g. a new role beyond DIMENSION/METRIC/DATETIME) requires
+ * coordinated changes to {@code parserImpls.ftl}, {@code SqlPinotColumnDeclaration},
+ * {@code DdlCompiler.toFieldSpec}, and {@code SchemaEmitter.emitColumn} — keep them in sync.
+ */
+package org.apache.pinot.sql.ddl;
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java
index d644b8fd5c6b..95acaaa54b2c 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java
@@ -138,6 +138,16 @@ public static String emit(Schema schema, TableConfig config, @Nullable String da
   }
 
   private static String emitTableType(TableType tableType) {
-    return tableType == TableType.OFFLINE ? "OFFLINE" : "REALTIME";
+    // Exhaustive switch so a future TableType (e.g. UNIFIED) is rejected at the emit boundary
+    // rather than silently rendered as REALTIME. The throw maps to the 500 path the controller
+    // resource already handles for RuntimeException from emit().
+    switch (tableType) {
+      case OFFLINE:
+        return "OFFLINE";
+      case REALTIME:
+        return "REALTIME";
+      default:
+        throw new IllegalArgumentException("Unsupported TableType for DDL emission: " + tableType);
+    }
   }
 }
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
index 7980d6e47084..5c10e84eb867 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
@@ -198,6 +198,38 @@ public void stringDefaultDoesNotLeakSqlQuotes() {
     assertEquals(defaultValue, "unknown");
   }
 
+  /**
+   * TABLE_TYPE parsing is case-insensitive (parseTableType uses equalsIgnoreCase), but the
+   * compiled TableConfig always stores the canonical uppercase form. Lock in the
+   * lowercase-input → uppercase-output behavior so a future grammar tightening cannot silently
+   * regress to case-sensitive matching.
+   */
+  @Test
+  public void lowercaseTableTypeAcceptedAndCanonicalized() {
+    CompiledCreateTable lowerCase = compileCreate(
+        "CREATE TABLE t (id INT) TABLE_TYPE = offline");
+    assertEquals(lowerCase.getTableConfig().getTableType(), TableType.OFFLINE);
+
+    CompiledCreateTable mixedCase = compileCreate(
+        "CREATE TABLE t (id INT) TABLE_TYPE = ReAlTiMe");
+    assertEquals(mixedCase.getTableConfig().getTableType(), TableType.REALTIME);
+  }
+
+  /**
+   * DEFAULT literals must be compatible with the column's declared data type. Non-numeric
+   * defaults on numeric columns must be rejected at compile time with a clear error rather
+   * than failing at first ingestion with a downstream-layer error.
+   */
+  @Test
+  public void defaultLiteralWrongTypeRejected() {
+    DdlCompilationException ex = expectThrows(DdlCompilationException.class, () -> compileCreate(
+        "CREATE TABLE t (id INT DEFAULT 'abc') TABLE_TYPE = OFFLINE"));
+    assertTrue(ex.getMessage() != null && ex.getMessage().contains("'abc'"),
+        "expected error to name the offending literal, got: " + ex.getMessage());
+    assertTrue(ex.getMessage().contains("id"),
+        "expected error to name the column, got: " + ex.getMessage());
+  }
+
   /**
    * SMALLINT and TINYINT are explicitly rejected to keep the type contract narrow: silently
    * widening to INT today would lock those DDLs into INT semantics if Pinot later adds

From ca69abee948b625166e6f29b331c0ff9a502ffc9 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 04:43:31 -0700
Subject: [PATCH 24/32] Fix tautological assertion in
 booleanDefaultEmittedAsSqlLiteral test
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The assertion `assertFalse(emitted.contains("DEFAULT 1") && !emitted.contains("DEFAULT TRUE"))`
was vacuous because the line above already asserted `emitted.contains("DEFAULT TRUE")` is
true, forcing the negated conjunct to false. The intent was to forbid a raw-integer
fallback, so simplify to `assertFalse(emitted.contains("DEFAULT 1"))` — which is the
actual property the test should lock in (per C6.5: tests must validate claimed behavior).

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
index 92d4c5c2a5d1..7cd4f21798fe 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
@@ -220,8 +220,8 @@ public void booleanDefaultEmittedAsSqlLiteral() {
     String emitted = CanonicalDdlEmitter.emit(schema, config);
     assertTrue(emitted.contains("DEFAULT TRUE"),
         "BOOLEAN default 1 must emit DEFAULT TRUE, got:\n" + emitted);
-    assertFalse(emitted.contains("DEFAULT 1") && !emitted.contains("DEFAULT TRUE"),
-        "must not emit raw integer encoding; got:\n" + emitted);
+    assertFalse(emitted.contains("DEFAULT 1"),
+        "must not emit raw integer encoding (DEFAULT 1); got:\n" + emitted);
   }
 
   /**

From 9e7da7ce18e99ca009d8bc944f717bb00944faf6 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 07:17:53 -0700
Subject: [PATCH 25/32] Fix CRITICAL BYTES default-value emit fallthrough
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

emitDefault's switch had no BYTES case, so byte[] defaults fell through to
the default arm which calls value.toString() on a byte[] — yielding the JVM
identity-hash form like "[B@1f32e575". SHOW CREATE TABLE on a column with a
non-natural BYTES default would emit structurally invalid DDL.

Added a BYTES case that hex-encodes the byte[] via BytesUtils.toHexString,
matching the convention used by FieldSpec.getDefaultNullValueString and
toJsonObject. Added a regression test asserting the emitted DDL contains
the quoted hex string and does not leak the byte[] identity-hash form.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../pinot/sql/ddl/reverse/SchemaEmitter.java  | 10 ++++++++++
 .../ddl/reverse/CanonicalDdlEmitterTest.java  | 20 +++++++++++++++++++
 2 files changed, 30 insertions(+)

diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
index 108471e316ea..cd2799d87247 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
@@ -31,6 +31,7 @@
 import org.apache.pinot.spi.data.MetricFieldSpec;
 import org.apache.pinot.spi.data.Schema;
 import org.apache.pinot.spi.data.TimeFieldSpec;
+import org.apache.pinot.spi.utils.BytesUtils;
 
 
 /**
@@ -233,6 +234,15 @@ private static String emitDefault(Object value, DataType dt) {
         // small magnitudes, which Calcite's Literal() rule does not accept. toPlainString()
         // always produces the decimal form so the round-trip stays grammar-legal.
         return value instanceof BigDecimal ? ((BigDecimal) value).toPlainString() : value.toString();
+      case BYTES:
+        // Pinot stores BYTES defaults internally as byte[]. byte[].toString() returns the JVM
+        // identity-hash form (e.g. "[B@1f32e575") which would emit structurally invalid DDL.
+        // Convert to hex matching FieldSpec.getDefaultNullValueString and toJsonObject so the
+        // canonical DDL re-parses back to the same byte content via FieldSpec.setDefaultNullValue.
+        if (value instanceof byte[]) {
+          return SqlIdentifiers.quoteString(BytesUtils.toHexString((byte[]) value));
+        }
+        return SqlIdentifiers.quoteString(value.toString());
       default:
         return SqlIdentifiers.quoteString(value.toString());
     }
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
index 7cd4f21798fe..15be8ab3aef7 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
@@ -224,6 +224,26 @@ public void booleanDefaultEmittedAsSqlLiteral() {
         "must not emit raw integer encoding (DEFAULT 1); got:\n" + emitted);
   }
 
+  /**
+   * Regression: BYTES non-natural-default emit must hex-encode the byte[] rather than fall
+   * through to value.toString() which would emit "[B@" identity-hash garbage.
+   */
+  @Test
+  public void bytesNonNaturalDefaultEmittedAsHex() {
+    Schema schema = new Schema();
+    schema.setSchemaName("t");
+    DimensionFieldSpec dim = new DimensionFieldSpec("blob", DataType.BYTES, true);
+    dim.setDefaultNullValue("deadbeef");
+    schema.addField(dim);
+
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build();
+    String emitted = CanonicalDdlEmitter.emit(schema, config);
+    assertTrue(emitted.contains("DEFAULT 'deadbeef'"),
+        "BYTES default must be emitted as quoted hex string; got:\n" + emitted);
+    assertFalse(emitted.contains("[B@"),
+        "BYTES default must not leak the JVM byte[] identity-hash form; got:\n" + emitted);
+  }
+
   /**
    * Regression: TIMESTAMP DEFAULT emission must use UTC ISO-8601 form (Instant.toString)
    * rather than java.sql.Timestamp.toString, which formats in the JVM's default time zone

From 7d890fc57ca4223782d8d29c054d3ba0a02cb736 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sat, 25 Apr 2026 07:49:50 -0700
Subject: [PATCH 26/32] Replace inline FQCNs in DDL test files with imports

Three test files used fully-qualified class names inline (java.math.BigDecimal,
java.util.LinkedHashMap, java.util.Map, org.apache.pinot.spi.config.table.TableTaskConfig,
TableCustomConfig) where the corresponding imports could be added. CLAUDE.md
prefers imports over FQCN; this is a style cleanup the reviewer surfaced as a
MINOR nit. No behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../PinotDdlRestletResourceUnitTest.java      |  5 ++--
 .../ddl/reverse/CanonicalDdlEmitterTest.java  | 23 +++++++++++--------
 .../sql/ddl/roundtrip/RoundTripTest.java      | 23 ++++++++++---------
 3 files changed, 29 insertions(+), 22 deletions(-)

diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
index 082cd09a9a6f..6c77eb1a815a 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
@@ -18,6 +18,7 @@
  */
 package org.apache.pinot.controller.api.resources;
 
+import java.math.BigDecimal;
 import java.util.Arrays;
 import java.util.Collections;
 import org.apache.pinot.spi.data.DateTimeFieldSpec;
@@ -177,13 +178,13 @@ public void acceptsScaleShiftedBigDecimalDefault() {
     Schema stored = new Schema();
     stored.setSchemaName("t");
     MetricFieldSpec storedSpec = new MetricFieldSpec("amount", DataType.BIG_DECIMAL,
-        new java.math.BigDecimal("1.0"));
+        new BigDecimal("1.0"));
     stored.addField(storedSpec);
 
     Schema compiled = new Schema();
     compiled.setSchemaName("t");
     MetricFieldSpec compiledSpec = new MetricFieldSpec("amount", DataType.BIG_DECIMAL,
-        new java.math.BigDecimal("1"));
+        new BigDecimal("1"));
     compiled.addField(compiledSpec);
 
     assertNull(PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled),
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
index 15be8ab3aef7..56dc1145ec5c 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
@@ -18,8 +18,13 @@
  */
 package org.apache.pinot.sql.ddl.reverse;
 
+import java.math.BigDecimal;
 import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.Map;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableCustomConfig;
+import org.apache.pinot.spi.config.table.TableTaskConfig;
 import org.apache.pinot.spi.config.table.TableType;
 import org.apache.pinot.spi.data.DateTimeFieldSpec;
 import org.apache.pinot.spi.data.DimensionFieldSpec;
@@ -114,7 +119,7 @@ public void streamConfigsRoundTripWithOriginalKeys() {
         .setSchemaName("clicks")
         .addDateTime("ts", DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
         .build();
-    java.util.Map streamCfgs = new java.util.LinkedHashMap<>();
+    Map streamCfgs = new LinkedHashMap<>();
     streamCfgs.put("stream.kafka.topic.name", "click_events");
     streamCfgs.put("stream.kafka.consumer.factory.class.name", "KafkaConsumerFactory");
     streamCfgs.put("realtime.segment.flush.threshold.rows", "500000");
@@ -174,7 +179,7 @@ public void bigDecimalDefaultEmitsPlainString() {
     Schema schema = new Schema();
     schema.setSchemaName("t");
     MetricFieldSpec metric = new MetricFieldSpec("amount", DataType.BIG_DECIMAL,
-        new java.math.BigDecimal("1E+30"));
+        new BigDecimal("1E+30"));
     schema.addField(metric);
 
     TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build();
@@ -195,7 +200,7 @@ public void bigDecimalAtNaturalDefaultDoesNotEmitDefault() {
     Schema schema = new Schema();
     schema.setSchemaName("t");
     MetricFieldSpec metric = new MetricFieldSpec("amount", DataType.BIG_DECIMAL,
-        new java.math.BigDecimal("0.0"));
+        new BigDecimal("0.0"));
     schema.addField(metric);
 
     TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build();
@@ -316,14 +321,14 @@ public void taskConfigsEmittedAsPrefixedKeys() {
         .setSchemaName("events")
         .addSingleValueDimension("id", DataType.INT)
         .build();
-    java.util.Map> tasks = new java.util.LinkedHashMap<>();
-    java.util.Map rtoCfg = new java.util.LinkedHashMap<>();
+    Map> tasks = new LinkedHashMap<>();
+    Map rtoCfg = new LinkedHashMap<>();
     rtoCfg.put("bucketTimePeriod", "1d");
     rtoCfg.put("maxNumRecordsPerSegment", "5000000");
     tasks.put("RealtimeToOfflineSegmentsTask", rtoCfg);
     TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
         .setTableName("events")
-        .setTaskConfig(new org.apache.pinot.spi.config.table.TableTaskConfig(tasks))
+        .setTaskConfig(new TableTaskConfig(tasks))
         .build();
 
     String emitted = CanonicalDdlEmitter.emit(schema, config);
@@ -341,7 +346,7 @@ public void customConfigEmittedVerbatim() {
         .build();
     TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
         .setTableName("t")
-        .setCustomConfig(new org.apache.pinot.spi.config.table.TableCustomConfig(
+        .setCustomConfig(new TableCustomConfig(
             Collections.singletonMap("mySpecialKey", "someValue")))
         .build();
 
@@ -374,7 +379,7 @@ public void customConfigKeyShadowingPromotedKeyRejected() {
         .build();
     TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
         .setTableName("t")
-        .setCustomConfig(new org.apache.pinot.spi.config.table.TableCustomConfig(
+        .setCustomConfig(new TableCustomConfig(
             Collections.singletonMap("ingestionConfig", "anything")))
         .build();
     try {
@@ -397,7 +402,7 @@ public void customConfigKeyShadowingTaskPrefixRejected() {
         .build();
     TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
         .setTableName("t")
-        .setCustomConfig(new org.apache.pinot.spi.config.table.TableCustomConfig(
+        .setCustomConfig(new TableCustomConfig(
             Collections.singletonMap("task.MyTask.foo", "bar")))
         .build();
     try {
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
index a376f9a5ad39..ccac382dc3f3 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
@@ -27,6 +27,7 @@
 import java.util.concurrent.TimeUnit;
 import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.TableCustomConfig;
+import org.apache.pinot.spi.config.table.TableTaskConfig;
 import org.apache.pinot.spi.config.table.TableType;
 import org.apache.pinot.spi.config.table.ingestion.BatchIngestionConfig;
 import org.apache.pinot.spi.config.table.ingestion.IngestionConfig;
@@ -114,9 +115,9 @@ public void offlineTableWithIndexingConfig() {
     TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
         .setTableName("events")
         .setSortedColumn("country")
-        .setInvertedIndexColumns(java.util.Arrays.asList("city"))
-        .setNoDictionaryColumns(java.util.Arrays.asList("amount"))
-        .setBloomFilterColumns(java.util.Arrays.asList("country"))
+        .setInvertedIndexColumns(Arrays.asList("city"))
+        .setNoDictionaryColumns(Arrays.asList("amount"))
+        .setBloomFilterColumns(Arrays.asList("country"))
         .setNullHandlingEnabled(true)
         .build();
     assertRoundTrip(schema, config);
@@ -158,7 +159,7 @@ public void offlineTableWithTaskConfig() {
     tasks.put("SegmentRefreshTask", refresh);
     TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
         .setTableName("events")
-        .setTaskConfig(new org.apache.pinot.spi.config.table.TableTaskConfig(tasks))
+        .setTaskConfig(new TableTaskConfig(tasks))
         .build();
     assertRoundTrip(schema, config);
   }
@@ -263,12 +264,12 @@ public void promotedScalarsAddedInSlice2RoundTrip() {
         .setBrokerTenant("tenantA")
         .setServerTenant("tenantB")
         .setSortedColumn("country")
-        .setInvertedIndexColumns(java.util.Arrays.asList("city"))
-        .setNoDictionaryColumns(java.util.Arrays.asList("amount"))
-        .setOnHeapDictionaryColumns(java.util.Arrays.asList("country"))
-        .setVarLengthDictionaryColumns(java.util.Arrays.asList("city"))
-        .setBloomFilterColumns(java.util.Arrays.asList("country"))
-        .setRangeIndexColumns(java.util.Arrays.asList("amount"))
+        .setInvertedIndexColumns(Arrays.asList("city"))
+        .setNoDictionaryColumns(Arrays.asList("amount"))
+        .setOnHeapDictionaryColumns(Arrays.asList("country"))
+        .setVarLengthDictionaryColumns(Arrays.asList("city"))
+        .setBloomFilterColumns(Arrays.asList("country"))
+        .setRangeIndexColumns(Arrays.asList("amount"))
         .setNullHandlingEnabled(true)
         .setAggregateMetrics(true)
         .setPeerSegmentDownloadScheme("https")
@@ -276,7 +277,7 @@ public void promotedScalarsAddedInSlice2RoundTrip() {
         .setSegmentVersion("v3")
         .setDeletedSegmentsRetentionPeriod("14d")
         .setDescription("a kitchen-sink test table")
-        .setTags(java.util.Arrays.asList("ourTeam", "metricsPipeline"))
+        .setTags(Arrays.asList("ourTeam", "metricsPipeline"))
         .build();
     assertRoundTrip(schema, config);
   }

From ed51cfb43549bd57ef63412c0b4afdf18d37f295 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Sun, 26 Apr 2026 19:01:06 -0700
Subject: [PATCH 27/32] Add DESIGN.md and module README for pinot-sql-ddl

DESIGN.md covers the SQL DDL feature end-to-end: goals/non-goals, module
layering, grammar productions, type mapping, property routing rules
(promoted scalar / JSON blob / stream prefix / task prefix / custom
fallback), validation pipeline shared with POST /tables, hybrid
second-variant CREATE semantics, canonical-DDL emission contract,
REST endpoint pipeline, no-fingerprinting auth, forward-compat hooks,
backward-compat guarantees, concurrency model, test strategy across
the six test suites, known limitations, and the decision log capturing
why we chose single dispatch endpoint, TABLE_TYPE clause shape,
no-rollback CREATE, up-front double-auth, SMALLINT/TINYINT rejection,
and the separate pinot-sql-ddl module placement.

README.md is a short module entry point with a quickstart example,
the sub-package layout, and pointers to DESIGN.md, package-info.java,
and PR #18241 for the user-manual-style examples.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .../src/main/codegen/includes/parserImpls.ftl |  40 +-
 .../pinot/sql/parsers/CalciteSqlParser.java   | 104 ++--
 .../parser/SqlPinotColumnDeclaration.java     |  50 +-
 .../parsers/parser/SqlPinotCreateTable.java   |  44 +-
 .../sql/parsers/parser/SqlPinotDropTable.java |  24 +-
 .../sql/parsers/parser/SqlPinotProperty.java  |  16 +-
 .../parser/SqlPinotShowCreateTable.java       |  20 +-
 .../parsers/parser/SqlPinotShowTables.java    |  14 +-
 .../pinot/sql/parsers/PinotDdlParserTest.java |   8 +-
 .../resources/PinotDdlRestletResource.java    | 456 ++++++++++-----
 .../api/resources/PinotQueryResource.java     |   6 +-
 .../resources/ddl/DdlExecutionRequest.java    |   2 +-
 .../resources/ddl/DdlExecutionResponse.java   |  24 +-
 .../api/PinotDdlRestletResourceTest.java      |  10 +-
 .../PinotDdlRestletResourceUnitTest.java      | 223 +++++++-
 .../api/resources/PinotQueryResourceTest.java |  29 +
 pinot-sql-ddl/DESIGN.md                       | 521 ++++++++++++++++++
 pinot-sql-ddl/README.md                       |  61 ++
 .../sql/ddl/compile/CompiledCreateTable.java  |   2 +-
 .../pinot/sql/ddl/compile/CompiledDdl.java    |  12 +-
 .../sql/ddl/compile/CompiledDropTable.java    |  10 +-
 .../ddl/compile/CompiledShowCreateTable.java  |  24 +-
 .../sql/ddl/compile/CompiledShowTables.java   |   2 +-
 .../pinot/sql/ddl/compile/DataTypeMapper.java |  16 +-
 .../ddl/compile/DdlCompilationException.java  |   8 +-
 .../pinot/sql/ddl/compile/DdlCompiler.java    |  75 ++-
 .../pinot/sql/ddl/compile/DdlOperation.java   |   2 +-
 .../sql/ddl/compile/PropertyMapping.java      | 112 ++--
 .../apache/pinot/sql/ddl/package-info.java    | 102 ++--
 .../pinot/sql/ddl/resolved/ColumnRole.java    |  12 +-
 .../resolved/ResolvedColumnDefinition.java    |  10 +-
 .../ddl/resolved/ResolvedTableDefinition.java |  22 +-
 .../sql/ddl/reverse/CanonicalDdlEmitter.java  |  70 ++-
 .../sql/ddl/reverse/PropertyExtractor.java    | 204 +++++--
 .../pinot/sql/ddl/reverse/SchemaEmitter.java  |  34 +-
 .../pinot/sql/ddl/reverse/SqlIdentifiers.java |  26 +-
 .../sql/ddl/compile/DdlCompilerTest.java      |  72 ++-
 .../ddl/reverse/CanonicalDdlEmitterTest.java  |  75 +--
 .../sql/ddl/roundtrip/RoundTripTest.java      | 127 ++++-
 39 files changed, 1884 insertions(+), 785 deletions(-)
 create mode 100644 pinot-sql-ddl/DESIGN.md
 create mode 100644 pinot-sql-ddl/README.md

diff --git a/pinot-common/src/main/codegen/includes/parserImpls.ftl b/pinot-common/src/main/codegen/includes/parserImpls.ftl
index 856a230bd9a1..de942540dcd4 100644
--- a/pinot-common/src/main/codegen/includes/parserImpls.ftl
+++ b/pinot-common/src/main/codegen/includes/parserImpls.ftl
@@ -45,10 +45,8 @@ SqlNodeList DataFileDefList() :
     }
 }
 
-/**
- * INSERT INTO [db_name.]table_name
- *   FROM [ FILE | ARCHIVE ] 'file_uri' [, [ FILE | ARCHIVE ] 'file_uri' ]
- */
+/// INSERT INTO [db_name.]table_name
+/// FROM [ FILE | ARCHIVE ] 'file_uri' [, [ FILE | ARCHIVE ] 'file_uri' ]
 SqlInsertFromFile SqlInsertFromFile() :
 {
     SqlParserPos pos;
@@ -110,15 +108,13 @@ SqlNode SqlPhysicalExplain() :
     }
 }
 
-/**
- * CREATE TABLE [IF NOT EXISTS] [db.]name (
- *   col TYPE [NULL | NOT NULL] [DEFAULT literal]
- *     [ DIMENSION | METRIC | DATETIME FORMAT 'fmt' GRANULARITY 'gran' ],
- *   ...
- * )
- * TABLE_TYPE = OFFLINE | REALTIME
- * [ PROPERTIES ( 'k' = 'v', ... ) ]
- */
+/// CREATE TABLE [IF NOT EXISTS] [db.]name (
+/// col TYPE [NULL | NOT NULL] [DEFAULT literal]
+/// [ DIMENSION | METRIC | DATETIME FORMAT 'fmt' GRANULARITY 'gran' ],
+/// ...
+/// )
+/// TABLE_TYPE = OFFLINE | REALTIME
+/// [ PROPERTIES ( 'k' = 'v', ... ) ]
 SqlNode SqlPinotCreateTable() :
 {
     SqlParserPos pos;
@@ -263,9 +259,7 @@ SqlPinotProperty PinotProperty() :
     }
 }
 
-/**
- * DROP TABLE [IF EXISTS] [db.]name [TYPE OFFLINE | REALTIME]
- */
+/// DROP TABLE [IF EXISTS] [db.]name [TYPE OFFLINE | REALTIME]
 SqlNode SqlPinotDropTable() :
 {
     SqlParserPos pos;
@@ -284,14 +278,12 @@ SqlNode SqlPinotDropTable() :
     }
 }
 
-/**
- * SHOW TABLES [FROM db]
- *  | SHOW CREATE TABLE [db.]name [TYPE OFFLINE | REALTIME]
- *
- * Both grammar branches share a leading  token; combining them into a single entry point
- * keeps the JavaCC choice unambiguous (no need for LOOKAHEAD across multiple statementParser
- * methods that all start with SHOW).
- */
+/// SHOW TABLES [FROM db]
+/// | SHOW CREATE TABLE [db.]name [TYPE OFFLINE | REALTIME]
+///
+/// Both grammar branches share a leading `SHOW` token; combining them into a single entry point
+/// keeps the JavaCC choice unambiguous (no need for LOOKAHEAD across multiple statementParser
+/// methods that all start with SHOW).
 SqlNode SqlPinotShow() :
 {
     SqlParserPos pos;
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
index 8159836520bd..576906fd457b 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
@@ -179,9 +179,7 @@ public static PinotQuery compileToPinotQuery(String sql)
     return compileToPinotQuery(compileToSqlNodeAndOptions(sql));
   }
 
-  /**
-   * Should only be used for testing query rewriters.
-   */
+  /// Should only be used for testing query rewriters.
   public static PinotQuery compileToPinotQueryWithoutRewrites(String sql) {
     return compileWithoutRewrite(compileToSqlNodeAndOptions(sql).getSqlNode());
   }
@@ -380,9 +378,7 @@ private static List getAliasLeftExpressionsFromDistinctExpression(Fu
     return expressions;
   }
 
-  /**
-   * Check recursively if an expression contains any reference not appearing in the GROUP BY clause.
-   */
+  /// Check recursively if an expression contains any reference not appearing in the GROUP BY clause.
   private static boolean expressionOutsideGroupByList(Expression expr, Set groupByExprs) {
     // return early for Literal, Aggregate and if we have an exact match
     if (expr.getType() == ExpressionType.LITERAL || isAggregateExpression(expr) || groupByExprs.contains(expr)) {
@@ -424,13 +420,11 @@ public static boolean isAsFunction(Expression expression) {
     return function != null && function.getOperator().equals("as");
   }
 
-  /**
-   * Extract all the identifiers from given expressions.
-   *
-   * @param expressions
-   * @param excludeAs if true, ignores the right side identifier for AS function.
-   * @return all the identifier names.
-   */
+  /// Extract all the identifiers from given expressions.
+  ///
+  /// @param expressions
+  /// @param excludeAs if true, ignores the right side identifier for AS function.
+  /// @return all the identifier names.
   public static Set extractIdentifiers(List expressions, boolean excludeAs) {
     Set identifiers = new HashSet<>();
     for (Expression expression : expressions) {
@@ -451,14 +445,12 @@ public static Set extractIdentifiers(List expressions, boole
     return identifiers;
   }
 
-  /**
-   * Compiles a String expression into {@link Expression}.
-   *
-   * @param expression String expression.
-   * @return {@link Expression} equivalent of the string.
-   *
-   * @throws SqlCompilationException if String is not a valid expression.
-   */
+  /// Compiles a String expression into [Expression].
+  ///
+  /// @param expression String expression.
+  /// @return [Expression] equivalent of the string.
+  ///
+  /// @throws SqlCompilationException if String is not a valid expression.
   public static Expression compileToExpression(String expression) {
     SqlNode sqlNode;
     try (StringReader inStream = new StringReader(expression)) {
@@ -624,14 +616,12 @@ public static void queryRewrite(PinotQuery pinotQuery) {
     validate(pinotQuery);
   }
 
-  /**
-   * Applies a specific query rewriter to the given PinotQuery and validates the result.
-   * This method searches for a rewriter by class name and applies it to transform the query.
-   *
-   * @param pinotQuery the query to be rewritten
-   * @param rewriterClass the class name of the query rewriter to apply
-   * @throws IllegalArgumentException if no rewriter with the specified class name is found
-   */
+  /// Applies a specific query rewriter to the given PinotQuery and validates the result.
+  /// This method searches for a rewriter by class name and applies it to transform the query.
+  ///
+  /// @param pinotQuery the query to be rewritten
+  /// @param rewriterClass the class name of the query rewriter to apply
+  /// @throws IllegalArgumentException if no rewriter with the specified class name is found
   public static void queryRewrite(PinotQuery pinotQuery, Class rewriterClass) {
     QueryRewriter queryRewriter = QUERY_REWRITERS.stream()
         .filter(rewriter -> rewriter.getClass().equals(rewriterClass))
@@ -720,13 +710,11 @@ private static Expression convertOrderBy(SqlNode node, boolean createAscExpressi
     return expression;
   }
 
-  /**
-   * DISTINCT is implemented as an aggregation function so need to take the select list items
-   * and convert them into a single function expression for handing over to execution engine
-   * either as a PinotQuery or BrokerRequest via conversion
-   * @param selectList select list items
-   * @return DISTINCT function expression
-   */
+  /// DISTINCT is implemented as an aggregation function so need to take the select list items
+  /// and convert them into a single function expression for handing over to execution engine
+  /// either as a PinotQuery or BrokerRequest via conversion
+  /// @param selectList select list items
+  /// @return DISTINCT function expression
   private static Expression convertDistinctAndSelectListToFunctionExpression(SqlNodeList selectList) {
     List operands = new ArrayList<>(selectList.size());
     for (SqlNode node : selectList) {
@@ -889,25 +877,23 @@ private static Expression compileFunctionExpression(SqlBasicCall functionNode) {
     }
   }
 
-  /**
-   * Convert Calcite operator tree made up of ITEM and DOT functions to an identifier. For example, the operator tree
-   * shown below will be converted to IDENTIFIER "jsoncolumn.data[0][1].a.b[0]".
-   *
-   * ├── ITEM(jsoncolumn.data[0][1].a.b[0])
-   *      ├── LITERAL (0)
-   *      └── DOT (jsoncolumn.daa[0][1].a.b)
-   *            ├── IDENTIFIER (b)
-   *            └── DOT (jsoncolumn.data[0][1].a)
-   *                  ├── IDENTIFIER (a)
-   *                  └── ITEM (jsoncolumn.data[0][1])
-   *                        ├── LITERAL (1)
-   *                        └── ITEM (jsoncolumn.data[0])
-   *                              ├── LITERAL (1)
-   *                              └── IDENTIFIER (jsoncolumn.data)
-   *
-   * @param functionNode Root node of the DOT and/or ITEM operator function chain.
-   * @param pathBuilder StringBuilder representation of path represented by DOT and/or ITEM function chain.
-   */
+  /// Convert Calcite operator tree made up of ITEM and DOT functions to an identifier. For example, the operator tree
+  /// shown below will be converted to IDENTIFIER "jsoncolumn.data[0][1].a.b[0]".
+  ///
+  /// ├── ITEM(jsoncolumn.data[0][1].a.b[0])
+  /// ├── LITERAL (0)
+  /// └── DOT (jsoncolumn.daa[0][1].a.b)
+  /// ├── IDENTIFIER (b)
+  /// └── DOT (jsoncolumn.data[0][1].a)
+  /// ├── IDENTIFIER (a)
+  /// └── ITEM (jsoncolumn.data[0][1])
+  /// ├── LITERAL (1)
+  /// └── ITEM (jsoncolumn.data[0])
+  /// ├── LITERAL (1)
+  /// └── IDENTIFIER (jsoncolumn.data)
+  ///
+  /// @param functionNode Root node of the DOT and/or ITEM operator function chain.
+  /// @param pathBuilder StringBuilder representation of path represented by DOT and/or ITEM function chain.
   private static void compilePathExpression(SqlBasicCall functionNode, StringBuilder pathBuilder) {
     List operands = functionNode.getOperandList();
 
@@ -940,9 +926,7 @@ private static void compilePathExpression(SqlBasicCall functionNode, StringBuild
     }
   }
 
-  /**
-   * Helper method to flatten the operands for the AND expression.
-   */
+  /// Helper method to flatten the operands for the AND expression.
   private static Expression compileAndExpression(SqlBasicCall andNode) {
     List operands = new ArrayList<>();
     for (SqlNode childNode : andNode.getOperandList()) {
@@ -956,9 +940,7 @@ private static Expression compileAndExpression(SqlBasicCall andNode) {
     return RequestUtils.getFunctionExpression(FilterKind.AND.name(), operands);
   }
 
-  /**
-   * Helper method to flatten the operands for the OR expression.
-   */
+  /// Helper method to flatten the operands for the OR expression.
   private static Expression compileOrExpression(SqlBasicCall orNode) {
     List operands = new ArrayList<>();
     for (SqlNode childNode : orNode.getOperandList()) {
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotColumnDeclaration.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotColumnDeclaration.java
index 70b40c96c31c..dac9a63f8085 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotColumnDeclaration.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotColumnDeclaration.java
@@ -33,29 +33,27 @@
 import org.apache.calcite.sql.parser.SqlParserPos;
 
 
-/**
- * Single column declaration inside the column list of a Pinot {@link SqlPinotCreateTable}.
- *
- * 
    Grammar:
- * 
    {@code
- *   col_name DATA_TYPE [NULL | NOT NULL] [DEFAULT literal]
- *     [ DIMENSION | METRIC | DATETIME FORMAT 'fmt' GRANULARITY 'gran' ]
- * }
    
- *
- * 
    The {@code role} field is one of "DIMENSION", "METRIC", "DATETIME", or {@code null}
- * (unspecified, defer inference to the compiler). When {@code role} is "DATETIME", the
- * {@code dateTimeFormat} and {@code dateTimeGranularity} string literals are required.
- * When {@code role} is "DIMENSION", the optional {@code ARRAY} suffix marks the column as
- * multi-value (Pinot MV dimension).
- *
- * 
    DEFAULT semantics: the {@code DEFAULT literal} clause maps to Pinot's
- * {@code FieldSpec.defaultNullValue}, NOT to standard SQL's "value substituted on insert when
- * column is omitted". Pinot's defaultNullValue is applied at ingestion time when the source
- * record contains a null/missing value for the column. Users coming from standard SQL should
- * not expect this clause to interact with INSERT statements.
- *
- * 
    This class is not thread-safe; instances should not be mutated after construction.
- */
+/// Single column declaration inside the column list of a Pinot [SqlPinotCreateTable].
+///
+/// Grammar:
+/// ```
+///   col_name DATA_TYPE [NULL | NOT NULL] [DEFAULT literal]
+///     [ DIMENSION | METRIC | DATETIME FORMAT 'fmt' GRANULARITY 'gran' ]
+/// ```
+///
+/// The `role` field is one of "DIMENSION", "METRIC", "DATETIME", or `null`
+/// (unspecified, defer inference to the compiler). When `role` is "DATETIME", the
+/// `dateTimeFormat` and `dateTimeGranularity` string literals are required.
+/// When `role` is "DIMENSION", the optional `ARRAY` suffix marks the column as
+/// multi-value (Pinot MV dimension).
+///
+/// **DEFAULT semantics**: the `DEFAULT literal` clause maps to Pinot's
+/// `FieldSpec.defaultNullValue`, NOT to standard SQL's "value substituted on insert when
+/// column is omitted". Pinot's defaultNullValue is applied at ingestion time when the source
+/// record contains a null/missing value for the column. Users coming from standard SQL should
+/// not expect this clause to interact with INSERT statements.
+///
+/// This class is not thread-safe; instances should not be mutated after construction.
 public class SqlPinotColumnDeclaration extends SqlCall {
   private static final SqlSpecialOperator OPERATOR =
       new SqlSpecialOperator("COLUMN_DECL", SqlKind.COLUMN_DECL);
@@ -106,9 +104,7 @@ public SqlNode getDefaultValue() {
     return _defaultValue;
   }
 
-  /**
-   * @return one of "DIMENSION", "METRIC", "DATETIME", or {@code null} when unspecified.
-   */
+  /// @return one of "DIMENSION", "METRIC", "DATETIME", or `null` when unspecified.
   @Nullable
   public String getRole() {
     return _role;
@@ -124,7 +120,7 @@ public SqlLiteral getDateTimeGranularity() {
     return _dateTimeGranularity;
   }
 
-  /** Returns true if this is a multi-value dimension (declared with {@code DIMENSION ARRAY}). */
+  /// Returns true if this is a multi-value dimension (declared with `DIMENSION ARRAY`).
   public boolean isMultiValue() {
     return _multiValue;
   }
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
index 1a4c08d84264..9295575eb5d9 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotCreateTable.java
@@ -33,29 +33,27 @@
 import org.apache.calcite.sql.parser.SqlParserPos;
 
 
-/**
- * Pinot-native {@code CREATE TABLE} DDL statement.
- *
- * 
    Syntax:
- * 
    {@code
- *   CREATE TABLE [IF NOT EXISTS] [db.]name (
- *     col TYPE [NULL | NOT NULL] [DEFAULT literal] [DIMENSION | METRIC],
- *     col TYPE DATETIME FORMAT 'fmt' GRANULARITY 'gran',
- *     ...
- *   )
- *   [PRIMARY KEY (col, ...)]
- *   TABLE_TYPE = OFFLINE | REALTIME
- *   PROPERTIES (
- *     'key' = 'value',
- *     ...
- *   )
- * }
    
- *
- * 
    This is a parse-time AST node only. Semantic validation (data type recognition, role
- * inference, property mapping) happens in {@code DdlCompiler} in the {@code pinot-sql-ddl} module.
- *
- * 
    This class is not thread-safe; instances should not be mutated after construction.
- */
+/// Pinot-native `CREATE TABLE` DDL statement.
+///
+/// Syntax:
+/// ```
+///   CREATE TABLE [IF NOT EXISTS] [db.]name (
+///     col TYPE [NULL | NOT NULL] [DEFAULT literal] [DIMENSION | METRIC],
+///     col TYPE DATETIME FORMAT 'fmt' GRANULARITY 'gran',
+///     ...
+///   )
+///   [PRIMARY KEY (col, ...)]
+///   TABLE_TYPE = OFFLINE | REALTIME
+///   PROPERTIES (
+///     'key' = 'value',
+///     ...
+///   )
+/// ```
+///
+/// This is a parse-time AST node only. Semantic validation (data type recognition, role
+/// inference, property mapping) happens in `DdlCompiler` in the `pinot-sql-ddl` module.
+///
+/// This class is not thread-safe; instances should not be mutated after construction.
 public class SqlPinotCreateTable extends SqlCall {
   private static final SqlSpecialOperator OPERATOR =
       new SqlSpecialOperator("CREATE_TABLE", SqlKind.CREATE_TABLE);
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotDropTable.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotDropTable.java
index 2640872da0d4..7564f5137758 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotDropTable.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotDropTable.java
@@ -32,16 +32,14 @@
 import org.apache.calcite.sql.parser.SqlParserPos;
 
 
-/**
- * Pinot-native {@code DROP TABLE} DDL statement.
- *
- * 
    Syntax: {@code DROP TABLE [IF EXISTS] [db.]name [TYPE OFFLINE | REALTIME]}
- *
- * 
    The optional {@code TYPE} clause restricts the drop to one physical table when the logical
- * name has both OFFLINE and REALTIME variants. When absent, both variants are dropped.
- *
- * 
    This class is not thread-safe; instances should not be mutated after construction.
- */
+/// Pinot-native `DROP TABLE` DDL statement.
+///
+/// Syntax: `DROP TABLE [IF EXISTS] [db.]name [TYPE OFFLINE | REALTIME]`
+///
+/// The optional `TYPE` clause restricts the drop to one physical table when the logical
+/// name has both OFFLINE and REALTIME variants. When absent, both variants are dropped.
+///
+/// This class is not thread-safe; instances should not be mutated after construction.
 public class SqlPinotDropTable extends SqlCall {
   private static final SqlSpecialOperator OPERATOR =
       new SqlSpecialOperator("DROP_TABLE", SqlKind.DROP_TABLE);
@@ -66,10 +64,8 @@ public boolean isIfExists() {
     return _ifExists;
   }
 
-  /**
-   * @return the explicit table type ("OFFLINE" or "REALTIME") if specified, or {@code null} for
-   *     "drop both variants".
-   */
+  /// @return the explicit table type ("OFFLINE" or "REALTIME") if specified, or `null` for
+  /// "drop both variants".
   @Nullable
   public SqlLiteral getTableType() {
     return _tableType;
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotProperty.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotProperty.java
index b0077ab752c1..f4bdf6c94264 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotProperty.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotProperty.java
@@ -30,15 +30,13 @@
 import org.apache.calcite.sql.parser.SqlParserPos;
 
 
-/**
- * One {@code 'key' = 'value'} entry inside a {@code PROPERTIES (...)} clause.
- *
- * 
    Both key and value are parsed as quoted string literals to keep the property surface area
- * uniform and forward-compatible. This lets stream/minion-task configs (whose keys evolve outside
- * the DDL grammar) be passed through verbatim without grammar changes.
- *
- * 
    This class is not thread-safe; instances should not be mutated after construction.
- */
+/// One `'key' = 'value'` entry inside a `PROPERTIES (...)` clause.
+///
+/// Both key and value are parsed as quoted string literals to keep the property surface area
+/// uniform and forward-compatible. This lets stream/minion-task configs (whose keys evolve outside
+/// the DDL grammar) be passed through verbatim without grammar changes.
+///
+/// This class is not thread-safe; instances should not be mutated after construction.
 public class SqlPinotProperty extends SqlCall {
   private static final SqlSpecialOperator OPERATOR =
       new SqlSpecialOperator("PROPERTY", SqlKind.OTHER);
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowCreateTable.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowCreateTable.java
index 2f4016243a92..799aa78ac243 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowCreateTable.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowCreateTable.java
@@ -32,15 +32,13 @@
 import org.apache.calcite.sql.parser.SqlParserPos;
 
 
-/**
- * Pinot-native {@code SHOW CREATE TABLE [db.]name [TYPE OFFLINE | REALTIME]} DDL statement.
- *
- * 
    Parse-time AST node only. The optional {@code TYPE OFFLINE | REALTIME} clause carries the
- * caller's explicit preference; absence leaves the choice to the executor. Disambiguation policy
- * for the both-variants-exist case is the controller's responsibility, not the parser's.
- *
- * 
    This class is not thread-safe; instances should not be mutated after construction.
- */
+/// Pinot-native `SHOW CREATE TABLE [db.]name [TYPE OFFLINE | REALTIME]` DDL statement.
+///
+/// Parse-time AST node only. The optional `TYPE OFFLINE | REALTIME` clause carries the
+/// caller's explicit preference; absence leaves the choice to the executor. Disambiguation policy
+/// for the both-variants-exist case is the controller's responsibility, not the parser's.
+///
+/// This class is not thread-safe; instances should not be mutated after construction.
 public class SqlPinotShowCreateTable extends SqlCall {
   private static final SqlSpecialOperator OPERATOR =
       new SqlSpecialOperator("SHOW_CREATE_TABLE", SqlKind.OTHER_DDL);
@@ -58,9 +56,7 @@ public SqlIdentifier getName() {
     return _name;
   }
 
-  /**
-   * @return the explicit table type ("OFFLINE" or "REALTIME") if specified, else {@code null}.
-   */
+  /// @return the explicit table type ("OFFLINE" or "REALTIME") if specified, else `null`.
   @Nullable
   public SqlLiteral getTableType() {
     return _tableType;
diff --git a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java
index 4e72d7306a18..676b88f5d226 100644
--- a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java
+++ b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPinotShowTables.java
@@ -31,12 +31,10 @@
 import org.apache.calcite.sql.parser.SqlParserPos;
 
 
-/**
- * Pinot-native {@code SHOW TABLES [FROM db]} DDL statement. Lists tables in the given database
- * (or the default database when none is specified).
- *
- * 
    This class is not thread-safe; instances should not be mutated after construction.
- */
+/// Pinot-native `SHOW TABLES [FROM db]` DDL statement. Lists tables in the given database
+/// (or the default database when none is specified).
+///
+/// This class is not thread-safe; instances should not be mutated after construction.
 public class SqlPinotShowTables extends SqlCall {
   private static final SqlSpecialOperator OPERATOR =
       new SqlSpecialOperator("SHOW_TABLES", SqlKind.OTHER_DDL);
@@ -48,9 +46,7 @@ public SqlPinotShowTables(SqlParserPos pos, @Nullable SqlIdentifier database) {
     _database = database;
   }
 
-  /**
-   * @return the explicit database identifier when {@code FROM } is supplied, else {@code null}.
-   */
+  /// @return the explicit database identifier when ``FROM db`` is supplied, else `null`.
   @Nullable
   public SqlIdentifier getDatabase() {
     return _database;
diff --git a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
index 18aa71ef7991..b944db6c0d8d 100644
--- a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
@@ -37,11 +37,9 @@
 import static org.testng.Assert.fail;
 
 
-/**
- * Parser-layer tests for the new Pinot DDL grammar (CREATE TABLE / DROP TABLE / SHOW TABLES).
- * Verifies that statements parse, produce the expected AST shape, and that {@link CalciteSqlParser}
- * classifies them as {@link PinotSqlType#DDL}.
- */
+/// Parser-layer tests for the new Pinot DDL grammar (CREATE TABLE / DROP TABLE / SHOW TABLES).
+/// Verifies that statements parse, produce the expected AST shape, and that [CalciteSqlParser]
+/// classifies them as [PinotSqlType#DDL].
 public class PinotDdlParserTest {
 
   // -------------------------------------------------------------------------------------------
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 11d139595884..98ad3a12204e 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -31,8 +31,10 @@
 import java.math.BigDecimal;
 import java.util.ArrayList;
 import java.util.Collections;
+import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
+import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
 import javax.annotation.Nullable;
@@ -48,6 +50,7 @@
 import javax.ws.rs.core.MediaType;
 import javax.ws.rs.core.Response;
 import org.apache.commons.lang3.StringUtils;
+import org.apache.helix.task.TaskState;
 import org.apache.pinot.common.exception.SchemaAlreadyExistsException;
 import org.apache.pinot.common.metadata.ZKMetadataProvider;
 import org.apache.pinot.common.utils.DatabaseUtils;
@@ -74,6 +77,7 @@
 import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.LogicalTableConfig;
 import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.exception.DatabaseConflictException;
 import org.apache.pinot.spi.utils.CommonConstants;
 import org.apache.pinot.spi.utils.JsonUtils;
 import org.apache.pinot.spi.utils.builder.TableNameBuilder;
@@ -93,24 +97,20 @@
 import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY;
 
 
-/**
- * Controller endpoint for executing Pinot SQL DDL statements. Currently supports CREATE TABLE,
- * DROP TABLE, SHOW TABLES, and SHOW CREATE TABLE.
- *
- * 
    Pipeline:
- * 
      
- *   
    1. {@link DdlCompiler} parses + compiles the SQL into a {@link CompiledDdl}.
      2. 
- *   
    2. Database/table names are translated through {@link DatabaseUtils#translateTableName}
- *       so the {@code Database} HTTP header is honoured uniformly.
      3. 
- *   
    3. Authorization is invoked based on the operation type.
      4. 
- *   
    4. Execution either persists via {@link PinotHelixResourceManager} or, when {@code dryRun}
- *       is true, returns the compiled artifacts without mutating cluster state.
      5. 
- * 
    
- *
- * 
    The endpoint is intentionally a single POST that dispatches by operation. This keeps the
- * client surface area small and matches the canonical {@code POST /sql/ddl} contract from the
- * design.
- */
+/// Controller endpoint for executing Pinot SQL DDL statements. Currently supports CREATE TABLE,
+/// DROP TABLE, SHOW TABLES, and SHOW CREATE TABLE.
+///
+/// Pipeline:
+/// 1. [DdlCompiler] parses + compiles the SQL into a [CompiledDdl].
+/// 1. Database/table names are translated through [DatabaseUtils#translateTableName]
+/// so the `Database` HTTP header is honoured uniformly.
+/// 1. Authorization is invoked based on the operation type.
+/// 1. Execution either persists via [PinotHelixResourceManager] or, when `dryRun`
+/// is true, returns the compiled artifacts without mutating cluster state.
+///
+/// The endpoint is intentionally a single POST that dispatches by operation. This keeps the
+/// client surface area small and matches the canonical `POST /sql/ddl` contract from the
+/// design.
 @Api(tags = "SQL DDL", authorizations = {@Authorization(value = SWAGGER_AUTHORIZATION_KEY),
     @Authorization(value = DATABASE)})
 @SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = {
@@ -123,12 +123,10 @@
 @Path("/")
 public class PinotDdlRestletResource {
   private static final Logger LOGGER = LoggerFactory.getLogger(PinotDdlRestletResource.class);
-  /**
-   * Maximum accepted SQL input length, measured in {@link String#length() Java characters}
-   * (UTF-16 code units), to prevent unbounded parser memory allocation. Up to ~4× this value
-   * in UTF-8 wire bytes can be accepted by Jackson before the length check rejects; operators
-   * sizing reverse-proxy body limits should plan accordingly.
-   */
+  /// Maximum accepted SQL input length, measured in [Java characters][String#length()]
+  /// (UTF-16 code units), to prevent unbounded parser memory allocation. Up to ~4× this value
+  /// in UTF-8 wire bytes can be accepted by Jackson before the length check rejects; operators
+  /// sizing reverse-proxy body limits should plan accordingly.
   private static final int MAX_DDL_SQL_CHARS = 256 * 1024;
 
   @Inject
@@ -222,7 +220,7 @@ private Response executeCreate(CompiledCreateTable create, String database,
       boolean dryRun, HttpHeaders headers, Request httpRequest) {
     // The compiled TableConfig.tableName carries the SQL `db.tbl` qualifier when one was given;
     // translateTableName then reconciles it against the Database header (and rejects conflicts).
-    String tableNameWithType = DatabaseUtils.translateTableName(
+    String tableNameWithType = translateTableNameForDdl(
         TableNameBuilder.forType(create.getTableConfig().getTableType())
             .tableNameWithType(create.getTableConfig().getTableName()), headers);
     create.getTableConfig().setTableName(tableNameWithType);
@@ -234,7 +232,7 @@ private Response executeCreate(CompiledCreateTable create, String database,
     String dottedSchemaName = create.getDatabaseName() == null
         ? compiledSchemaName
         : create.getDatabaseName() + "." + compiledSchemaName;
-    String schemaName = DatabaseUtils.translateTableName(dottedSchemaName, headers);
+    String schemaName = translateTableNameForDdl(dottedSchemaName, headers);
     create.getSchema().setSchemaName(schemaName);
 
     // Authorize against the FULLY-QUALIFIED, post-translation table name. Checking the bare
@@ -278,7 +276,8 @@ private Response executeCreate(CompiledCreateTable create, String database,
     boolean schemaPreexisted = storedSchema != null;
 
     if (schemaPreexisted) {
-      // Compare only the column-shape attributes that the DDL column list actually controls.
+      // Compare only the column-shape attributes that the DDL column list actually controls, plus
+      // schema metadata that was explicitly supplied in this DDL statement (currently PRIMARY KEY).
       // Comparing full JSON would include schema-level metadata (primary keys, null-handling,
       // tags, description) that the DDL does not express when a column list is given for the
       // second hybrid variant — e.g. a DDL without PRIMARY KEY would spuriously conflict with
@@ -303,6 +302,7 @@ private Response executeCreate(CompiledCreateTable create, String database,
     // DDL's column-list-only projection — otherwise upsert/dedup tables would falsely fail PK
     // validation when the DDL omits PRIMARY KEY in the second variant.
     Schema schemaForValidation = schemaPreexisted ? storedSchema : create.getSchema();
+    response.setSchema(toJson(schemaForValidation));
     // Apply tuner configs before validation, mirroring POST /tables. Tuners may rewrite the
     // table config (e.g. fill in defaulted index configs) and the validators must run against
     // the post-tuner shape, otherwise a tuner-introduced setting bypasses validation.
@@ -313,6 +313,7 @@ private Response executeCreate(CompiledCreateTable create, String database,
     // would mis-predict what a real CREATE would write).
     response.setTableConfig(toJson(create.getTableConfig()));
     validateTableConfig(schemaForValidation, create.getTableConfig());
+    PinotTableRestletResource.tableTasksValidation(create.getTableConfig(), _pinotHelixTaskResourceManager);
 
     if (dryRun) {
       response.setMessage("Dry run: validated CREATE TABLE without persisting.");
@@ -336,10 +337,18 @@ private Response executeCreate(CompiledCreateTable create, String database,
       // (legitimate hybrid-pair pattern), and a hasTable re-check followed by deleteSchema is
       // racy in the same way the generic-failure branch below is. Stale schemas can be removed
       // via DELETE /schemas/{name} if needed.
+      if (create.isIfNotExists() && _pinotHelixResourceManager.hasTable(tableNameWithType)) {
+        response.setMessage("Table " + tableNameWithType
+            + " already exists; CREATE IF NOT EXISTS is a no-op.");
+        return Response.ok(response).build();
+      }
       throw new ControllerApplicationException(LOGGER, e.getMessage(), Response.Status.CONFLICT, e);
     } catch (SchemaAlreadyExistsException e) {
-      // The override=false addSchema call lost a race with another schema writer. Surface 409.
-      throw new ControllerApplicationException(LOGGER, e.getMessage(), Response.Status.CONFLICT, e);
+      // The override=false addSchema call lost a race with another schema writer. If the table was
+      // concurrently created and the statement is idempotent, honour IF NOT EXISTS. Otherwise,
+      // verify the raced schema is compatible and continue with addTable(), matching the normal
+      // pre-existing-schema hybrid path.
+      return retryCreateAfterSchemaRace(create, response, schemaName, tableNameWithType, e);
     } catch (Exception e) {
       // Intentionally do NOT roll back the schema on a generic addTable() failure. The two
       // hasOfflineTable/hasRealtimeTable reads required to decide "is this schema orphaned?"
@@ -361,16 +370,54 @@ private Response executeCreate(CompiledCreateTable create, String database,
     }
   }
 
-  /**
-   * Compares two schemas by the column-shape attributes that a DDL column list actually controls
-   * (column name, data type, field type, single/multi-value, NOT NULL, default null value, and —
-   * for DATETIME columns — format and granularity) and returns a human-readable description of
-   * the first mismatch, or {@code null} if the shapes are equivalent. Schema-level metadata that
-   * a DDL column list does not express ({@code primaryKeyColumns}, {@code tags},
-   * {@code enableColumnBasedNullHandling}, {@code description}) is intentionally ignored so the
-   * second hybrid variant can be created via DDL without restating metadata set by the first
-   * variant.
-   */
+  private Response retryCreateAfterSchemaRace(CompiledCreateTable create, DdlExecutionResponse response,
+      String schemaName, String tableNameWithType, SchemaAlreadyExistsException schemaFailure) {
+    if (create.isIfNotExists() && _pinotHelixResourceManager.hasTable(tableNameWithType)) {
+      response.setMessage("Table " + tableNameWithType + " already exists; CREATE IF NOT EXISTS is a no-op.");
+      return Response.ok(response).build();
+    }
+    Schema racedSchema = _pinotHelixResourceManager.getSchema(schemaName);
+    if (racedSchema == null) {
+      throw new ControllerApplicationException(LOGGER, schemaFailure.getMessage(), Response.Status.CONFLICT,
+          schemaFailure);
+    }
+    String mismatch = describeColumnShapeMismatch(racedSchema, create.getSchema());
+    if (mismatch != null) {
+      throw new ControllerApplicationException(LOGGER,
+          "Schema '" + schemaName + "' was concurrently created and does not match the column list in the DDL: "
+              + mismatch,
+          Response.Status.CONFLICT, schemaFailure);
+    }
+    response.setSchema(toJson(racedSchema));
+    validateTableConfig(racedSchema, create.getTableConfig());
+    PinotTableRestletResource.tableTasksValidation(create.getTableConfig(), _pinotHelixTaskResourceManager);
+    try {
+      _pinotHelixResourceManager.addTable(create.getTableConfig());
+      response.setMessage("Successfully created table " + tableNameWithType);
+      LOGGER.info("DDL created table {} after concurrent schema create", tableNameWithType);
+      return Response.status(Response.Status.CREATED).entity(response).build();
+    } catch (TableAlreadyExistsException e) {
+      if (create.isIfNotExists() && _pinotHelixResourceManager.hasTable(tableNameWithType)) {
+        response.setMessage("Table " + tableNameWithType
+            + " already exists; CREATE IF NOT EXISTS is a no-op.");
+        return Response.ok(response).build();
+      }
+      throw new ControllerApplicationException(LOGGER, e.getMessage(), Response.Status.CONFLICT, e);
+    } catch (Exception e) {
+      throw new ControllerApplicationException(LOGGER,
+          "Failed to create table " + tableNameWithType + " after concurrent schema create: " + e.getMessage(),
+          Response.Status.INTERNAL_SERVER_ERROR, e);
+    }
+  }
+
+  /// Compares two schemas by the column-shape attributes that a DDL column list actually controls
+  /// (column name, data type, field type, single/multi-value, NOT NULL, default null value, and —
+  /// for DATETIME columns — format and granularity) and schema metadata explicitly supplied by
+  /// DDL (`PRIMARY KEY`) and returns a human-readable description of the first mismatch, or `null`
+  /// if the shapes are equivalent. Schema-level metadata that a DDL column list does not express
+  /// (`tags`, `enableColumnBasedNullHandling`, `description`, and omitted primary keys) is
+  /// intentionally ignored so the second hybrid variant can be created via DDL without restating
+  /// metadata set by the first variant.
   // Package-private for unit testing.
   @Nullable
   static String describeColumnShapeMismatch(Schema stored, Schema compiled) {
@@ -436,14 +483,18 @@ static String describeColumnShapeMismatch(Schema stored, Schema compiled) {
         }
       }
     }
+    List compiledPrimaryKeys = compiled.getPrimaryKeyColumns();
+    if (compiledPrimaryKeys != null && !compiledPrimaryKeys.isEmpty()
+        && !Objects.equals(stored.getPrimaryKeyColumns(), compiledPrimaryKeys)) {
+      return "PRIMARY KEY columns differ (stored=" + stored.getPrimaryKeyColumns()
+          + ", DDL=" + compiledPrimaryKeys + ")";
+    }
     return null;
   }
 
-  /**
-   * Compares two default-null-values for content equality, accounting for type-specific
-   * gotchas: BYTES requires Arrays.equals (each getter allocates a fresh byte[]); BIG_DECIMAL
-   * requires compareTo so different scales of the same numeric value compare equal.
-   */
+  /// Compares two default-null-values for content equality, accounting for type-specific
+  /// gotchas: BYTES requires Arrays.equals (each getter allocates a fresh byte[]); BIG_DECIMAL
+  /// requires compareTo so different scales of the same numeric value compare equal.
   private static boolean defaultValuesEqual(FieldSpec.DataType dataType,
       @Nullable Object storedDefault, @Nullable Object compiledDefault) {
     if (storedDefault == null || compiledDefault == null) {
@@ -456,13 +507,11 @@ private static boolean defaultValuesEqual(FieldSpec.DataType dataType,
     return dataType.equals(storedDefault, compiledDefault);
   }
 
-  /**
-   * Runs the same schema/table validation stack that {@code POST /tables} and
-   * {@code /tableConfigs} apply before any ZK write, so DDL-created configs are subject to the
-   * same rules as JSON-API-created configs. Delegates to {@link TableConfigValidationUtils} so
-   * the two endpoints share a single validation pipeline (min replicas, storage quota, hybrid
-   * pair compatibility, instance assignment, tenant tags, task configs, registry-level checks).
-   */
+  /// Runs the same schema/table validation stack that `POST /tables` and
+  /// `/tableConfigs` apply before any ZK write, so DDL-created configs are subject to the
+  /// same rules as JSON-API-created configs. Delegates to [TableConfigValidationUtils] so
+  /// the two endpoints share a single validation pipeline (min replicas, storage quota, hybrid
+  /// pair compatibility, instance assignment, tenant tags, task configs, registry-level checks).
   private void validateTableConfig(Schema schema, TableConfig tableConfig) {
     try {
       TableConfigValidationUtils.validateTableConfig(tableConfig, schema, null,
@@ -497,7 +546,7 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
     String dottedRaw = drop.getDatabaseName() == null
         ? drop.getRawTableName()
         : drop.getDatabaseName() + "." + drop.getRawTableName();
-    String fullyQualifiedRaw = DatabaseUtils.translateTableName(dottedRaw, headers);
+    String fullyQualifiedRaw = translateTableNameForDdl(dottedRaw, headers);
 
     // Compute the candidate typed names BEFORE existence filtering so we can authorize against
     // the user's intent (not just whatever happens to exist now). This prevents an unauthorized
@@ -523,7 +572,8 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
 
     List targets = new ArrayList<>(2);
     for (String candidate : candidates) {
-      if (_pinotHelixResourceManager.hasTable(candidate)) {
+      if (_pinotHelixResourceManager.hasTable(candidate)
+          || _pinotHelixResourceManager.getTableConfig(candidate) != null) {
         targets.add(candidate);
       }
     }
@@ -552,52 +602,21 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
         .setTableType(drop.getTableType() == null ? null : drop.getTableType().toString())
         .setDeletedTables(targets);
 
+    // Reject drop if any target is referenced by a logical table, matching the safeguard in
+    // the existing /tables and /tableConfigs DELETE endpoints.
+    assertNoLogicalTableReferences(targets);
+    assertNoActiveTasksBeforeDrop(targets);
+
     if (dryRun) {
       response.setMessage("Dry run: " + targets.size() + " table(s) would be dropped.");
       return response;
     }
 
-    // Reject drop if any target is referenced by a logical table, matching the safeguard in
-    // the existing /tables and /tableConfigs DELETE endpoints.
-    List allLogicalTableConfigs =
-        ZKMetadataProvider.getAllLogicalTableConfigs(_pinotHelixResourceManager.getPropertyStore());
-    for (String target : targets) {
-      for (LogicalTableConfig logicalTableConfig : allLogicalTableConfigs) {
-        if (LogicalTableConfigUtils.checkPhysicalTableRefExists(logicalTableConfig, target)) {
-          throw new ControllerApplicationException(LOGGER,
-              "Cannot drop table '" + target + "': it is referenced by logical table '"
-                  + logicalTableConfig.getTableName() + "'.",
-              Response.Status.CONFLICT);
-        }
-      }
-    }
-
-    // Drop each target individually and track outcomes. A failure on one variant should not
-    // prevent the response from reporting what was already deleted — partial deletes on a hybrid
-    // table are expensive to recover from, so we surface per-target status instead of surfacing
-    // only the first failure and hiding the rest.
     List dropped = new ArrayList<>();
-    List failed = new ArrayList<>();
-    Exception firstFailure = null;
-    // Track the integer status code rather than the Response.Status enum: the enum
-    // covers only the well-known IANA codes, and Response.Status.fromStatusCode() returns
-    // null for non-standard codes (422, 423, 451, etc.) — a null would silently fall back
-    // to 500 and hide the original 4xx information from the caller.
-    int firstFailureStatusCode = Response.Status.INTERNAL_SERVER_ERROR.getStatusCode();
-    // Targets whose tableTasksCleanup succeeded (removing scheduled task triggers) but whose
-    // deleteTable subsequently failed. These tables remain in the cluster but will no longer
-    // run their scheduled tasks until the operator either retries the DROP successfully or
-    // restores the SCHEDULE_KEY entries on the surviving table config.
-    List taskSchedulesCleared = new ArrayList<>();
     for (String target : targets) {
       boolean tasksCleaned = false;
       try {
-        // Remove task schedules before deletion so tasks are not triggered during the drop.
-        // tableTasksCleanup may throw ControllerApplicationException (e.g. BAD_REQUEST when
-        // active tasks are still running). That status code carries actionable user-level
-        // information and must be preserved instead of being collapsed to 500 below.
-        PinotTableRestletResource.tableTasksCleanup(target, false,
-            _pinotHelixResourceManager, _pinotHelixTaskResourceManager);
+        cleanupTableTasksBeforeDrop(target);
         tasksCleaned = true;
         // deleteTable(rawName, type, retention) takes the raw name and re-derives the typed
         // name internally via TableNameBuilder.forType(type).tableNameWithType(rawName); see
@@ -608,58 +627,205 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
         dropped.add(target);
         LOGGER.info("DDL dropped table {}", target);
       } catch (ControllerApplicationException e) {
-        // The CAE constructor already logs the underlying error at the appropriate level, and
-        // the wrapping CAE thrown after the loop will log again. A third log here would be
-        // redundant noise — record a one-line breadcrumb at WARN without the throwable.
         LOGGER.warn("DROP TABLE on {} failed: {}", target, e.getMessage());
-        failed.add(target);
-        if (tasksCleaned) {
-          taskSchedulesCleared.add(target);
-        }
-        if (firstFailure == null) {
-          firstFailure = e;
-          firstFailureStatusCode = e.getResponse().getStatus();
-        }
+        throw dropFailed(target, dropped, tasksCleaned, e.getResponse().getStatus(), e);
       } catch (Exception e) {
-        // The wrapping CAE thrown after the loop will log the firstFailure with full stack;
-        // record only a one-line breadcrumb here so operators can correlate per-target context
-        // without seeing the same stack trace twice.
         LOGGER.warn("DROP TABLE on {} failed unexpectedly: {}", target, e.toString());
-        failed.add(target);
-        if (tasksCleaned) {
-          taskSchedulesCleared.add(target);
-        }
-        if (firstFailure == null) {
-          firstFailure = e;
-          firstFailureStatusCode = Response.Status.INTERNAL_SERVER_ERROR.getStatusCode();
-        }
+        throw dropFailed(target, dropped, tasksCleaned,
+            Response.Status.INTERNAL_SERVER_ERROR.getStatusCode(), e);
       }
     }
     // Intentionally do NOT delete the shared schema when the last physical variant is removed.
     // This matches the existing `/tables/{name}` DELETE contract, which also leaves the schema
     // intact. Two doors into the same state machine must have the same side effects; a caller
     // who wants to remove the schema can issue an explicit DELETE /schemas/{name} afterwards.
-    if (failed.isEmpty()) {
-      response.setMessage("Dropped " + dropped.size() + " table(s).");
-      LOGGER.info("DDL dropped tables {}", dropped);
-      return response;
+    response.setDeletedTables(dropped);
+    response.setMessage("Dropped " + dropped.size() + " table metadata target(s).");
+    LOGGER.info("DDL dropped tables {}", dropped);
+    return response;
+  }
+
+  private void assertNoLogicalTableReferences(List targets) {
+    List allLogicalTableConfigs =
+        ZKMetadataProvider.getAllLogicalTableConfigs(_pinotHelixResourceManager.getPropertyStore());
+    for (String target : targets) {
+      for (LogicalTableConfig logicalTableConfig : allLogicalTableConfigs) {
+        if (LogicalTableConfigUtils.checkPhysicalTableRefExists(logicalTableConfig, target)) {
+          throw new ControllerApplicationException(LOGGER,
+              "Cannot drop table '" + target + "': it is referenced by logical table '"
+                  + logicalTableConfig.getTableName() + "'.",
+              Response.Status.CONFLICT);
+        }
+      }
+    }
+  }
+
+  private void assertNoActiveTasksBeforeDrop(List targets) {
+    List pendingTasks = new ArrayList<>();
+    for (String target : targets) {
+      TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(target);
+      if (tableConfig == null || tableConfig.getTaskConfig() == null) {
+        continue;
+      }
+      for (String taskType : tableConfig.getTaskConfig().getTaskTypeConfigsMap().keySet()) {
+        Map taskStates;
+        try {
+          taskStates = _pinotHelixTaskResourceManager.getTaskStatesByTable(taskType, target);
+        } catch (IllegalArgumentException e) {
+          LOGGER.info(e.getMessage());
+          continue;
+        }
+        for (Map.Entry taskState : taskStates.entrySet()) {
+          String taskName = taskState.getKey();
+          if (TaskState.IN_PROGRESS.equals(taskState.getValue())
+              && _pinotHelixTaskResourceManager.getTaskCount(taskName).getRunning() > 0) {
+            pendingTasks.add(taskName);
+          }
+        }
+      }
+    }
+    if (!pendingTasks.isEmpty()) {
+      throw new ControllerApplicationException(LOGGER,
+          "DROP TABLE blocked because active running tasks exist: " + pendingTasks
+              + ". No table metadata was changed by this DDL preflight; retry once the tasks finish.",
+          Response.Status.BAD_REQUEST);
     }
-    // At least one target failed. Surface a structured error that names both what succeeded
-    // and what failed so the operator knows which variant needs manual cleanup. Preserve the
-    // first failure's status code so client-actionable failures (e.g. active tasks → 400)
-    // remain visible to the caller; only fall back to 500 for genuinely unexpected failures.
-    String causeDesc = firstFailure.getMessage() != null
-        ? firstFailure.getMessage() : firstFailure.getClass().getSimpleName();
+  }
+
+  private void cleanupTableTasksBeforeDrop(String tableWithType)
+      throws Exception {
+    TableConfig tableConfig = _pinotHelixResourceManager.getTableConfig(tableWithType);
+    if (tableConfig == null || tableConfig.getTaskConfig() == null) {
+      return;
+    }
+    Map> taskTypeConfigsMap = tableConfig.getTaskConfig().getTaskTypeConfigsMap();
+    Set taskTypes = new HashSet<>(taskTypeConfigsMap.keySet());
+    TaskCleanupScan scan = scanTasksBeforeDrop(tableWithType, taskTypes);
+    if (!scan._pendingTasks.isEmpty()) {
+      throw activeTasksBlocked(scan._pendingTasks, "No table metadata was changed by this DDL preflight");
+    }
+
+    Map removedSchedules = new HashMap<>();
+    for (String taskType : taskTypes) {
+      String schedule = taskTypeConfigsMap.get(taskType).remove(PinotTaskManager.SCHEDULE_KEY);
+      if (schedule != null) {
+        removedSchedules.put(taskType, schedule);
+      }
+    }
+    boolean schedulesPersisted = persistTaskScheduleRemoval(tableConfig, removedSchedules);
+    try {
+      scan = scanTasksBeforeDrop(tableWithType, taskTypes);
+      if (!scan._pendingTasks.isEmpty()) {
+        if (schedulesPersisted) {
+          restoreTaskSchedules(tableWithType, tableConfig, removedSchedules);
+        }
+        throw activeTasksBlocked(scan._pendingTasks,
+            "Task schedules were restored because active tasks appeared during DROP TABLE preflight");
+      }
+      for (String taskName : scan._deletableTasks) {
+        _pinotHelixTaskResourceManager.deleteTask(taskName, true);
+      }
+    } catch (ControllerApplicationException e) {
+      throw e;
+    } catch (Exception e) {
+      if (schedulesPersisted) {
+        try {
+          restoreTaskSchedules(tableWithType, tableConfig, removedSchedules);
+        } catch (ControllerApplicationException restoreFailure) {
+          e.addSuppressed(restoreFailure);
+        }
+      }
+      throw e;
+    }
+  }
+
+  private boolean persistTaskScheduleRemoval(TableConfig tableConfig, Map removedSchedules) {
+    if (removedSchedules.isEmpty()) {
+      return false;
+    }
+    try {
+      _pinotHelixResourceManager.updateTableConfig(tableConfig);
+      return true;
+    } catch (Exception e) {
+      restoreTaskSchedulesInMemory(tableConfig, removedSchedules);
+      LOGGER.warn("Unable to remove task schedules before DROP TABLE on {}. "
+              + "Proceeding with table deletion because no active tasks were found. Reason: {}",
+          tableConfig.getTableName(), e.getMessage());
+      return false;
+    }
+  }
+
+  private void restoreTaskSchedules(String tableWithType, TableConfig tableConfig,
+      Map removedSchedules) {
+    restoreTaskSchedulesInMemory(tableConfig, removedSchedules);
+    try {
+      _pinotHelixResourceManager.updateTableConfig(tableConfig);
+    } catch (Exception e) {
+      throw new ControllerApplicationException(LOGGER,
+          "DROP TABLE detected active tasks after clearing task schedules for " + tableWithType
+              + " and failed to restore those schedules: " + e.getMessage(),
+          Response.Status.INTERNAL_SERVER_ERROR, e);
+    }
+  }
+
+  private static void restoreTaskSchedulesInMemory(TableConfig tableConfig,
+      Map removedSchedules) {
+    Map> taskTypeConfigsMap = tableConfig.getTaskConfig().getTaskTypeConfigsMap();
+    for (Map.Entry removedSchedule : removedSchedules.entrySet()) {
+      taskTypeConfigsMap.get(removedSchedule.getKey())
+          .put(PinotTaskManager.SCHEDULE_KEY, removedSchedule.getValue());
+    }
+  }
+
+  private TaskCleanupScan scanTasksBeforeDrop(String tableWithType, Set taskTypes) {
+    TaskCleanupScan scan = new TaskCleanupScan();
+    for (String taskType : taskTypes) {
+      Map taskStates;
+      try {
+        taskStates = _pinotHelixTaskResourceManager.getTaskStatesByTable(taskType, tableWithType);
+      } catch (IllegalArgumentException e) {
+        LOGGER.info(e.getMessage());
+        continue;
+      }
+      for (Map.Entry taskState : taskStates.entrySet()) {
+        String taskName = taskState.getKey();
+        if (TaskState.IN_PROGRESS.equals(taskState.getValue())
+            && _pinotHelixTaskResourceManager.getTaskCount(taskName).getRunning() > 0) {
+          scan._pendingTasks.add(taskName);
+        } else {
+          scan._deletableTasks.add(taskName);
+        }
+      }
+    }
+    return scan;
+  }
+
+  private ControllerApplicationException activeTasksBlocked(List pendingTasks, String mutationContext) {
+    return new ControllerApplicationException(LOGGER,
+        "DROP TABLE blocked because active running tasks exist: " + pendingTasks
+            + ". " + mutationContext + "; retry once the tasks finish.",
+        Response.Status.BAD_REQUEST);
+  }
+
+  private static final class TaskCleanupScan {
+    private final List _pendingTasks = new ArrayList<>();
+    private final List _deletableTasks = new ArrayList<>();
+  }
+
+  private ControllerApplicationException dropFailed(String target, List dropped,
+      boolean taskSchedulesCleared, int statusCode, Exception cause) {
+    String causeDesc = cause.getMessage() != null ? cause.getMessage() : cause.getClass().getSimpleName();
     String partialPrefix = dropped.isEmpty() ? "" : "Partial DROP TABLE: dropped " + dropped + ", ";
-    String taskScheduleHint = taskSchedulesCleared.isEmpty() ? ""
-        : ". Task schedules were already cleared for " + taskSchedulesCleared
-            + "; if those tables remain in service the operator must restore the schedule entries"
-            + " in their TableConfig taskTypeConfigsMap before scheduled tasks resume.";
     String reCreateHint = dropped.isEmpty() ? ""
         : ". The successfully-dropped variant must be re-created if the original DROP was unintended.";
-    String msg = partialPrefix + "failed to drop " + failed + ": " + causeDesc
-        + reCreateHint + taskScheduleHint;
-    throw new ControllerApplicationException(LOGGER, msg, firstFailureStatusCode, firstFailure);
+    String taskScheduleHint = taskSchedulesCleared
+        ? ". Task schedules were already cleared for " + target
+            + "; if that table remains in service the operator must restore the schedule entries"
+            + " in its TableConfig taskTypeConfigsMap before scheduled tasks resume."
+        : "";
+    return new ControllerApplicationException(LOGGER,
+        partialPrefix + "failed to drop " + target + ": " + causeDesc + reCreateHint + taskScheduleHint,
+        statusCode, cause);
   }
 
   // -------------------------------------------------------------------------------------------
@@ -675,7 +841,7 @@ private DdlExecutionResponse executeShowCreate(CompiledShowCreateTable show, Str
     String dottedRaw = show.getDatabaseName() == null
         ? show.getRawTableName()
         : show.getDatabaseName() + "." + show.getRawTableName();
-    String fullyQualifiedRaw = DatabaseUtils.translateTableName(dottedRaw, headers);
+    String fullyQualifiedRaw = translateTableNameForDdl(dottedRaw, headers);
 
     // Resolve which typed variant to render. Explicit TYPE clause wins; otherwise check both.
     // Authorize BEFORE existence checks so an unauthorized caller cannot distinguish
@@ -818,18 +984,22 @@ private DdlExecutionResponse executeShow(String database, HttpHeaders headers, R
   // Helpers
   // -------------------------------------------------------------------------------------------
 
-  /**
-   * Returns the database name to use for this request. Precedence: explicit {@code db.name} in
-   * the SQL statement → {@code Database} HTTP header → {@code null} (default database).
-   */
+  /// Returns the database name to use for this request. Precedence: explicit `db.name` in
+  /// the SQL statement → `Database` HTTP header → `null` (default database).
   private static String resolveDatabase(String fromSql, HttpHeaders headers) {
+    String fromHeader = headers == null ? null : headers.getHeaderString(DATABASE);
     if (fromSql != null) {
+      if (StringUtils.isNotEmpty(fromHeader) && !fromSql.equals(fromHeader)
+          && !(CommonConstants.DEFAULT_DATABASE.equalsIgnoreCase(fromSql)
+          && CommonConstants.DEFAULT_DATABASE.equalsIgnoreCase(fromHeader))) {
+        throw new ControllerApplicationException(LOGGER,
+            "Database name '" + fromSql + "' from SQL does not match database name '"
+                + fromHeader + "' from header",
+            Response.Status.BAD_REQUEST);
+      }
       return fromSql;
     }
-    if (headers == null) {
-      return null;
-    }
-    return headers.getHeaderString(DATABASE);
+    return fromHeader;
   }
 
   private static JsonNode toJson(Object obj) {
@@ -845,4 +1015,14 @@ private static JsonNode toJson(Object obj) {
   private static ControllerApplicationException badRequest(String message) {
     return new ControllerApplicationException(LOGGER, message, Response.Status.BAD_REQUEST);
   }
+
+  private static String translateTableNameForDdl(String tableName, HttpHeaders headers) {
+    try {
+      return DatabaseUtils.translateTableName(tableName, headers);
+    } catch (DatabaseConflictException | IllegalArgumentException e) {
+      throw new ControllerApplicationException(LOGGER,
+          "Invalid database/table reference '" + tableName + "': " + e.getMessage(),
+          Response.Status.BAD_REQUEST, e);
+    }
+  }
 }
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java
index 19c6476ce876..b407da3fb546 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java
@@ -393,6 +393,11 @@ private StreamingOutput executeSqlQuery(@Context HttpHeaders httpHeaders, String
       Map optionsFromString = RequestUtils.getOptionsFromString(queryOptions);
       sqlNodeAndOptions.setExtraOptions(optionsFromString);
     }
+    PinotSqlType sqlType = sqlNodeAndOptions.getSqlType();
+    if (sqlType == PinotSqlType.DDL) {
+      throw QueryErrorCode.QUERY_VALIDATION.asException(
+          "DDL statements are not supported on /sql; use POST /sql/ddl instead.");
+    }
 
     // Determine which engine to used based on query options.
     boolean isMse = Boolean.parseBoolean(options.get(QueryOptionKey.USE_MULTISTAGE_ENGINE));
@@ -403,7 +408,6 @@ private StreamingOutput executeSqlQuery(@Context HttpHeaders httpHeaders, String
       throw QueryErrorCode.INTERNAL.asException("V2 Multi-Stage query engine not enabled.");
     }
 
-    PinotSqlType sqlType = sqlNodeAndOptions.getSqlType();
     switch (sqlType) {
       case DQL:
         return isMse
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionRequest.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionRequest.java
index e42c17631287..1bd908e6c4ca 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionRequest.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionRequest.java
@@ -22,7 +22,7 @@
 import com.fasterxml.jackson.annotation.JsonProperty;
 
 
-/** Request body for {@code POST /sql/ddl}. */
+/// Request body for `POST /sql/ddl`.
 public class DdlExecutionRequest {
   private final String _sql;
 
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java
index 5fb97a8aac1b..c794fbd8966e 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java
@@ -25,19 +25,15 @@
 import org.apache.pinot.sql.ddl.compile.DdlOperation;
 
 
-/**
- * Response body for {@code POST /sql/ddl}.
- *
- * 
    The shape of the response varies by operation. {@link JsonInclude.Include#NON_NULL} keeps
- * the wire payload focused on the fields that actually apply to the operation that ran.
- *
- * 
      
- *   
    • CREATE_TABLE: {@code tableName, tableType, schema, tableConfig, ifNotExists, warnings}
      • 
- *   
    • DROP_TABLE: {@code tableName, tableType, deletedTables, ifExists}
      • 
- *   
    • SHOW_TABLES: {@code tableNames}
      • 
- *   
    • SHOW_CREATE_TABLE: {@code tableName, tableType, ddl}
      • 
- * 
    
- */
+/// Response body for `POST /sql/ddl`.
+///
+/// The shape of the response varies by operation. [JsonInclude.Include#NON_NULL] keeps
+/// the wire payload focused on the fields that actually apply to the operation that ran.
+///
+/// - CREATE_TABLE: `tableName, tableType, schema, tableConfig, ifNotExists, warnings`
+/// - DROP_TABLE: `tableName, tableType, deletedTables, ifExists`
+/// - SHOW_TABLES: `tableNames`
+/// - SHOW_CREATE_TABLE: `tableName, tableType, ddl`
 @JsonInclude(JsonInclude.Include.NON_NULL)
 public class DdlExecutionResponse {
   private DdlOperation _operation;
@@ -178,7 +174,7 @@ public String getDdl() {
     return _ddl;
   }
 
-  /** Canonical CREATE TABLE statement returned by {@code SHOW CREATE TABLE}. */
+  /// Canonical CREATE TABLE statement returned by `SHOW CREATE TABLE`.
   public DdlExecutionResponse setDdl(@Nullable String ddl) {
     _ddl = ddl;
     return this;
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
index 8a6fd4e2f5b0..7a8b7e449003 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
@@ -38,12 +38,10 @@
 import static org.testng.Assert.fail;
 
 
-/**
- * End-to-end integration tests for {@code POST /sql/ddl}.
- *
- * 
    Each test uses a unique table name prefix so failures do not cascade through the shared
- * controller test instance.
- */
+/// End-to-end integration tests for `POST /sql/ddl`.
+///
+/// Each test uses a unique table name prefix so failures do not cascade through the shared
+/// controller test instance.
 public class PinotDdlRestletResourceTest extends ControllerTest {
 
   private static final String TBL_BASIC = "ddlBasicOffline";
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
index 6c77eb1a815a..c7c4a4ca5794 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResourceUnitTest.java
@@ -21,29 +21,160 @@
 import java.math.BigDecimal;
 import java.util.Arrays;
 import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import javax.ws.rs.core.HttpHeaders;
+import javax.ws.rs.core.Response;
+import org.apache.helix.task.TaskState;
+import org.apache.pinot.common.metadata.ZKMetadataProvider;
+import org.apache.pinot.controller.api.access.AccessControl;
+import org.apache.pinot.controller.api.access.AccessControlFactory;
+import org.apache.pinot.controller.api.exception.ControllerApplicationException;
+import org.apache.pinot.controller.api.resources.ddl.DdlExecutionRequest;
+import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
+import org.apache.pinot.controller.helix.core.minion.PinotHelixTaskResourceManager;
+import org.apache.pinot.controller.helix.core.minion.PinotTaskManager;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableTaskConfig;
+import org.apache.pinot.spi.config.table.TableType;
 import org.apache.pinot.spi.data.DateTimeFieldSpec;
 import org.apache.pinot.spi.data.DimensionFieldSpec;
 import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.data.LogicalTableConfig;
 import org.apache.pinot.spi.data.MetricFieldSpec;
+import org.apache.pinot.spi.data.PhysicalTableConfig;
 import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.glassfish.grizzly.http.server.Request;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
 import org.testng.annotations.Test;
 
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
 import static org.testng.Assert.assertNotNull;
 import static org.testng.Assert.assertNull;
 import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.expectThrows;
 
 
-/**
- * Unit tests for {@link PinotDdlRestletResource#describeColumnShapeMismatch}. This is the
- * hybrid-table CREATE gate that decides whether a DDL-compiled schema is compatible with the
- * schema already stored in ZK for a hybrid pair's sibling variant. The comparator must accept
- * differences in schema-level metadata a DDL column list cannot express (primary keys, tags,
- * null-handling) and reject differences in per-column attributes that the DDL does control.
- */
+/// Unit tests for [PinotDdlRestletResource#describeColumnShapeMismatch]. This is the
+/// hybrid-table CREATE gate that decides whether a DDL-compiled schema is compatible with the
+/// schema already stored in ZK for a hybrid pair's sibling variant. The comparator must accept
+/// differences in schema-level metadata a DDL column list cannot express (primary keys, tags,
+/// null-handling) and reject differences in per-column attributes that the DDL does control.
 public class PinotDdlRestletResourceUnitTest {
 
-  /** Compiled DDL with matching columns but no primary keys must accept a stored schema that has them. */
+  /// A dry-run DROP must fail on the same non-mutating logical-table reference guard as a live
+  /// DROP. Otherwise dry-run can promise a deletion that the real request will later reject.
+  @Test
+  public void dropDryRunRejectsLogicalTableReferences() {
+    PinotDdlRestletResource resource = new PinotDdlRestletResource();
+    resource._pinotHelixResourceManager = mock(PinotHelixResourceManager.class);
+    resource._accessControlFactory = mock(AccessControlFactory.class);
+    when(resource._accessControlFactory.create()).thenReturn(new AccessControl() {
+    });
+    when(resource._pinotHelixResourceManager.hasTable("events_OFFLINE")).thenReturn(true);
+
+    LogicalTableConfig logicalTableConfig = new LogicalTableConfig();
+    logicalTableConfig.setTableName("logical_events");
+    logicalTableConfig.setPhysicalTableConfigMap(
+        Collections.singletonMap("events_OFFLINE", new PhysicalTableConfig()));
+
+    Request request = mock(Request.class);
+    when(request.getRequestURL()).thenReturn(new StringBuilder("http://localhost/sql/ddl"));
+
+    try (MockedStatic metadataProvider = Mockito.mockStatic(ZKMetadataProvider.class)) {
+      metadataProvider.when(() -> ZKMetadataProvider.getAllLogicalTableConfigs(Mockito.any()))
+          .thenReturn(Collections.singletonList(logicalTableConfig));
+
+      ControllerApplicationException e = expectThrows(ControllerApplicationException.class,
+          () -> resource.executeDdl(new DdlExecutionRequest("DROP TABLE events TYPE OFFLINE"), true,
+              mock(HttpHeaders.class), request));
+      assertEquals(e.getResponse().getStatus(), Response.Status.CONFLICT.getStatusCode());
+      assertTrue(e.getMessage().contains("logical_events"), e.getMessage());
+    }
+  }
+
+  /// If an active task appears after DROP's initial preflight, DDL must fail without first
+  /// clearing the table task schedule. This guards the race that the shared JSON delete helper
+  /// cannot prevent because it removes schedules before checking for active tasks.
+  @Test
+  public void dropDoesNotClearTaskSchedulesWhenTaskAppearsAfterPreflight()
+      throws Exception {
+    PinotDdlRestletResource resource = new PinotDdlRestletResource();
+    resource._pinotHelixResourceManager = mock(PinotHelixResourceManager.class);
+    resource._pinotHelixTaskResourceManager = mock(PinotHelixTaskResourceManager.class);
+    resource._accessControlFactory = mock(AccessControlFactory.class);
+    when(resource._accessControlFactory.create()).thenReturn(new AccessControl() {
+    });
+    when(resource._pinotHelixResourceManager.hasTable("events_OFFLINE")).thenReturn(true);
+
+    Map> taskConfigs = new HashMap<>();
+    Map refreshTaskConfig = new HashMap<>();
+    refreshTaskConfig.put(PinotTaskManager.SCHEDULE_KEY, "0 0 * * * ?");
+    taskConfigs.put("SegmentRefreshTask", refreshTaskConfig);
+    TableConfig tableConfig = new TableConfigBuilder(TableType.OFFLINE)
+        .setTableName("events")
+        .setTaskConfig(new TableTaskConfig(taskConfigs))
+        .build();
+    when(resource._pinotHelixResourceManager.getTableConfig("events_OFFLINE")).thenReturn(tableConfig);
+
+    PinotHelixTaskResourceManager.TaskCount taskCount = mock(PinotHelixTaskResourceManager.TaskCount.class);
+    when(taskCount.getRunning()).thenReturn(1);
+    when(resource._pinotHelixTaskResourceManager.getTaskCount("task_0")).thenReturn(taskCount);
+    when(resource._pinotHelixTaskResourceManager.getTaskStatesByTable("SegmentRefreshTask", "events_OFFLINE"))
+        .thenReturn(Collections.emptyMap())
+        .thenReturn(Collections.singletonMap("task_0", TaskState.IN_PROGRESS));
+
+    Request request = mock(Request.class);
+    when(request.getRequestURL()).thenReturn(new StringBuilder("http://localhost/sql/ddl"));
+
+    try (MockedStatic metadataProvider = Mockito.mockStatic(ZKMetadataProvider.class)) {
+      metadataProvider.when(() -> ZKMetadataProvider.getAllLogicalTableConfigs(Mockito.any()))
+          .thenReturn(Collections.emptyList());
+
+      ControllerApplicationException e = expectThrows(ControllerApplicationException.class,
+          () -> resource.executeDdl(new DdlExecutionRequest("DROP TABLE events TYPE OFFLINE"), false,
+              mock(HttpHeaders.class), request));
+      assertEquals(e.getResponse().getStatus(), Response.Status.BAD_REQUEST.getStatusCode());
+      assertTrue(e.getMessage().contains("active running tasks"), e.getMessage());
+    }
+
+    verify(resource._pinotHelixResourceManager, never()).updateTableConfig(tableConfig);
+    verify(resource._pinotHelixResourceManager, never()).deleteTable("events", TableType.OFFLINE, null);
+    assertEquals(tableConfig.getTaskConfig().getTaskTypeConfigsMap()
+        .get("SegmentRefreshTask").get(PinotTaskManager.SCHEDULE_KEY), "0 0 * * * ?");
+  }
+
+  /// SQL database qualifiers and the Database header must agree; conflicts are caller input
+  /// errors, not controller failures.
+  @Test
+  public void conflictingSqlAndHeaderDatabaseReturnsBadRequest() {
+    HttpHeaders headers = mock(HttpHeaders.class);
+    when(headers.getHeaderString(CommonConstants.DATABASE)).thenReturn("db2");
+
+    assertDatabaseConflictReturnsBadRequest(
+        "CREATE TABLE db1.events (id INT) TABLE_TYPE = OFFLINE", headers);
+    assertDatabaseConflictReturnsBadRequest("DROP TABLE db1.events TYPE OFFLINE", headers);
+    assertDatabaseConflictReturnsBadRequest("SHOW CREATE TABLE db1.events TYPE OFFLINE", headers);
+    assertDatabaseConflictReturnsBadRequest("SHOW TABLES FROM db1", headers);
+  }
+
+  private static void assertDatabaseConflictReturnsBadRequest(String sql, HttpHeaders headers) {
+    ControllerApplicationException e = expectThrows(ControllerApplicationException.class,
+        () -> new PinotDdlRestletResource().executeDdl(new DdlExecutionRequest(sql), true, headers,
+            mock(Request.class)));
+    assertEquals(e.getResponse().getStatus(), Response.Status.BAD_REQUEST.getStatusCode());
+    assertTrue(e.getMessage().contains("does not match"), e.getMessage());
+  }
+
+  /// Compiled DDL with matching columns but no primary keys must accept a stored schema that has them.
   @Test
   public void acceptsMatchingColumnsWhenStoredHasExtraPrimaryKeyMetadata() {
     Schema stored = new Schema();
@@ -60,7 +191,45 @@ public void acceptsMatchingColumnsWhenStoredHasExtraPrimaryKeyMetadata() {
         "schema-level metadata the DDL column list cannot express must not drive a mismatch");
   }
 
-  /** A missing or extra column must be rejected with a named column set in the message. */
+  /// Compiled DDL with an explicit matching PRIMARY KEY must accept a stored schema with the same
+  /// primary-key contract.
+  @Test
+  public void acceptsExplicitPrimaryKeyWhenStoredSchemaMatches() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    stored.addField(new DimensionFieldSpec("id", DataType.INT, true));
+    stored.setPrimaryKeyColumns(Collections.singletonList("id"));
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    compiled.addField(new DimensionFieldSpec("id", DataType.INT, true));
+    compiled.setPrimaryKeyColumns(Collections.singletonList("id"));
+
+    assertNull(PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled));
+  }
+
+  /// If the DDL explicitly supplies PRIMARY KEY, reusing a stored schema with a different key would
+  /// silently persist the old key while the response advertises the new one. Reject that mismatch.
+  @Test
+  public void rejectsExplicitPrimaryKeyMismatch() {
+    Schema stored = new Schema();
+    stored.setSchemaName("t");
+    stored.addField(new DimensionFieldSpec("id", DataType.INT, true));
+    stored.addField(new DimensionFieldSpec("other", DataType.INT, true));
+    stored.setPrimaryKeyColumns(Collections.singletonList("id"));
+
+    Schema compiled = new Schema();
+    compiled.setSchemaName("t");
+    compiled.addField(new DimensionFieldSpec("id", DataType.INT, true));
+    compiled.addField(new DimensionFieldSpec("other", DataType.INT, true));
+    compiled.setPrimaryKeyColumns(Collections.singletonList("other"));
+
+    String msg = PinotDdlRestletResource.describeColumnShapeMismatch(stored, compiled);
+    assertNotNull(msg);
+    assertTrue(msg.contains("PRIMARY KEY"), msg);
+  }
+
+  /// A missing or extra column must be rejected with a named column set in the message.
   @Test
   public void rejectsColumnSetDifference() {
     Schema stored = new Schema();
@@ -78,7 +247,7 @@ public void rejectsColumnSetDifference() {
         "message should call out the offending column set difference: " + msg);
   }
 
-  /** Different data type for the same column must be rejected. */
+  /// Different data type for the same column must be rejected.
   @Test
   public void rejectsDataTypeMismatch() {
     Schema stored = new Schema();
@@ -94,7 +263,7 @@ public void rejectsDataTypeMismatch() {
     assertTrue(msg.contains("data type differs"), msg);
   }
 
-  /** DIMENSION vs METRIC for the same column must be rejected. */
+  /// DIMENSION vs METRIC for the same column must be rejected.
   @Test
   public void rejectsFieldTypeMismatch() {
     Schema stored = new Schema();
@@ -110,7 +279,7 @@ public void rejectsFieldTypeMismatch() {
     assertTrue(msg.contains("field type differs"), msg);
   }
 
-  /** Single-value vs multi-value mismatch must be rejected. */
+  /// Single-value vs multi-value mismatch must be rejected.
   @Test
   public void rejectsSingleValuedFlagMismatch() {
     Schema stored = new Schema();
@@ -126,7 +295,7 @@ public void rejectsSingleValuedFlagMismatch() {
     assertTrue(msg.contains("single-valued flag"), msg);
   }
 
-  /** NOT NULL flag mismatch must be rejected. */
+  /// NOT NULL flag mismatch must be rejected.
   @Test
   public void rejectsNotNullFlagMismatch() {
     Schema stored = new Schema();
@@ -144,11 +313,9 @@ public void rejectsNotNullFlagMismatch() {
     assertTrue(msg.contains("NOT NULL flag"), msg);
   }
 
-  /**
-   * BYTES columns produce a fresh byte[] on each getDefaultNullValue() call. Equality must be
-   * by content, not reference, otherwise every hybrid second-variant CREATE on a BYTES schema
-   * with a custom default would falsely trip the column-shape mismatch.
-   */
+  /// BYTES columns produce a fresh byte[] on each getDefaultNullValue() call. Equality must be
+  /// by content, not reference, otherwise every hybrid second-variant CREATE on a BYTES schema
+  /// with a custom default would falsely trip the column-shape mismatch.
   @Test
   public void acceptsMatchingBytesDefaultNullValue() {
     Schema stored = new Schema();
@@ -167,12 +334,10 @@ public void acceptsMatchingBytesDefaultNullValue() {
         "matching BYTES defaults must not trip a content-vs-reference equality regression");
   }
 
-  /**
-   * Regression: BigDecimal.equals is scale-sensitive (1 != 1.0); the comparator must use
-   * compareTo so the same numeric default arriving via different literal forms is treated
-   * as equivalent. Without this fix, a hybrid second-variant CREATE on a BIG_DECIMAL column
-   * whose stored default arrived as "1.0" but DDL re-states "1" would falsely 409 CONFLICT.
-   */
+  /// Regression: BigDecimal.equals is scale-sensitive (1 != 1.0); the comparator must use
+  /// compareTo so the same numeric default arriving via different literal forms is treated
+  /// as equivalent. Without this fix, a hybrid second-variant CREATE on a BIG_DECIMAL column
+  /// whose stored default arrived as "1.0" but DDL re-states "1" would falsely 409 CONFLICT.
   @Test
   public void acceptsScaleShiftedBigDecimalDefault() {
     Schema stored = new Schema();
@@ -191,7 +356,7 @@ public void acceptsScaleShiftedBigDecimalDefault() {
         "scale-shifted BIG_DECIMAL defaults must compare equal via compareTo");
   }
 
-  /** BYTES default mismatch must still be rejected with content-aware comparison. */
+  /// BYTES default mismatch must still be rejected with content-aware comparison.
   @Test
   public void rejectsBytesDefaultNullValueMismatch() {
     Schema stored = new Schema();
@@ -211,7 +376,7 @@ public void rejectsBytesDefaultNullValueMismatch() {
     assertTrue(msg.contains("default null value"), msg);
   }
 
-  /** Default-null-value mismatch must be rejected. */
+  /// Default-null-value mismatch must be rejected.
   @Test
   public void rejectsDefaultNullValueMismatch() {
     Schema stored = new Schema();
@@ -231,7 +396,7 @@ public void rejectsDefaultNullValueMismatch() {
     assertTrue(msg.contains("default null value"), msg);
   }
 
-  /** DATETIME format mismatch must be rejected. */
+  /// DATETIME format mismatch must be rejected.
   @Test
   public void rejectsDateTimeFormatMismatch() {
     Schema stored = new Schema();
@@ -249,7 +414,7 @@ public void rejectsDateTimeFormatMismatch() {
     assertTrue(msg.contains("DATETIME format"), msg);
   }
 
-  /** DATETIME granularity mismatch must be rejected. */
+  /// DATETIME granularity mismatch must be rejected.
   @Test
   public void rejectsDateTimeGranularityMismatch() {
     Schema stored = new Schema();
@@ -267,7 +432,7 @@ public void rejectsDateTimeGranularityMismatch() {
     assertTrue(msg.contains("DATETIME granularity"), msg);
   }
 
-  /** Matching multi-column, multi-type schemas must be accepted. */
+  /// Matching multi-column, multi-type schemas must be accepted.
   @Test
   public void acceptsMatchingMixedColumnSchema() {
     Schema stored = new Schema();
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceTest.java
index 676874a7771e..70f7f02bce6f 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceTest.java
@@ -79,6 +79,35 @@ public void testInvalidQuery() {
     Assert.assertFalse(response.contains("retry the query using the multi-stage query engine"));
   }
 
+  @Test
+  public void testDdlOnQueryEndpointReturnsValidationError() {
+    String response = streamingOutputToString(
+        _pinotQueryResource.handleGetSql("CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE", null, null, null)
+    );
+    Assert.assertTrue(response.contains(String.valueOf(QueryErrorCode.QUERY_VALIDATION.getId())));
+    Assert.assertTrue(response.contains("/sql/ddl"));
+  }
+
+  @Test
+  public void testDdlOnQueryEndpointWithMseOptionReturnsValidationError() {
+    String response = streamingOutputToString(
+        _pinotQueryResource.handleGetSql("CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE", null,
+            "useMultistageEngine=true", null)
+    );
+    Assert.assertTrue(response.contains(String.valueOf(QueryErrorCode.QUERY_VALIDATION.getId())));
+    Assert.assertTrue(response.contains("/sql/ddl"));
+  }
+
+  @Test
+  public void testDdlOnQueryEndpointWithSetMseOptionReturnsValidationError() {
+    String response = streamingOutputToString(
+        _pinotQueryResource.handleGetSql(
+            "SET useMultistageEngine = 'true'; CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE", null, null, null)
+    );
+    Assert.assertTrue(response.contains(String.valueOf(QueryErrorCode.QUERY_VALIDATION.getId())));
+    Assert.assertTrue(response.contains("/sql/ddl"));
+  }
+
   public static String streamingOutputToString(StreamingOutput streamingOutput) {
     try (ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream()) {
       streamingOutput.write(byteArrayOutputStream);
diff --git a/pinot-sql-ddl/DESIGN.md b/pinot-sql-ddl/DESIGN.md
new file mode 100644
index 000000000000..c03028b7d67f
--- /dev/null
+++ b/pinot-sql-ddl/DESIGN.md
@@ -0,0 +1,521 @@
+
+# Pinot SQL DDL — Design Document
+
+Status: implemented.
+Tracked PR: [apache/pinot#18241](https://github.com/apache/pinot/pull/18241).
+
+## 1. Goals
+
+1. Add `CREATE TABLE`, `DROP TABLE`, `SHOW TABLES`, `SHOW CREATE TABLE` as first-class
+   SQL DDL on the controller.
+2. Make DDL a **thin sugar layer** over the existing `Schema` + `TableConfig` model: a DDL
+   statement compiles into the exact same `(Schema, TableConfig)` pair that the JSON
+   `/schemas` and `/tables` endpoints accept, and the controller persists through the
+   same `PinotHelixResourceManager` paths and the same `TableConfigValidationUtils`
+   pipeline. There must be **one** state machine, not two.
+3. `SHOW CREATE TABLE` must be a deterministic, idempotent inverse of `CREATE TABLE`:
+   `emit → parse → compile → emit` is a fixed point for any DDL-expressible
+   `(Schema, TableConfig)` pair.
+4. The grammar and `PROPERTIES (...)` routing must be **forward-compatible** with future
+   TableConfig evolution — adding a new nested config or a new stream key must not
+   require grammar changes in the common case.
+5. The DDL endpoint must not introduce a parallel validation surface — every check the
+   JSON API runs (min replicas, storage quota, hybrid pair, instance assignment, tenant
+   tags, task configs, registry validators, tuners) must run for DDL too.
+
+### 1.1 Non-goals
+
+- No new query semantics. DDL is admin-only; DQL/DML are unchanged.
+- No new JSON wire format for TableConfig — the canonical persisted shape stays
+  identical. DDL is a presentation/grammar layer.
+- No automatic schema migration on `CREATE TABLE`. The hybrid second-variant case
+  is supported, but altering an existing column or shape change is out of scope
+  (deferred — would surface via a future `ALTER TABLE`).
+- No `ALTER TABLE`, no `INSERT INTO ... VALUES`. (`INSERT INTO ... FROM FILE` already
+  exists separately.)
+
+## 2. Architecture
+
+```
+                ┌─────────────────────────────┐
+   POST /sql/ddl│   PinotDdlRestletResource   │  (pinot-controller)
+                └──────────────┬──────────────┘
+                               │ DdlCompiler.compile(sql)
+                               ▼
+                ┌─────────────────────────────┐
+                │   pinot-sql-ddl module      │  ← new module, this PR
+                │  ┌───────────────────────┐  │
+                │  │   compile/            │  │
+                │  │     DdlCompiler       │  │
+                │  │     PropertyMapping   │  │
+                │  │     DataTypeMapper    │  │
+                │  │     CompiledDdl …     │  │
+                │  ├───────────────────────┤  │
+                │  │   resolved/          │  │
+                │  │     ResolvedColumnDef │  │
+                │  │     ResolvedTableDef  │  │
+                │  ├───────────────────────┤  │
+                │  │   reverse/            │  │
+                │  │     CanonicalDdlEmit  │  │
+                │  │     SchemaEmitter     │  │
+                │  │     PropertyExtractor │  │
+                │  │     SqlIdentifiers    │  │
+                │  └───────────────────────┘  │
+                └──────────────┬──────────────┘
+                               │ uses
+                               ▼
+                ┌─────────────────────────────┐
+                │  Calcite parser             │  (pinot-common)
+                │  parserImpls.ftl            │
+                │  + 6 SqlPinot* AST classes  │
+                └─────────────────────────────┘
+```
+
+Module boundary rationale:
+- The Calcite AST nodes live in **`pinot-common`** so the parser can be invoked by other
+  modules without pulling in the DDL compile/reverse logic. The broker query path uses
+  the same parser to classify SQL into DQL/DML/DDL.
+- The compile/reverse logic lives in **`pinot-sql-ddl`** — a new module that depends only
+  on `pinot-spi` (for `Schema`, `TableConfig`, `FieldSpec`), `pinot-common` (for the AST
+  nodes), and `calcite-babel` (for the parser builder). No reverse dependencies.
+- The REST endpoint and integration tests live in **`pinot-controller`**. The controller
+  depends on `pinot-sql-ddl` one-way; nothing else imports it.
+
+## 3. Grammar
+
+The grammar lives in `pinot-common/src/main/codegen/includes/parserImpls.ftl` and is
+generated into `SqlParserImpl` by the existing JavaCC + FMPP build chain.
+
+### 3.1 Productions
+
+```
+SqlPinotCreateTable :=
+    CREATE TABLE [IF NOT EXISTS] CompoundIdentifier
+    PinotColumnList
+    [PinotPrimaryKeyList]
+    TABLE_TYPE = (OFFLINE | REALTIME)
+    [PROPERTIES (PinotPropertyList)]
+
+SqlPinotDropTable :=
+    DROP TABLE [IF EXISTS] CompoundIdentifier
+    [TYPE (OFFLINE | REALTIME)]
+
+SqlPinotShow :=
+    SHOW (
+        TABLES [FROM SimpleIdentifier]
+      | CREATE TABLE CompoundIdentifier [TYPE (OFFLINE | REALTIME)]
+    )
+
+PinotColumnDeclaration :=
+    SimpleIdentifier DataType [NOT NULL | NULL] [DEFAULT Literal]
+    [ DIMENSION [ARRAY] | METRIC | DATETIME FORMAT StringLiteral GRANULARITY StringLiteral ]
+```
+
+### 3.2 New tokens (config.fmpp)
+
+`DIMENSION`, `METRIC`, `GRANULARITY`, `OFFLINE`, `REALTIME`, `PROPERTIES`, `TABLES`,
+`TABLE_TYPE`, `IF`, plus the Pinot-native data type names `LONG`, `BIG_DECIMAL`, `STRING`,
+`BYTES`. **All are added to both `keywords` and `nonReservedKeywordsToAdd`** so existing
+identifiers using these names continue to parse — they remain usable as table/column names.
+
+### 3.3 Parser disambiguation
+
+- `LOOKAHEAD(3)` for `IF NOT EXISTS` (3 tokens needed to disambiguate from `IF`-as-identifier).
+- `LOOKAHEAD(2)` for `IF EXISTS`.
+- `LOOKAHEAD(2)` for the optional PRIMARY KEY clause (` ` is enough; the
+  body fails clearly with "expected `(`" on malformed `PRIMARY KEY id` syntax rather
+  than backtracking to a misleading "expected TABLE_TYPE").
+- The `SHOW` branches are united under a single `SqlPinotShow` entry so JavaCC choice
+  logic doesn't need cross-statement lookahead.
+
+## 4. Compile path: AST → (Schema, TableConfig)
+
+### 4.1 DdlCompiler entry points
+
+`DdlCompiler.compile(String sql)` — single static entry point, returns a `CompiledDdl`
+discriminated by `DdlOperation`:
+
+- `CREATE_TABLE → CompiledCreateTable { Schema, TableConfig, isIfNotExists, warnings }`
+- `DROP_TABLE → CompiledDropTable { rawTableName, databaseName?, tableType?, ifExists }`
+- `SHOW_TABLES → CompiledShowTables { databaseName? }`
+- `SHOW_CREATE_TABLE → CompiledShowCreateTable { rawTableName, databaseName?, tableType? }`
+
+### 4.2 Resolved IR
+
+The compiler first resolves the parser AST into a typed intermediate representation
+(`ResolvedColumnDefinition`, `ResolvedTableDefinition`) that consolidates duplicate-name
+detection, role inference, default-value extraction, and warnings. The IR is what
+populates the `Schema` (via `toFieldSpec`) and the `TableConfig` (via
+`PropertyMapping.apply`).
+
+### 4.3 Type mapping
+
+`DataTypeMapper.resolve(String)` is case-insensitive (`Locale.ROOT`) and accepts both
+ANSI-SQL and Pinot-native names:
+
+| DDL alias | Pinot type |
+|---|---|
+| `INT`, `INTEGER` | INT |
+| `BIGINT`, `LONG` | LONG |
+| `FLOAT`, `REAL` | FLOAT |
+| `DOUBLE` | DOUBLE |
+| `DECIMAL`, `NUMERIC`, `BIG_DECIMAL` | BIG_DECIMAL (precision/scale not enforced; warning emitted) |
+| `BOOLEAN` | BOOLEAN |
+| `TIMESTAMP` | TIMESTAMP |
+| `VARCHAR`, `CHAR`, `STRING` | STRING |
+| `VARBINARY`, `BINARY`, `BYTES` | BYTES |
+| `JSON` | JSON |
+
+`SMALLINT` and `TINYINT` are **rejected explicitly**. Silently widening them to INT today
+would lock those DDLs into INT semantics if Pinot later adds INT8/INT16. A rejected type
+can become accepted later; a silently-promoted type cannot be narrowed without breaking
+users (Hyrum's law / [C1.2](../kb/code-review-principles.md)).
+
+### 4.4 Default-value validation
+
+`DDL DEFAULT 'foo'` for an `INT` column is rejected at compile time, not at first
+ingestion. The compiler runs `dataType.convert(literalString)` and surfaces a
+`DdlCompilationException` mapped to HTTP 400 if it fails. `DEFAULT NULL` is also
+rejected explicitly — Pinot's "default null value" semantic is the value used **when the
+source row is null**, so DEFAULT NULL is meaningless and we fail loudly rather than
+silently no-op.
+
+### 4.5 Property routing — `PropertyMapping.apply`
+
+The `PROPERTIES (...)` clause is the escape hatch for everything not expressed by
+first-class column attributes. Routing is applied in this order:
+
+1. **Promoted scalar.** Lower-cased key matches a known builder field — call
+   `TableConfigBuilder.setX(value)`. List-typed builders (e.g. `invertedIndexColumns`)
+   accept comma-separated values.
+2. **JSON blob.** Lower-cased key matches a known nested-config name —
+   `JsonUtils.stringToObject(value, ConcreteClass.class)` and call the builder setter.
+3. **Stream config.** Key is `streamType` or starts with `stream.` or `realtime.` —
+   store in `IndexingConfig.streamConfigs` verbatim (prefix preserved). REALTIME-only.
+4. **Task prefix.** Key matches `task..` — route the remainder into
+   `TableTaskConfig.taskTypeConfigsMap[taskType]`.
+5. **Custom (fallback).** Anything else — store in `TableCustomConfig.customConfigs`
+   verbatim.
+
+Rules 2-5 are the forward-compat hook: stream and minion-task config schemas evolve
+independently and need not be in lock-step with the DDL grammar. Adding a new
+`fooBarConfig` to TableConfig today and supplying it via DDL **just works** through rule
+2 (after a one-line addition to `applyJsonBlob`) or via rule 5 (zero changes — round-trips
+through `TableCustomConfig`).
+
+The promoted-scalar catalog and the JSON-blob catalog are kept in `RESERVED_ROUND_TRIP_KEYS`
+so the reverse compiler can detect a `TableCustomConfig` entry whose key would be
+re-routed to a different home on round-trip and reject it at emission time. This prevents
+silent loss: a user who legitimately needs a custom-config key named `replication`
+(say, for some downstream tooling) is told at SHOW CREATE TABLE time to rename it.
+
+### 4.6 Hybrid-table second-variant CREATE
+
+When the second physical variant of a hybrid pair is created via DDL with a column list,
+the compiled schema is compared against the **stored** schema (which already exists from
+the first variant) using `describeColumnShapeMismatch`:
+
+- Compares per-column attributes the DDL column list expresses: name, dataType,
+  fieldType (DIMENSION/METRIC/DATETIME), single-value flag, NOT NULL, default null
+  value (typed comparison — BIG_DECIMAL via `compareTo`, BYTES via `Arrays.equals`),
+  DATETIME format, DATETIME granularity.
+- Ignores schema-level metadata the DDL column list cannot express:
+  `primaryKeyColumns`, `tags`, `description`, `enableColumnBasedNullHandling`. These
+  are already set by the first variant; the second-variant DDL inherits them.
+
+`validateTableConfig` is then called against the **stored** schema (when one exists), not
+the compiled schema, so upsert/dedup PK validation sees the canonical PK list rather
+than the synthesized empty list from the DDL column-list-only projection.
+
+### 4.7 Validation pipeline
+
+DDL CREATE delegates to `TableConfigValidationUtils.validateTableConfig` — the same
+helper `POST /tables` uses. This pipeline:
+
+1. `TableConfigUtils.validateTableName`.
+2. `TableConfigUtils.validate(tableConfig, schema, typesToSkip)`.
+3. `TableConfigUtils.ensureMinReplicas(tableConfig, controllerConf.getDefaultTableMinReplicas())`.
+4. `TableConfigUtils.ensureStorageQuotaConstraints(tableConfig, controllerConf.getDimTableMaxSize())`.
+5. `checkHybridTableConfig` (verifies hybrid pair compatibility).
+6. `TaskConfigUtils.validateTaskConfigs`.
+7. `validateInstanceAssignment` (calls `TableRebalancer.getInstancePartitionsMap`).
+8. `validateTableTenantConfig`, `validateTableTaskMinionInstanceTagConfig`.
+9. `TableConfigValidatorRegistry.validate`.
+
+`TableConfigTunerUtils.applyTunerConfigs` runs **before** validation, mirroring
+`POST /tables`. Tuner mutations on the in-flight TableConfig are reflected back into the
+response's `tableConfig` snapshot before returning, so dry-run accurately predicts
+the post-persist shape.
+
+### 4.8 Exception → HTTP-status contract
+
+| Exception type | HTTP status |
+|---|---|
+| `DdlCompilationException` | 400 |
+| `IllegalArgumentException` from `CanonicalDdlEmitter.emit` (unsupported column type, reserved-key collision) | 400 |
+| `IllegalArgumentException`/`IllegalStateException` from `validateTableConfig` | 400 |
+| `TableAlreadyExistsException` (CREATE race) | 409 |
+| `SchemaAlreadyExistsException` (CREATE race) | 409 |
+| Logical-table reference blocks DROP | 409 |
+| `tableTasksCleanup` raises CAE (e.g. active tasks) | preserved (typically 400) |
+| Any other RuntimeException from emit | 500 |
+| Any other Exception from validate / addSchema / addTable | 500 |
+
+The DROP loop tracks the integer status code rather than the `Response.Status` enum so
+non-standard 4xx codes (422, 423, 451) are preserved end-to-end instead of collapsing to 500.
+
+## 5. Reverse path: (Schema, TableConfig) → canonical DDL
+
+`CanonicalDdlEmitter.emit(schema, tableConfig, databaseName?)` produces a deterministic
+DDL string. The contract:
+
+- **Lexicographic property ordering.** `PROPERTIES (...)` entries are sorted by key —
+  same input → same output.
+- **Identifier quoting.** Column and table names that collide with reserved/data-type/DDL
+  keywords are double-quoted (e.g. a column named `int` becomes `"int"`).
+  `SqlIdentifiers.ALWAYS_QUOTE` is intentionally over-cautious: a few extra quotes never
+  break a re-parse, but a missing quote would.
+- **Natural-default elision.** A column at its data-type's natural default does **not**
+  emit a `DEFAULT` clause; this matches Pinot's JSON serialization rule. Type-aware
+  comparison: `Arrays.equals` for BYTES; `BigDecimal.compareTo == 0` for BIG_DECIMAL
+  (scale-insensitive).
+- **Type-canonical default emission.**
+  - BOOLEAN: emit `TRUE` / `FALSE` from internal Integer 0/1.
+  - TIMESTAMP: emit quoted UTC ISO-8601 (`Instant.ofEpochMilli(…).toString()`) — never
+    `java.sql.Timestamp.toString()` which is JVM-timezone-dependent.
+  - BIG_DECIMAL: emit via `toPlainString()` — never scientific notation.
+  - BYTES: emit quoted hex via `BytesUtils.toHexString` — never the byte[] identity-hash form.
+- **Database qualifier.** If the table's effective database is non-default OR the
+  caller passed an explicit database name, emit `db.tableName` so the DDL replays
+  correctly without the `Database:` header.
+- **Hybrid disambiguation.** `SHOW CREATE TABLE foo` on a hybrid pair returns 400 with
+  the message "Use 'TYPE OFFLINE' or 'TYPE REALTIME' to specify which" — the user must
+  disambiguate explicitly.
+- **Unsupported column types fail loudly.** MAP/LIST/STRUCT/UNKNOWN/COMPLEX FieldSpecs
+  throw `IllegalArgumentException` at emission time (mapped to 400) — silent column
+  drop is forbidden.
+- **Unsupported schema/table metadata fails loudly.** Schema metadata that CREATE DDL
+  cannot express yet (`description`, `tags`, `enableColumnBasedNullHandling`) and
+  long-tail `TableConfig` fields not covered by `PropertyExtractor` return 400 when
+  set to non-default values. Returning incomplete DDL that replays to a weaker schema
+  or table config is forbidden.
+
+### 5.1 Reserved-key collision detection
+
+`PropertyExtractor` calls `PropertyMapping.isReservedRoundTripKey(lowerKey)` on every
+custom-config entry it encounters. If the user's JSON-API CREATE put a custom-config key
+that would shadow a promoted scalar or JSON-blob key on round-trip, SHOW CREATE TABLE
+returns 400 with a clear message naming the offending key. This is preferable to a
+silent corruption where the DDL replay would route the value to a different field.
+
+## 6. REST endpoint
+
+```
+POST /sql/ddl[?dryRun=true|false]
+Content-Type: application/json
+Authorization: 
+Database: 
+
+{ "sql": "" }
+```
+
+### 6.1 Pipeline
+
+1. **Input size guard.** Reject SQL longer than `MAX_DDL_SQL_CHARS = 256 * 1024` Java
+   characters with 400 to bound parser allocation.
+2. **Compile.** `DdlCompiler.compile(sql)` → `CompiledDdl`. Compile errors → 400.
+3. **Database resolution.** Translate `(databaseName-from-SQL, Database header)` into
+   the effective database via `DatabaseUtils.translateTableName` (which rejects
+   conflicts).
+4. **Authorize.** `ResourceUtils.checkPermissionAndAccess` against the **post-translation**
+   resource name, with the operation-appropriate `AccessType` and `Action`.
+5. **Dispatch.** Switch on `DdlOperation` to `executeCreate`, `executeDrop`, `executeShow`,
+   or `executeShowCreate`.
+
+### 6.2 CREATE flow
+
+```
+hasTable(typed)? --yes IF NOT EXISTS --> 200 no-op
+              \-- yes !IF NOT EXISTS --> 409
+              \-- no --> readStoredSchema(rawName)
+                         schemaPreexisted? --> describeColumnShapeMismatch
+                         applyTunerConfigs
+                         refresh response.tableConfig snapshot
+                         validateTableConfig(stored or compiled, tableConfig)
+                         if dryRun: 200
+                         else: addSchema (if not preexisted) → addTable → 201
+```
+
+On `addTable` failure: do **not** roll back `addSchema`. The two-read race window
+(`hasOfflineTable + hasRealtimeTable` to decide if the schema is orphaned) could orphan
+a sibling's live table. Pinot already lets schemas outlive tables; stale schemas can be
+removed via `DELETE /schemas/{name}` if needed.
+
+### 6.3 DROP flow
+
+```
+candidates = [foo_OFFLINE, foo_REALTIME]   ── from TYPE clause or both
+authorize(candidates) up-front              ── preserves no-fingerprinting
+targets = candidates.filter(hasTable)
+if targets empty: IF EXISTS? 200 : 404
+checkLogicalTableReferences(targets)        ── 409 if any target referenced
+for target in targets:
+    tableTasksCleanup(target)              ── may throw CAE (e.g. active tasks → 400)
+    deleteTable(rawName, type)
+    track success/failure + tasksCleared
+if all success: 200
+else: surface partial-failure error with first-failure status code preserved,
+      list of dropped/failed targets, and a recovery hint pointing at
+      task-schedule restoration if tasksCleared is non-empty
+```
+
+### 6.4 SHOW CREATE TABLE flow
+
+```
+resolvedType = TYPE clause? : auto
+if resolvedType:                            ── single-variant path
+    authorize(resolvedType.typedName)
+    hasTable? : 404 otherwise
+else:                                       ── no-TYPE: authorize BOTH variants up-front
+    authorize(off, rt)                      ── preserves no-fingerprinting under per-type ACLs
+    offExists, rtExists = hasTable(off), hasTable(rt)
+    both?  -> 400 disambiguation (must specify TYPE)
+    off?   -> resolved = OFFLINE
+    rt?    -> resolved = REALTIME
+    none?  -> 404
+fetch tableConfig (null after hasTable=true => 500 ZK inconsistency)
+fetch schema     (null  -> 404; schemas can be deleted independently)
+emit canonical DDL
+```
+
+### 6.5 Authorization contract
+
+- All authorization runs **after** database-name translation, against the
+  post-translation `db.tableName_TYPE` resource. This prevents a Database header from
+  substituting the resource that auth was checked against (cross-DB privilege
+  escalation).
+- The no-TYPE forms of DROP and SHOW CREATE require permission on **both** variants
+  up-front. Under per-type ACL plugins, this prevents a partial-permission caller from
+  fingerprinting the existence of the variant they don't have access to. Partial-permission
+  callers must use the explicit `TYPE OFFLINE | REALTIME` clause.
+
+## 7. Forward-compatibility
+
+- **New TableConfig nested config (e.g. `fooBarConfig`):** add one `case "foobarconfig"`
+  to `applyJsonBlob` and one branch to `PropertyExtractor.extract` — no grammar change.
+  Until that's done, a stored first-class `TableConfig` field should be guarded by the
+  reverse path so SHOW CREATE TABLE fails fast instead of emitting incomplete DDL.
+  Unknown custom metadata still round-trips through `TableCustomConfig` (rule 5).
+- **New Kafka stream property:** zero changes — rule 3 routes `streamType` and any
+  `stream.*` / `realtime.*` key verbatim.
+- **New minion task type:** zero changes — rule 4 routes any `task..`.
+- **New simple TableConfig builder field:** add one case to `applyPromoted` and one to
+  `PropertyExtractor`. Forgetting one half is a CI failure: `RoundTripTest.promotedScalarsAddedInSlice2RoundTrip`
+  is the kitchen-sink test that fails when a key is emitted by the extractor but not
+  consumed by the mapping.
+
+## 8. Backward-compatibility
+
+- The DDL feature is purely additive. No existing endpoint, SPI signature, enum, or
+  wire format is renamed or removed.
+- New SQL keywords (`DIMENSION`, `METRIC`, `GRANULARITY`, `OFFLINE`, `REALTIME`,
+  `PROPERTIES`, `TABLES`, `TABLE_TYPE`, `IF`) are added as **non-reserved**, so existing
+  identifier usage continues to parse. A column named `metric` works in DQL exactly as
+  before; on canonical-DDL emission it is double-quoted.
+- A pre-DDL controller serving the same cluster simply 404s on `POST /sql/ddl` —
+  rolling-upgrade safe.
+
+## 9. Concurrency and consistency
+
+- All DDL helpers (`DdlCompiler`, `CanonicalDdlEmitter`, `SchemaEmitter`,
+  `PropertyExtractor`, `PropertyMapping`, `SqlIdentifiers`) are stateless static utilities
+  and thread-safe.
+- The REST resource is request-scoped (JAX-RS); the per-target outcome-tracking
+  collections in `executeDrop` are local to the request thread.
+- ZK writes (`addSchema`, `addTable`, `deleteTable`) go through the existing
+  `PinotHelixResourceManager` paths — no new ZK write surface introduced. Concurrency
+  semantics match `POST /tables` and `DELETE /tables/{name}` exactly.
+- The `hasTable → addTable` race window during CREATE is detected via
+  `TableAlreadyExistsException` (mapped to 409). The schema is intentionally **not**
+  rolled back on `addTable` failure to avoid the non-atomic two-read race that could
+  delete a concurrent sibling's schema.
+
+## 10. Testing strategy
+
+| Suite | LOC | Focus |
+|---|---|---|
+| `PinotDdlParserTest` | 399 | Grammar — every syntax variant, `IF NOT EXISTS`, `IF EXISTS`, `SHOW CREATE TABLE`, `SHOW TABLES FROM`, keyword-as-identifier, malformed `PRIMARY KEY` error |
+| `DdlCompilerTest` | 483 | Compile pipeline — every property routing rule, all data type aliases, role inference, default-value type compatibility, negative cases, DECIMAL precision warning, JSON-blob round-trip |
+| `CanonicalDdlEmitterTest` | 433 | Reverse path golden-output — canonical clause order, lexicographic property order, BIG_DECIMAL plain-string, BOOLEAN literals, TIMESTAMP ISO-8601, BYTES hex, ComplexFieldSpec rejection, custom-config key shadowing rejection |
+| `RoundTripTest` | 441 | `emit → parse → compile → emit` idempotence across stream/task/custom configs, ingestion JSON-blob, MV dimensions, primary keys, identifiers needing quoting, kitchen-sink promoted-scalars |
+| `PinotDdlRestletResourceUnitTest` | 290 | `describeColumnShapeMismatch` — every column attribute, BYTES content equality, BIG_DECIMAL `compareTo` equivalence, DATETIME format/granularity |
+| `PinotDdlRestletResourceTest` | 362 | Controller integration on a real `ControllerTest` cluster — CREATE/DROP/SHOW round-trip, dry-run, IF [NOT] EXISTS, 201/200/404/409/400 status codes, DB-qualified DDL, oversize input rejection |
+
+## 11. Known limitations and future work
+
+- **No `ALTER TABLE`.** Schema/config evolution still goes through `PUT /tables` and
+  `PUT /schemas`. Adding `ALTER TABLE` is a natural follow-up slice.
+- **PropertyExtractor long-tail.** Some `IndexingConfig` /
+  `SegmentsValidationAndRetentionConfig` fields are not yet emitted by the reverse
+  path. When those fields are set to non-default values, SHOW CREATE TABLE fails fast
+  with a 400 instead of returning DDL that would silently drop them. Tracked as Slice 4.
+- **PRIMARY KEY round-trip on legacy `TimeFieldSpec`.** The legacy time field path emits
+  as `DATETIME` but doesn't yet carry NOT NULL / DEFAULT through. Narrow case;
+  fixable in a follow-up.
+- **Auth integration tests.** Current integration tests run with the default
+  AllowAll access controller. A follow-up should add tests with a per-type ACL mock
+  to lock in the no-fingerprinting auth-ordering invariants.
+- **Mixed-version classifier test.** No automated test exercises a mixed-version
+  cluster (old broker, new controller) hitting the new endpoint. Documented behavior
+  is that an old controller 404s on `/sql/ddl`, which is rolling-upgrade-safe.
+
+## 12. Decision log
+
+- **Single dispatch endpoint vs one-per-operation.** Chose single `POST /sql/ddl`. The
+  client surface is smaller, dispatch logic is in one place, and the response shape is
+  uniform (single DTO with operation-specific fields filtered via `@JsonInclude(NON_NULL)`).
+- **`TABLE_TYPE = X PROPERTIES (...)` vs SQL-standard `WITH (k=v, ...)`.** Chose the
+  Pinot-flavored form. Reasoning: `TABLE_TYPE` is required and semantically unique to
+  Pinot; making it a positional clause separates it from arbitrary key/value properties.
+  Trino and Snowflake users will recognize the shape; Postgres users will not. We
+  accepted this divergence for clarity over portability.
+- **No-rollback on CREATE failure.** Chose to leave the schema in place if `addTable`
+  fails. The alternative (rollback) requires non-atomic existence checks that race a
+  concurrent sibling CREATE. Stale schemas are recoverable (`DELETE /schemas/{name}`);
+  orphan tables (with their schema deleted) are not.
+- **No-fingerprinting auth.** Chose up-front double-auth on no-TYPE forms. The
+  alternative (auth-after-existence) leaks variant existence to partial-permission
+  callers under per-type ACL plugins.
+- **Reject SMALLINT/TINYINT.** Chose explicit rejection over silent INT widening.
+  Reversible (we can later accept these names with narrower semantics); the inverse
+  is not.
+- **`pinot-sql-ddl` as a separate module vs folding into `pinot-common`.** Chose
+  separate module. The compile/reverse logic is substantial (1500+ LOC) and pulls
+  Calcite-Babel; isolating it lets `pinot-common` consumers (e.g. broker query path)
+  use the parser without paying for the compiler.
+
+## 13. References
+
+- Code review principles: [`kb/code-review-principles.md`](../kb/code-review-principles.md)
+- Pinot table config reference: [`pinot-spi/.../TableConfig.java`](../pinot-spi/src/main/java/org/apache/pinot/spi/config/table/TableConfig.java)
+- Validation pipeline: [`pinot-controller/.../TableConfigValidationUtils.java`](../pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/TableConfigValidationUtils.java)
+- Calcite parser extension docs: [Calcite SQL Parser](https://calcite.apache.org/docs/reference.html)
diff --git a/pinot-sql-ddl/README.md b/pinot-sql-ddl/README.md
new file mode 100644
index 000000000000..7f2872fa0d91
--- /dev/null
+++ b/pinot-sql-ddl/README.md
@@ -0,0 +1,61 @@
+
+# pinot-sql-ddl
+
+Pinot SQL DDL compiler and reverse-emitter. Translates `CREATE TABLE`, `DROP TABLE`,
+`SHOW TABLES`, and `SHOW CREATE TABLE` into the existing `Schema` + `TableConfig` model
+and back.
+
+## Quick example
+
+```sql
+CREATE TABLE events (
+  id    INT NOT NULL DIMENSION,
+  city  STRING DIMENSION,
+  amount DOUBLE METRIC,
+  ts    LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'
+)
+TABLE_TYPE = OFFLINE
+PROPERTIES (
+  'timeColumnName' = 'ts',
+  'replication' = '3'
+);
+```
+
+POST the SQL to `/sql/ddl` on the controller (see [`DESIGN.md`](DESIGN.md) §6 and the
+PR description for the full REST contract and more examples).
+
+## Module layout
+
+| Sub-package | Responsibility |
+|---|---|
+| `compile/` | Forward path: AST → `(Schema, TableConfig)`. Entry point `DdlCompiler`. |
+| `resolved/` | Typed intermediate representation between AST and final config. |
+| `reverse/` | Reverse path: stored `(Schema, TableConfig)` → canonical DDL string. Entry point `CanonicalDdlEmitter`. |
+
+## Where to read more
+
+- [`DESIGN.md`](DESIGN.md) — design document covering grammar, routing rules, validation
+  pipeline, exception → HTTP status contract, and decision log.
+- [`src/main/java/org/apache/pinot/sql/ddl/package-info.java`](src/main/java/org/apache/pinot/sql/ddl/package-info.java)
+  — module-level Javadoc with the same structure for IDE discoverability.
+- [PR #18241](https://github.com/apache/pinot/pull/18241) — landing PR with the full
+  user-manual-style examples.
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledCreateTable.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledCreateTable.java
index f5d4a2fd13f1..4b5adefe20c9 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledCreateTable.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledCreateTable.java
@@ -24,7 +24,7 @@
 import org.apache.pinot.spi.data.Schema;
 
 
-/** Result of compiling {@code CREATE TABLE ...}. */
+/// Result of compiling `CREATE TABLE ...`.
 public final class CompiledCreateTable extends CompiledDdl {
   private final Schema _schema;
   private final TableConfig _tableConfig;
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDdl.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDdl.java
index a9b8789b1af4..8d7f145866a3 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDdl.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDdl.java
@@ -23,11 +23,9 @@
 import javax.annotation.Nullable;
 
 
-/**
- * Base type for the result of compiling a Pinot DDL statement. Concrete subtypes carry the
- * operation-specific payload (Schema/TableConfig for {@code CREATE}, target name for {@code DROP},
- * etc.). The controller dispatches on {@link #getOperation()}.
- */
+/// Base type for the result of compiling a Pinot DDL statement. Concrete subtypes carry the
+/// operation-specific payload (Schema/TableConfig for `CREATE`, target name for `DROP`,
+/// etc.). The controller dispatches on [#getOperation()].
 public abstract class CompiledDdl {
   private final DdlOperation _operation;
   private final String _databaseName;
@@ -43,13 +41,13 @@ public DdlOperation getOperation() {
     return _operation;
   }
 
-  /** Database name from {@code db.table} or {@code SHOW TABLES FROM db}; may be {@code null}. */
+  /// Database name from `db.table` or `SHOW TABLES FROM db`; may be `null`.
   @Nullable
   public String getDatabaseName() {
     return _databaseName;
   }
 
-  /** Non-fatal compile-time warnings (e.g. unknown property routed to TableCustomConfig). */
+  /// Non-fatal compile-time warnings (e.g. unknown property routed to TableCustomConfig).
   public List getWarnings() {
     return _warnings;
   }
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDropTable.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDropTable.java
index 029877ec7ae0..54eebbca8ee3 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDropTable.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledDropTable.java
@@ -23,7 +23,7 @@
 import org.apache.pinot.spi.config.table.TableType;
 
 
-/** Result of compiling {@code DROP TABLE ...}. */
+/// Result of compiling `DROP TABLE ...`.
 public final class CompiledDropTable extends CompiledDdl {
   private final String _rawTableName;
   private final TableType _tableType;
@@ -37,15 +37,13 @@ public CompiledDropTable(@Nullable String databaseName, String rawTableName,
     _ifExists = ifExists;
   }
 
-  /** Bare table name with no database prefix and no _OFFLINE/_REALTIME suffix. */
+  /// Bare table name with no database prefix and no _OFFLINE/_REALTIME suffix.
   public String getRawTableName() {
     return _rawTableName;
   }
 
-  /**
-   * @return the requested type to drop, or {@code null} when both OFFLINE and REALTIME variants
-   *     should be dropped.
-   */
+  /// @return the requested type to drop, or `null` when both OFFLINE and REALTIME variants
+  /// should be dropped.
   @Nullable
   public TableType getTableType() {
     return _tableType;
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java
index 98ce3372b054..10d291e01e71 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowCreateTable.java
@@ -23,13 +23,11 @@
 import org.apache.pinot.spi.config.table.TableType;
 
 
-/**
- * Result of compiling {@code SHOW CREATE TABLE [db.]name [TYPE OFFLINE | REALTIME]}.
- *
- * 
    This is a lookup-only compile result: it carries the target identifier and (optional)
- * type filter; the controller is responsible for fetching the persisted Schema + TableConfig and
- * running them through the canonical DDL emitter.
- */
+/// Result of compiling `SHOW CREATE TABLE [db.]name [TYPE OFFLINE | REALTIME]`.
+///
+/// This is a lookup-only compile result: it carries the target identifier and (optional)
+/// type filter; the controller is responsible for fetching the persisted Schema + TableConfig and
+/// running them through the canonical DDL emitter.
 public final class CompiledShowCreateTable extends CompiledDdl {
   private final String _rawTableName;
   private final TableType _tableType;
@@ -41,17 +39,15 @@ public CompiledShowCreateTable(@Nullable String databaseName, String rawTableNam
     _tableType = tableType;
   }
 
-  /** Bare table name with no database prefix and no _OFFLINE/_REALTIME suffix. */
+  /// Bare table name with no database prefix and no _OFFLINE/_REALTIME suffix.
   public String getRawTableName() {
     return _rawTableName;
   }
 
-  /**
-   * @return the requested type, or {@code null} when the user omitted the {@code TYPE} clause.
-   *     When null, the controller picks the variant that exists; when both OFFLINE and REALTIME
-   *     variants exist, the controller returns 400 BAD_REQUEST and the caller must specify
-   *     {@code TYPE OFFLINE} or {@code TYPE REALTIME} explicitly.
-   */
+  /// @return the requested type, or `null` when the user omitted the `TYPE` clause.
+  /// When null, the controller picks the variant that exists; when both OFFLINE and REALTIME
+  /// variants exist, the controller returns 400 BAD_REQUEST and the caller must specify
+  /// `TYPE OFFLINE` or `TYPE REALTIME` explicitly.
   @Nullable
   public TableType getTableType() {
     return _tableType;
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowTables.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowTables.java
index 39edd5a4110a..9d3beedea203 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowTables.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/CompiledShowTables.java
@@ -22,7 +22,7 @@
 import javax.annotation.Nullable;
 
 
-/** Result of compiling {@code SHOW TABLES [FROM db]}. Carries no payload beyond the database. */
+/// Result of compiling `SHOW TABLES [FROM db]`. Carries no payload beyond the database.
 public final class CompiledShowTables extends CompiledDdl {
   public CompiledShowTables(@Nullable String databaseName) {
     super(DdlOperation.SHOW_TABLES, databaseName, Collections.emptyList());
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java
index b769786d6c81..4fbdf4ae79e2 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DataTypeMapper.java
@@ -24,11 +24,9 @@
 import org.apache.pinot.spi.data.FieldSpec.DataType;
 
 
-/**
- * Maps SQL data type names to Pinot {@link DataType} values. Recognizes both standard SQL names
- * (BIGINT, VARCHAR, etc.) and Pinot-native aliases (LONG, STRING, BIG_DECIMAL, BYTES) that the
- * Calcite grammar already exposes via {@code config.fmpp}.
- */
+/// Maps SQL data type names to Pinot [DataType] values. Recognizes both standard SQL names
+/// (BIGINT, VARCHAR, etc.) and Pinot-native aliases (LONG, STRING, BIG_DECIMAL, BYTES) that the
+/// Calcite grammar already exposes via `config.fmpp`.
 public final class DataTypeMapper {
   private static final Map NAME_TO_DATATYPE;
 
@@ -59,11 +57,9 @@ public final class DataTypeMapper {
   private DataTypeMapper() {
   }
 
-  /**
-   * Resolves a SQL type name (case-insensitive) to a Pinot {@link DataType}.
-   *
-   * @throws DdlCompilationException if the type is not supported.
-   */
+  /// Resolves a SQL type name (case-insensitive) to a Pinot [DataType].
+  ///
+  /// @throws DdlCompilationException if the type is not supported.
   public static DataType resolve(String sqlTypeName) {
     // Locale.ROOT: in Turkish locale "int".toUpperCase() yields "İNT" which fails the lookup.
     String upper = sqlTypeName.toUpperCase(Locale.ROOT);
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompilationException.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompilationException.java
index ca66223e6934..330a6821622f 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompilationException.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompilationException.java
@@ -18,11 +18,9 @@
  */
 package org.apache.pinot.sql.ddl.compile;
 
-/**
- * Runtime exception raised when Pinot DDL compilation fails (parse errors, semantic errors,
- * unsupported syntax, conflicting clauses, etc.). The controller surfaces the message verbatim
- * to the API caller.
- */
+/// Runtime exception raised when Pinot DDL compilation fails (parse errors, semantic errors,
+/// unsupported syntax, conflicting clauses, etc.). The controller surfaces the message verbatim
+/// to the API caller.
 public class DdlCompilationException extends RuntimeException {
   public DdlCompilationException(String message) {
     super(message);
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
index 41af7509620d..cc5181dc4857 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
@@ -54,33 +54,29 @@
 import org.apache.pinot.sql.parsers.parser.SqlPinotShowTables;
 
 
-/**
- * Compiles a Pinot SQL DDL statement into an executable {@link CompiledDdl}.
- *
- * 
    Top-level pipeline:
- * 
    - *   SQL String
- *     → CalciteSqlParser (in pinot-common, generated parser)
- *     → SqlNode (one of SqlPinot{CreateTable,DropTable,ShowTables})
- *     → ResolvedTableDefinition (CREATE only)
- *     → Schema + TableConfig (CREATE only) via {@link PropertyMapping}
- *     → CompiledDdl
- * 
    
- *
- * 
    Stateless and thread-safe. All entry points are static.
- */
+/// Compiles a Pinot SQL DDL statement into an executable [CompiledDdl].
+///
+/// Top-level pipeline:
+/// ```
+///   SQL String
+///     → CalciteSqlParser (in pinot-common, generated parser)
+///     → SqlNode (one of SqlPinot{CreateTable,DropTable,ShowTables})
+///     → ResolvedTableDefinition (CREATE only)
+///     → Schema + TableConfig (CREATE only) via [PropertyMapping]
+///     → CompiledDdl
+/// ```
+///
+/// Stateless and thread-safe. All entry points are static.
 public final class DdlCompiler {
 
   private DdlCompiler() {
   }
 
-  /**
-   * Parses and compiles a DDL statement.
-   *
-   * @param sql the raw SQL string (single statement)
-   * @return a {@link CompiledDdl} subclass appropriate for the operation
-   * @throws DdlCompilationException for parse failures or semantic errors
-   */
+  /// Parses and compiles a DDL statement.
+  ///
+  /// @param sql the raw SQL string (single statement)
+  /// @return a [CompiledDdl] subclass appropriate for the operation
+  /// @throws DdlCompilationException for parse failures or semantic errors
   public static CompiledDdl compile(String sql) {
     SqlNode node = parse(sql);
     if (node instanceof SqlPinotCreateTable) {
@@ -111,6 +107,10 @@ private static CompiledShowCreateTable compileShowCreate(SqlPinotShowCreateTable
   private static SqlNode parse(String sql) {
     try {
       SqlNodeAndOptions parsed = CalciteSqlParser.compileToSqlNodeAndOptions(sql);
+      if (!parsed.getOptions().isEmpty()) {
+        throw new DdlCompilationException("DDL statements do not support query options: "
+            + parsed.getOptions().keySet());
+      }
       return parsed.getSqlNode();
     } catch (SqlCompilationException e) {
       throw new DdlCompilationException("Failed to parse DDL: " + e.getMessage(), e);
@@ -213,15 +213,13 @@ private static List resolveColumns(List colum
     return result;
   }
 
-  /**
-   * Extracts the bare string value of a {@link SqlLiteral} (e.g. {@code 'foo'} → {@code foo},
-   * {@code 0.0} → {@code "0.0"}). Uses {@link SqlLiteral#toValue()} which strips the SQL-wire
-   * quoting that {@code toString()} would otherwise leak into Pinot's defaultNullValue field.
-   *
-   * 
    Calcite's {@code toValue()} throws {@link UnsupportedOperationException} for binary
-   * string and interval literals; we catch and surface a typed {@link DdlCompilationException}
-   * so the caller sees a 400 with a useful message rather than a 500.
-   */
+  /// Extracts the bare string value of a [SqlLiteral] (e.g. `'foo'` → `foo`,
+  /// `0.0` → `"0.0"`). Uses [SqlLiteral#toValue()] which strips the SQL-wire
+  /// quoting that `toString()` would otherwise leak into Pinot's defaultNullValue field.
+  ///
+  /// Calcite's `toValue()` throws [UnsupportedOperationException] for binary
+  /// string and interval literals; we catch and surface a typed [DdlCompilationException]
+  /// so the caller sees a 400 with a useful message rather than a 500.
   private static String extractLiteralValue(@Nullable SqlNode literal) {
     if (literal == null) {
       return null;
@@ -257,7 +255,7 @@ private static String extractLiteralValue(@Nullable SqlNode literal) {
   private static ColumnRole inferRole(SqlPinotColumnDeclaration col, DataType dt) {
     String role = col.getRole();
     if (role == null) {
-      // Default: DIMENSION. Numeric columns can be promoted to METRIC by explicit annotation;
+      // Default: DIMENSION. Metric-compatible columns can be promoted by explicit annotation;
       // we never silently infer METRIC because misclassification causes aggregation surprises.
       return ColumnRole.DIMENSION;
     }
@@ -267,8 +265,8 @@ private static ColumnRole inferRole(SqlPinotColumnDeclaration col, DataType dt)
       case "METRIC":
         if (!isMetricCompatible(dt)) {
           throw new DdlCompilationException(
-              "METRIC role requires a numeric data type; column '" + col.getColumnName().getSimple()
-                  + "' is " + dt + ".");
+              "METRIC role requires a metric-compatible data type (INT, LONG, FLOAT, DOUBLE, BIG_DECIMAL, "
+                  + "or BYTES); column '" + col.getColumnName().getSimple() + "' is " + dt + ".");
         }
         return ColumnRole.METRIC;
       case "DATETIME":
@@ -285,6 +283,7 @@ private static boolean isMetricCompatible(DataType dt) {
       case FLOAT:
       case DOUBLE:
       case BIG_DECIMAL:
+      case BYTES:
         return true;
       default:
         return false;
@@ -384,10 +383,8 @@ private static TableConfig buildTableConfig(ResolvedTableDefinition resolved, Li
     return tableConfig;
   }
 
-  /**
-   * Cross-checks resolved fields against the produced TableConfig (e.g. {@code timeColumnName}
-   * must reference a DATETIME column). Adds advisory warnings for missing-but-recommended fields.
-   */
+  /// Cross-checks resolved fields against the produced TableConfig (e.g. `timeColumnName`
+  /// must reference a DATETIME column). Adds advisory warnings for missing-but-recommended fields.
   private static void validateConsistency(ResolvedTableDefinition resolved, Schema schema,
       TableConfig tableConfig, List warnings) {
     String timeColumnName = tableConfig.getValidationConfig().getTimeColumnName();
@@ -447,7 +444,7 @@ private static TableType parseTableType(String value) {
     throw new DdlCompilationException("Unknown table type: " + value);
   }
 
-  /** Splits a parser identifier into (databaseName, tableName). */
+  /// Splits a parser identifier into (databaseName, tableName).
   private static QualifiedName parseQualifiedName(SqlIdentifier identifier) {
     List names = identifier.names;
     if (names.size() == 1) {
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlOperation.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlOperation.java
index 478c8e478158..964089d11ef2 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlOperation.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlOperation.java
@@ -18,7 +18,7 @@
  */
 package org.apache.pinot.sql.ddl.compile;
 
-/** Discriminator for the operation a {@link CompiledDdl} represents. */
+/// Discriminator for the operation a [CompiledDdl] represents.
 public enum DdlOperation {
   CREATE_TABLE,
   DROP_TABLE,
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
index 6fd6eed4be71..bc32fd4967c0 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/PropertyMapping.java
@@ -33,7 +33,6 @@
 import org.apache.pinot.spi.config.table.DedupConfig;
 import org.apache.pinot.spi.config.table.DimensionTableConfig;
 import org.apache.pinot.spi.config.table.FieldConfig;
-import org.apache.pinot.spi.config.table.IndexingConfig;
 import org.apache.pinot.spi.config.table.JsonIndexConfig;
 import org.apache.pinot.spi.config.table.MultiColumnTextIndexConfig;
 import org.apache.pinot.spi.config.table.QueryConfig;
@@ -59,41 +58,37 @@
 import org.apache.pinot.sql.ddl.resolved.ResolvedTableDefinition;
 
 
-/**
- * Routes Pinot DDL {@code PROPERTIES (...)} entries onto a {@link TableConfigBuilder}.
- *
- * 
    Routing rules (applied in order):
- * 
      
- *   
    1. If the key (case-insensitive) is in the promoted catalog, set the corresponding
- *       {@link TableConfigBuilder} field directly.
      2. 
- *   
    2. If the key starts with {@code stream.} or {@code realtime.}, route the entry into
- *       {@code IndexingConfig.streamConfigs} verbatim (the prefix is preserved, not stripped, so
- *       the key matches existing Pinot stream config conventions like
- *       {@code stream.kafka.topic.name}). REALTIME-only.
      3. 
- *   
    3. If the key starts with {@code task..}, route the remainder into
- *       {@code TableTaskConfig.taskTypeConfigsMap[taskType]}.
      4. 
- *   
    4. Otherwise, store verbatim in {@link TableCustomConfig#getCustomConfigs()}.
- *       This guarantees no silent loss of meaningful config: every DDL property survives the
- *       compile / persist round-trip, even when the DDL grammar is older than the property.
      5. 
- * 
    
- *
- * 
    The free-form pass-through (rules 2-4) is the forward-compatibility hook: stream and minion
- * task config schemas evolve independently and need not be in lock-step with the DDL grammar.
- */
+/// Routes Pinot DDL `PROPERTIES (...)` entries onto a [TableConfigBuilder].
+///
+/// Routing rules (applied in order):
+/// 1. If the key (case-insensitive) is in the promoted catalog, set the corresponding
+/// [TableConfigBuilder] field directly.
+/// 1. If the key is `streamType` or starts with `stream.` or `realtime.`, route the entry into
+/// `IndexingConfig.streamConfigs` verbatim (the prefix is preserved, not stripped, so
+/// the key matches existing Pinot stream config conventions like `streamType` and
+/// `stream.kafka.topic.name`). REALTIME-only.
+/// 1. If the key starts with `task..`, route the remainder into
+/// `TableTaskConfig.taskTypeConfigsMap[taskType]`.
+/// 1. Otherwise, store verbatim in [TableCustomConfig#getCustomConfigs()].
+/// This guarantees no silent loss of meaningful config: every DDL property survives the
+/// compile / persist round-trip, even when the DDL grammar is older than the property.
+///
+/// The free-form pass-through (rules 2-4) is the forward-compatibility hook: stream and minion
+/// task config schemas evolve independently and need not be in lock-step with the DDL grammar.
 public final class PropertyMapping {
 
   private static final String STREAM_PREFIX = "stream.";
+  private static final String STREAM_TYPE_PROPERTY = "streamType";
+  private static final String STREAM_TYPE_KEY = "streamtype";
   private static final String REALTIME_PREFIX = "realtime.";
   private static final String TASK_PREFIX = "task.";
 
-  /** Property keys that carry table-type semantics, handled separately by the caller. */
+  /// Property keys that carry table-type semantics, handled separately by the caller.
   static final Set RESERVED_KEYS = ImmutableSet.of("tabletype", "tablename", "ifnotexists");
 
-  /**
-   * Lowercase property keys that have a dedicated PropertyMapping handler (promoted scalar or
-   * JSON-blob deserialization). Used by the reverse compiler to detect TableCustomConfig keys
-   * that would shadow a reserved key on round-trip and reject them up front.
-   */
+  /// Lowercase property keys that have a dedicated PropertyMapping handler (promoted scalar or
+  /// JSON-blob deserialization). Used by the reverse compiler to detect TableCustomConfig keys
+  /// that would shadow a reserved key on round-trip and reject them up front.
   private static final Set RESERVED_ROUND_TRIP_KEYS;
   static {
     Set keys = new HashSet<>();
@@ -150,20 +145,19 @@ public final class PropertyMapping {
     RESERVED_ROUND_TRIP_KEYS = Collections.unmodifiableSet(keys);
   }
 
-  /**
-   * Returns {@code true} if {@code lowerKey} (already lower-cased) is consumed by a dedicated
-   * PropertyMapping handler — i.e. it would not round-trip safely as a TableCustomConfig entry.
-   *
-   * 
    Catches both exact-match keys (promoted scalars + JSON-blob keys) and the
-   * prefix-routed paths ({@code stream.}, {@code realtime.}, {@code task.}). A custom-config
-   * entry with a key like {@code task.MinionTask.foo} would otherwise be silently routed into
-   * {@code TableTaskConfig} on re-parse.
-   */
+  /// Returns `true` if `lowerKey` (already lower-cased) is consumed by a dedicated
+  /// PropertyMapping handler — i.e. it would not round-trip safely as a TableCustomConfig entry.
+  ///
+  /// Catches both exact-match keys (promoted scalars + JSON-blob keys) and the
+  /// prefix-routed paths (`streamType`, `stream.`, `realtime.`, `task.`). A custom-config
+  /// entry with a key like `task.MinionTask.foo` would otherwise be silently routed into
+  /// `TableTaskConfig` on re-parse.
   public static boolean isReservedRoundTripKey(String lowerKey) {
     if (RESERVED_ROUND_TRIP_KEYS.contains(lowerKey)) {
       return true;
     }
-    return lowerKey.startsWith(STREAM_PREFIX)
+    return STREAM_TYPE_KEY.equals(lowerKey)
+        || lowerKey.startsWith(STREAM_PREFIX)
         || lowerKey.startsWith(REALTIME_PREFIX)
         || lowerKey.startsWith(TASK_PREFIX);
   }
@@ -171,17 +165,15 @@ public static boolean isReservedRoundTripKey(String lowerKey) {
   private PropertyMapping() {
   }
 
-  /**
-   * Applies all properties from {@code definition} onto {@code builder}.
-   *
-   * 
    Returns the full list of sorted columns parsed from the {@code sortedColumn} property so
-   * the caller can apply them directly to {@link IndexingConfig#setSortedColumn(List)} after
-   * {@code builder.build()}. The builder's {@code setSortedColumn(String)} only stores a single
-   * value; using the returned list avoids silently dropping all but the first sort column.
-   *
-   * @throws DdlCompilationException if a promoted key has a non-coercible value (e.g. non-integer
-   *     replication).
-   */
+  /// Applies all properties from `definition` onto `builder`.
+  ///
+  /// Returns the full list of sorted columns parsed from the `sortedColumn` property so
+  /// the caller can apply them directly to [IndexingConfig#setSortedColumn(List)] after
+  /// `builder.build()`. The builder's `setSortedColumn(String)` only stores a single
+  /// value; using the returned list avoids silently dropping all but the first sort column.
+  ///
+  /// @throws DdlCompilationException if a promoted key has a non-coercible value (e.g. non-integer
+  /// replication).
   public static List apply(ResolvedTableDefinition definition, TableConfigBuilder builder) {
     Map streamConfigs = new LinkedHashMap<>();
     Map> taskConfigs = new LinkedHashMap<>();
@@ -223,8 +215,9 @@ public static List apply(ResolvedTableDefinition definition, TableConfig
         continue;
       }
 
-      if (lower.startsWith(STREAM_PREFIX) || lower.startsWith(REALTIME_PREFIX)) {
-        // Both "stream.*" (Pinot stream connection configs) and "realtime.*" (e.g.
+      if (STREAM_TYPE_KEY.equals(lower) || lower.startsWith(STREAM_PREFIX)
+          || lower.startsWith(REALTIME_PREFIX)) {
+        // "streamType", "stream.*" (Pinot stream connection configs), and "realtime.*" (e.g.
         // realtime.segment.flush.threshold.rows / .size / .segment.size) live in
         // IndexingConfig.streamConfigs in real Pinot table configs. Routing them anywhere else
         // makes them silently inert.
@@ -233,9 +226,10 @@ public static List apply(ResolvedTableDefinition definition, TableConfig
           throw new DdlCompilationException(
               kind + " property '" + rawKey + "' is only valid for REALTIME tables.");
         }
-        // Preserve the original key (including its prefix) so existing Pinot stream configs
-        // round-trip identically.
-        streamConfigs.put(rawKey, value);
+        // Preserve ordinary stream keys verbatim so existing Pinot stream configs round-trip
+        // identically. Canonicalize the special un-prefixed streamType key because Pinot's stream
+        // config readers use that exact casing.
+        streamConfigs.put(STREAM_TYPE_KEY.equals(lower) ? STREAM_TYPE_PROPERTY : rawKey, value);
         continue;
       }
 
@@ -268,7 +262,7 @@ public static List apply(ResolvedTableDefinition definition, TableConfig
     return sortedColumns;
   }
 
-  /** Returns true if {@code lowerKey} matched a promoted property and was applied. */
+  /// Returns true if `lowerKey` matched a promoted property and was applied.
   private static boolean applyPromoted(String lowerKey, String value, TableConfigBuilder builder) {
     switch (lowerKey) {
       case "timecolumnname":
@@ -361,11 +355,9 @@ private static boolean applyPromoted(String lowerKey, String value, TableConfigB
     }
   }
 
-  /**
-   * Recognizes JSON-blob property keys for complex nested TableConfig fields and deserializes
-   * them back into the right setter on {@link TableConfigBuilder}. Returns true when the key
-   * was handled (regardless of value validity — invalid JSON throws so the user sees a 400).
-   */
+  /// Recognizes JSON-blob property keys for complex nested TableConfig fields and deserializes
+  /// them back into the right setter on [TableConfigBuilder]. Returns true when the key
+  /// was handled (regardless of value validity — invalid JSON throws so the user sees a 400).
   private static boolean applyJsonBlob(String rawKey, String lowerKey, String value,
       TableConfigBuilder builder) {
     try {
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/package-info.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/package-info.java
index 14c2a46803f8..332e003d3c1b 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/package-info.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/package-info.java
@@ -17,59 +17,51 @@
  * under the License.
  */
 
-/**
- * Apache Pinot SQL DDL feature: compile and reverse-emit DDL statements.
- *
- * 
    Module layering
    
- * The grammar (FreeMarker {@code parserImpls.ftl}) and the AST nodes ({@code SqlPinotCreateTable},
- * {@code SqlPinotDropTable}, etc.) live in {@code pinot-common} so the controller can invoke the
- * Calcite parser without pulling this module. This module ({@code pinot-sql-ddl}) contains the
- * Pinot-specific compile/resolve/reverse-emit logic that turns parser AST nodes into
- * {@link org.apache.pinot.spi.config.table.TableConfig} / {@link org.apache.pinot.spi.data.Schema}
- * pairs and back. Dependencies: {@code pinot-spi} + {@code pinot-common} + {@code calcite-core}.
- *
- * 
    Sub-packages
    
- * 
      
- *   
    • {@link org.apache.pinot.sql.ddl.compile} — forward path: parser AST → compiled DDL
- *       artifact ({@code CompiledCreateTable}, {@code CompiledDropTable}, etc.). The entry point
- *       is {@link org.apache.pinot.sql.ddl.compile.DdlCompiler#compile(String)}.
      • 
- *   
    • {@link org.apache.pinot.sql.ddl.resolved} — typed intermediate representation of resolved
- *       column declarations and table metadata, consumed only by the compiler.
      • 
- *   
    • {@link org.apache.pinot.sql.ddl.reverse} — reverse path: stored {@code Schema} +
- *       {@code TableConfig} → canonical DDL string. Used by {@code SHOW CREATE TABLE}.
      • 
- * 
    
- *
- * 
    Thread safety
    
- * All compiler / emitter classes are stateless and safe for concurrent use. The compiled artifacts
- * ({@code CompiledCreateTable} etc.) are immutable views over freshly-constructed {@code Schema}
- * and {@code TableConfig} objects; callers are responsible for not mutating them after compilation.
- *
- * 
    Exception → HTTP-status contract
    
- * The DDL compiler signals errors via two exception types, which the REST resource translates as:
- * 
      
- *   
    • {@link org.apache.pinot.sql.ddl.compile.DdlCompilationException}: caller-actionable
- *       compile-time errors (unsupported types, malformed property values, type-incompatible
- *       defaults, reserved keys). Surfaced as HTTP 400.
      • 
- *   
    • {@link java.lang.IllegalArgumentException} from {@code CanonicalDdlEmitter.emit(...)}:
- *       canonical DDL grammar cannot represent the schema/config (e.g. unsupported column types
- *       like MAP/LIST/STRUCT, or TableCustomConfig keys that collide with reserved DDL property
- *       names). Surfaced as HTTP 400.
      • 
- *   
    • Any other {@link java.lang.RuntimeException} from emit/compile is treated as a controller
- *       defect and surfaced as HTTP 500.
      • 
- * 
    
- *
- * 
    Evolution policy
    
- * Adding a new TableConfig property:
- * 
      
- *   
    1. If it has a builder setter and round-trips as a single string, add it to
- *       {@link org.apache.pinot.sql.ddl.compile.PropertyMapping#applyPromoted} (forward path) and
- *       {@link org.apache.pinot.sql.ddl.reverse.PropertyExtractor} (reverse path).
      2. 
- *   
    2. Add the lowercase key to {@code RESERVED_ROUND_TRIP_KEYS} so user-supplied
- *       {@code TableCustomConfig} entries cannot shadow the promoted name.
      3. 
- *   
    3. Cover the round-trip in {@code RoundTripTest}.
      4. 
- * 
    
- * Adding a new column attribute (e.g. a new role beyond DIMENSION/METRIC/DATETIME) requires
- * coordinated changes to {@code parserImpls.ftl}, {@code SqlPinotColumnDeclaration},
- * {@code DdlCompiler.toFieldSpec}, and {@code SchemaEmitter.emitColumn} — keep them in sync.
- */
+/// Apache Pinot SQL DDL feature: compile and reverse-emit DDL statements.
+///
+/// ## Module layering
+/// The grammar (FreeMarker `parserImpls.ftl`) and the AST nodes (`SqlPinotCreateTable`,
+/// `SqlPinotDropTable`, etc.) live in `pinot-common` so the controller can invoke the
+/// Calcite parser without pulling this module. This module (`pinot-sql-ddl`) contains the
+/// Pinot-specific compile/resolve/reverse-emit logic that turns parser AST nodes into
+/// [org.apache.pinot.spi.config.table.TableConfig] / [org.apache.pinot.spi.data.Schema]
+/// pairs and back. Dependencies: `pinot-spi` + `pinot-common` + `calcite-core`.
+///
+/// ## Sub-packages
+/// - [org.apache.pinot.sql.ddl.compile] — forward path: parser AST → compiled DDL
+/// artifact (`CompiledCreateTable`, `CompiledDropTable`, etc.). The entry point
+/// is [org.apache.pinot.sql.ddl.compile.DdlCompiler#compile(String)].
+/// - [org.apache.pinot.sql.ddl.resolved] — typed intermediate representation of resolved
+/// column declarations and table metadata, consumed only by the compiler.
+/// - [org.apache.pinot.sql.ddl.reverse] — reverse path: stored `Schema` +
+/// `TableConfig` → canonical DDL string. Used by `SHOW CREATE TABLE`.
+///
+/// ## Thread safety
+/// All compiler / emitter classes are stateless and safe for concurrent use. The compiled artifacts
+/// (`CompiledCreateTable` etc.) are immutable views over freshly-constructed `Schema`
+/// and `TableConfig` objects; callers are responsible for not mutating them after compilation.
+///
+/// ## Exception → HTTP-status contract
+/// The DDL compiler signals errors via two exception types, which the REST resource translates as:
+/// - [org.apache.pinot.sql.ddl.compile.DdlCompilationException]: caller-actionable
+/// compile-time errors (unsupported types, malformed property values, type-incompatible
+/// defaults, reserved keys). Surfaced as HTTP 400.
+/// - [java.lang.IllegalArgumentException] from `CanonicalDdlEmitter.emit(...)`:
+/// canonical DDL grammar cannot represent the schema/config (e.g. unsupported column types
+/// like MAP/LIST/STRUCT, or TableCustomConfig keys that collide with reserved DDL property
+/// names). Surfaced as HTTP 400.
+/// - Any other [java.lang.RuntimeException] from emit/compile is treated as a controller
+/// defect and surfaced as HTTP 500.
+///
+/// ## Evolution policy
+/// Adding a new TableConfig property:
+/// 1. If it has a builder setter and round-trips as a single string, add it to
+/// [org.apache.pinot.sql.ddl.compile.PropertyMapping#applyPromoted] (forward path) and
+/// [org.apache.pinot.sql.ddl.reverse.PropertyExtractor] (reverse path).
+/// 1. Add the lowercase key to `RESERVED_ROUND_TRIP_KEYS` so user-supplied
+/// `TableCustomConfig` entries cannot shadow the promoted name.
+/// 1. Cover the round-trip in `RoundTripTest`.
+/// Adding a new column attribute (e.g. a new role beyond DIMENSION/METRIC/DATETIME) requires
+/// coordinated changes to `parserImpls.ftl`, `SqlPinotColumnDeclaration`,
+/// `DdlCompiler.toFieldSpec`, and `SchemaEmitter.emitColumn` — keep them in sync.
 package org.apache.pinot.sql.ddl;
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ColumnRole.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ColumnRole.java
index 8372365622cb..f553b1a51ebd 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ColumnRole.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ColumnRole.java
@@ -18,15 +18,13 @@
  */
 package org.apache.pinot.sql.ddl.resolved;
 
-/**
- * Schema role for a resolved DDL column. This drives which Pinot {@code FieldSpec} subclass the
- * column compiles to (DimensionFieldSpec / MetricFieldSpec / DateTimeFieldSpec).
- */
+/// Schema role for a resolved DDL column. This drives which Pinot `FieldSpec` subclass the
+/// column compiles to (DimensionFieldSpec / MetricFieldSpec / DateTimeFieldSpec).
 public enum ColumnRole {
-  /** Maps to {@code DimensionFieldSpec}. Filterable / groupable column. */
+  /// Maps to `DimensionFieldSpec`. Filterable / groupable column.
   DIMENSION,
-  /** Maps to {@code MetricFieldSpec}. Aggregatable numeric column. */
+  /// Maps to `MetricFieldSpec`. Aggregatable numeric column.
   METRIC,
-  /** Maps to {@code DateTimeFieldSpec}. Requires explicit format and granularity. */
+  /// Maps to `DateTimeFieldSpec`. Requires explicit format and granularity.
   DATETIME
 }
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedColumnDefinition.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedColumnDefinition.java
index e726671197e1..6f76362790a3 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedColumnDefinition.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedColumnDefinition.java
@@ -22,12 +22,10 @@
 import org.apache.pinot.spi.data.FieldSpec.DataType;
 
 
-/**
- * Normalized column definition produced by the DDL compiler. Independent of the Calcite parse
- * tree so downstream stages (Schema generation, reverse compilation) do not depend on the parser.
- *
- * 
    Immutable; instances are safe to share across threads.
- */
+/// Normalized column definition produced by the DDL compiler. Independent of the Calcite parse
+/// tree so downstream stages (Schema generation, reverse compilation) do not depend on the parser.
+///
+/// Immutable; instances are safe to share across threads.
 public final class ResolvedColumnDefinition {
   private final String _name;
   private final DataType _dataType;
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedTableDefinition.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedTableDefinition.java
index 0f1792b59a66..ae8dee8d1b7d 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedTableDefinition.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/resolved/ResolvedTableDefinition.java
@@ -26,15 +26,13 @@
 import org.apache.pinot.spi.config.table.TableType;
 
 
-/**
- * Normalized {@code CREATE TABLE} definition produced by the DDL compiler. Independent of the
- * Calcite parse tree.
- *
- * 
    Properties retain insertion order via a {@link LinkedHashMap} so that downstream reverse
- * compilation can produce deterministic canonical DDL.
- *
- * 
    Immutable; instances are safe to share across threads.
- */
+/// Normalized `CREATE TABLE` definition produced by the DDL compiler. Independent of the
+/// Calcite parse tree.
+///
+/// Properties retain insertion order via a [LinkedHashMap] so that downstream reverse
+/// compilation can produce deterministic canonical DDL.
+///
+/// Immutable; instances are safe to share across threads.
 public final class ResolvedTableDefinition {
   private final String _databaseName;
   private final String _rawTableName;
@@ -53,13 +51,13 @@ public ResolvedTableDefinition(@Nullable String databaseName, String rawTableNam
     _properties = Collections.unmodifiableMap(new LinkedHashMap<>(properties));
   }
 
-  /** Returns the database name when one was supplied via {@code db.tableName}, else {@code null}. */
+  /// Returns the database name when one was supplied via `db.tableName`, else `null`.
   @Nullable
   public String getDatabaseName() {
     return _databaseName;
   }
 
-  /** Returns the bare table name (no database prefix, no _OFFLINE/_REALTIME suffix). */
+  /// Returns the bare table name (no database prefix, no _OFFLINE/_REALTIME suffix).
   public String getRawTableName() {
     return _rawTableName;
   }
@@ -76,7 +74,7 @@ public List getColumns() {
     return _columns;
   }
 
-  /** Returns property map in declaration order. */
+  /// Returns property map in declaration order.
   public Map getProperties() {
     return _properties;
   }
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java
index 95acaaa54b2c..c95c1d38dc9d 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java
@@ -27,22 +27,18 @@
 import org.apache.pinot.spi.utils.builder.TableNameBuilder;
 
 
-/**
- * Renders a {@code CREATE TABLE} statement in canonical Pinot DDL form from a {@link Schema} and
- * {@link TableConfig}. Designed so that {@code parse(emit(schema, config))} round-trips back to
- * a semantically-equivalent (Schema, TableConfig) pair.
- *
- * 
    Canonical formatting rules:
- * 
      
- *   
    • Two-space indentation, one column per line, trailing comma between entries.
      • 
- *   
    • Clause order: column block, {@code TABLE_TYPE}, {@code PROPERTIES}.
      • 
- *   
    • Property keys in lexicographic order (provided by {@link PropertyExtractor}).
      • 
- *   
    • Identifiers double-quoted only when required (reserved words, special chars).
      • 
- *   
    • String literals always single-quoted; embedded single quotes doubled.
      • 
- * 
    
- *
- * 
    Stateless and thread-safe.
- */
+/// Renders a `CREATE TABLE` statement in canonical Pinot DDL form from a [Schema] and
+/// [TableConfig]. Designed so that `parse(emit(schema, config))` round-trips back to
+/// a semantically-equivalent (Schema, TableConfig) pair.
+///
+/// Canonical formatting rules:
+/// - Two-space indentation, one column per line, trailing comma between entries.
+/// - Clause order: column block, `TABLE_TYPE`, `PROPERTIES`.
+/// - Property keys in lexicographic order (provided by [PropertyExtractor]).
+/// - Identifiers double-quoted only when required (reserved words, special chars).
+/// - String literals always single-quoted; embedded single quotes doubled.
+///
+/// Stateless and thread-safe.
 public final class CanonicalDdlEmitter {
 
   private static final String INDENT = "  ";
@@ -50,24 +46,21 @@ public final class CanonicalDdlEmitter {
   private CanonicalDdlEmitter() {
   }
 
-  /**
-   * Renders the canonical DDL for the given schema + table config.
-   *
-   * @param schema the table's schema; column declarations are derived from its field specs.
-   * @param config the table config; the table name (with type suffix stripped) and all
-   *     non-default settings are emitted.
-   * @return canonical DDL ending with a semicolon and trailing newline.
-   */
+  /// Renders the canonical DDL for the given schema + table config.
+  ///
+  /// @param schema the table's schema; column declarations are derived from its field specs.
+  /// @param config the table config; the table name (with type suffix stripped) and all
+  /// non-default settings are emitted.
+  /// @return canonical DDL ending with a semicolon and trailing newline.
   public static String emit(Schema schema, TableConfig config) {
     return emit(schema, config, null);
   }
 
-  /**
-   * Renders the canonical DDL, scoped under {@code databaseName} when non-null. The database name
-   * is rendered as a leading {@code db.} qualifier on the table name; this matches the parser's
-   * {@code [db.]name} grammar.
-   */
+  /// Renders the canonical DDL, scoped under `databaseName` when non-null. The database name
+  /// is rendered as a leading `db.` qualifier on the table name; this matches the parser's
+  /// `[db.]name` grammar.
   public static String emit(Schema schema, TableConfig config, @Nullable String databaseName) {
+    rejectUnsupportedSchemaMetadata(schema);
     StringBuilder sb = new StringBuilder(512);
 
     String rawTableName = TableNameBuilder.extractRawTableName(config.getTableName());
@@ -137,6 +130,25 @@ public static String emit(Schema schema, TableConfig config, @Nullable String da
     return sb.toString();
   }
 
+  private static void rejectUnsupportedSchemaMetadata(Schema schema) {
+    if (schema.getDescription() != null && !schema.getDescription().isEmpty()) {
+      rejectUnsupportedSchemaField("description");
+    }
+    List tags = schema.getTags();
+    if (tags != null && !tags.isEmpty()) {
+      rejectUnsupportedSchemaField("tags");
+    }
+    if (schema.isEnableColumnBasedNullHandling()) {
+      rejectUnsupportedSchemaField("enableColumnBasedNullHandling");
+    }
+  }
+
+  private static void rejectUnsupportedSchemaField(String field) {
+    throw new IllegalArgumentException("SHOW CREATE TABLE cannot emit schema field '" + field
+        + "' in canonical DDL yet; replaying the emitted DDL would silently drop that setting. "
+        + "Use the JSON schema API for this table until the DDL grammar supports it.");
+  }
+
   private static String emitTableType(TableType tableType) {
     // Exhaustive switch so a future TableType (e.g. UNIFIED) is rejected at the emit boundary
     // rather than silently rendered as REALTIME. The throw maps to the 500 path the controller
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java
index e33729839413..f33767850a43 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/PropertyExtractor.java
@@ -34,49 +34,44 @@
 import org.apache.pinot.sql.ddl.compile.PropertyMapping;
 
 
-/**
- * Inverse of {@code PropertyMapping}: walks a {@link TableConfig} and emits a property map
- * suitable for the {@code PROPERTIES (...)} clause in canonical DDL. The map is sorted
- * lexicographically so canonical output is deterministic.
- *
- * 
    Routing rules mirror {@code PropertyMapping}:
- * 
      
- *   
    • Promoted fields (timeColumnName, replication, brokerTenant, …) emit under their
- *       known property keys.
      • 
- *   
    • Stream configs are emitted verbatim using their original key (preserving the
- *       {@code stream.}/{@code realtime.} prefix).
      • 
- *   
    • Task configs are emitted as {@code task.<type>.<key> = <value>}.
      • 
- *   
    • Table custom-config entries are emitted verbatim.
      • 
- *   
    • Complex nested configs (ingestionConfig, upsertConfig, routingConfig, etc.) that have no
- *       first-class clause yet are emitted as JSON-string property values under the same key
- *       used by Pinot's own JSON serialization. The forward {@code PropertyMapping} parses
- *       these blobs back into the corresponding TableConfig fields.
      • 
- * 
    
- *
- * 
    Defaults are skipped where doing so doesn't lose information (e.g. {@code loadMode=MMAP}
- * or {@code replication=1}) so canonical output stays compact.
- *
- * 
    Coverage scope (Slice 2): this extractor covers the TableConfig fields most users
- * configure today plus a JSON-blob fallback for the major nested configs. Several long-tail
- * fields are intentionally not yet emitted — round-trip will silently lose them — and are
- * tracked for Slice 4 hardening:
- * 
      
- *   
    • SegmentsValidationAndRetentionConfig: {@code segmentAssignmentStrategy} (deprecated),
- *       {@code segmentPushType}/{@code segmentPushFrequency} (deprecated; use IngestionConfig),
- *       {@code replicasPerPartition}, {@code untrackedSegmentsDeletion*} fields.
      • 
- *   
    • IndexingConfig: {@code rangeIndexVersion}, {@code enableDefaultStarTree},
- *       {@code enableDynamicStarTreeCreation}, {@code columnMajorSegmentBuilderEnabled},
- *       {@code skipSegmentPreprocess}, {@code optimizeNoDictStatsCollection},
- *       {@code optimizeDictionary*}, {@code noDictionarySize/CardinalityRatioThreshold},
- *       {@code segmentNameGeneratorType}, {@code columnMinMaxValueGeneratorMode},
- *       {@code autoGeneratedInvertedIndex}, {@code createInvertedIndexDuringSegmentGeneration}.
      • 
- * 
    
- * Slice 4 will either add explicit handlers for each or introduce a guard test that fails
- * compilation when a TableConfig sets a non-default value for an unhandled field.
- */
+/// Inverse of `PropertyMapping`: walks a [TableConfig] and emits a property map
+/// suitable for the `PROPERTIES (...)` clause in canonical DDL. The map is sorted
+/// lexicographically so canonical output is deterministic.
+///
+/// Routing rules mirror `PropertyMapping`:
+/// - Promoted fields (timeColumnName, replication, brokerTenant, …) emit under their
+/// known property keys.
+/// - Stream configs are emitted verbatim using their original key (preserving the
+/// `stream.`/`realtime.` prefix).
+/// - Task configs are emitted as `task.. = `.
+/// - Table custom-config entries are emitted verbatim.
+/// - Complex nested configs (ingestionConfig, upsertConfig, routingConfig, etc.) that have no
+/// first-class clause yet are emitted as JSON-string property values under the same key
+/// used by Pinot's own JSON serialization. The forward `PropertyMapping` parses
+/// these blobs back into the corresponding TableConfig fields.
+///
+/// Defaults are skipped where doing so doesn't lose information (e.g. `loadMode=MMAP`
+/// or `replication=1`) so canonical output stays compact.
+///
+/// **Coverage scope (Slice 2):** this extractor covers the TableConfig fields most users
+/// configure today plus a JSON-blob fallback for the major nested configs. Several long-tail
+/// fields are intentionally not yet emitted; when they are set to non-default values, canonical
+/// DDL emission fails fast instead of returning a replay script that would silently lose them:
+/// - SegmentsValidationAndRetentionConfig: `segmentPushType`/`segmentPushFrequency`
+/// (deprecated; use IngestionConfig), `replacedSegmentsRetentionPeriod`,
+/// `lineageEntryCleanupRetentionPeriod`, `minimizeDataMovement`,
+/// `untrackedSegmentsDeletion*` fields.
+/// - IndexingConfig: `rangeIndexVersion`, `bloomFilterConfigs`,
+/// `noDictionaryConfig`, `enableDefaultStarTree`,
+/// `enableDynamicStarTreeCreation`, `columnMajorSegmentBuilderEnabled`,
+/// `skipSegmentPreprocess`, `optimizeNoDictStatsCollection`,
+/// `optimizeDictionary*`, `noDictionarySize/CardinalityRatioThreshold`,
+/// `segmentNameGeneratorType`, `columnMinMaxValueGeneratorMode`,
+/// `autoGeneratedInvertedIndex`, `createInvertedIndexDuringSegmentGeneration`.
+/// Later slices can replace these guards with explicit property handlers as the grammar expands.
 public final class PropertyExtractor {
 
-  /** Property keys whose values are JSON-serialized blobs of the corresponding nested config. */
+  /// Property keys whose values are JSON-serialized blobs of the corresponding nested config.
   static final String KEY_INGESTION_CONFIG = "ingestionConfig";
   static final String KEY_UPSERT_CONFIG = "upsertConfig";
   static final String KEY_DEDUP_CONFIG = "dedupConfig";
@@ -103,8 +98,9 @@ public final class PropertyExtractor {
   private PropertyExtractor() {
   }
 
-  /** Extracts a deterministic, lexicographically-sorted property map from {@code config}. */
+  /// Extracts a deterministic, lexicographically-sorted property map from `config`.
   public static Map extract(TableConfig config) {
+    rejectUnsupportedFields(config);
     Map props = new TreeMap<>();
     extractValidation(config.getValidationConfig(), props);
     extractTenant(config.getTenantConfig(), props);
@@ -116,6 +112,116 @@ public static Map extract(TableConfig config) {
     return props;
   }
 
+  private static void rejectUnsupportedFields(TableConfig config) {
+    rejectUnsupportedValidation(config.getValidationConfig());
+    rejectUnsupportedIndexing(config.getIndexingConfig());
+  }
+
+  private static void rejectUnsupportedValidation(
+      @Nullable SegmentsValidationAndRetentionConfig validationConfig) {
+    if (validationConfig == null) {
+      return;
+    }
+    rejectIfPresent("segmentsConfig.replacedSegmentsRetentionPeriod",
+        validationConfig.getReplacedSegmentsRetentionPeriod());
+    rejectIfPresent("segmentsConfig.lineageEntryCleanupRetentionPeriod",
+        validationConfig.getLineageEntryCleanupRetentionPeriod());
+    rejectIfPresent("segmentsConfig.segmentPushFrequency",
+        validationConfig.getSegmentPushFrequency());
+    if (validationConfig.getSegmentPushType() != null
+        && !"APPEND".equalsIgnoreCase(validationConfig.getSegmentPushType())) {
+      rejectUnsupported("segmentsConfig.segmentPushType");
+    }
+    if (validationConfig.isMinimizeDataMovement()) {
+      rejectUnsupported("segmentsConfig.minimizeDataMovement");
+    }
+    rejectIfPresent("segmentsConfig.untrackedSegmentsDeletionBatchSize",
+        validationConfig.getUntrackedSegmentsDeletionBatchSize());
+    rejectIfPresent("segmentsConfig.untrackedSegmentsRetentionTimeUnit",
+        validationConfig.getUntrackedSegmentsRetentionTimeUnit());
+    rejectIfPresent("segmentsConfig.untrackedSegmentsRetentionTimeValue",
+        validationConfig.getUntrackedSegmentsRetentionTimeValue());
+  }
+
+  private static void rejectUnsupportedIndexing(@Nullable IndexingConfig indexingConfig) {
+    if (indexingConfig == null) {
+      return;
+    }
+    IndexingConfig defaultIndexingConfig = new IndexingConfig();
+    if (indexingConfig.getRangeIndexVersion() != defaultIndexingConfig.getRangeIndexVersion()) {
+      rejectUnsupported("tableIndexConfig.rangeIndexVersion");
+    }
+    rejectIfNotEmpty("tableIndexConfig.bloomFilterConfigs",
+        indexingConfig.getBloomFilterConfigs());
+    rejectIfNotEmpty("tableIndexConfig.noDictionaryConfig",
+        indexingConfig.getNoDictionaryConfig());
+    if (indexingConfig.isEnableDefaultStarTree()) {
+      rejectUnsupported("tableIndexConfig.enableDefaultStarTree");
+    }
+    if (indexingConfig.isEnableDynamicStarTreeCreation()) {
+      rejectUnsupported("tableIndexConfig.enableDynamicStarTreeCreation");
+    }
+    if (!indexingConfig.isColumnMajorSegmentBuilderEnabled()) {
+      rejectUnsupported("tableIndexConfig.columnMajorSegmentBuilderEnabled");
+    }
+    if (indexingConfig.isSkipSegmentPreprocess()) {
+      rejectUnsupported("tableIndexConfig.skipSegmentPreprocess");
+    }
+    if (indexingConfig.isOptimizeNoDictStatsCollection()) {
+      rejectUnsupported("tableIndexConfig.optimizeNoDictStatsCollection");
+    }
+    if (indexingConfig.isOptimizeDictionary()) {
+      rejectUnsupported("tableIndexConfig.optimizeDictionary");
+    }
+    if (indexingConfig.isOptimizeDictionaryForMetrics()) {
+      rejectUnsupported("tableIndexConfig.optimizeDictionaryForMetrics");
+    }
+    if (indexingConfig.isOptimizeDictionaryType()) {
+      rejectUnsupported("tableIndexConfig.optimizeDictionaryType");
+    }
+    double noDictionarySizeRatioThreshold = indexingConfig.getNoDictionarySizeRatioThreshold();
+    if (Double.compare(noDictionarySizeRatioThreshold, 0.0d) != 0
+        && Double.compare(noDictionarySizeRatioThreshold,
+            IndexingConfig.DEFAULT_NO_DICTIONARY_SIZE_RATIO_THRESHOLD) != 0) {
+      rejectUnsupported("tableIndexConfig.noDictionarySizeRatioThreshold");
+    }
+    Double cardinalityRatioThreshold = indexingConfig.getNoDictionaryCardinalityRatioThreshold();
+    if (cardinalityRatioThreshold != null && Double.compare(cardinalityRatioThreshold, 0.0d) != 0) {
+      rejectUnsupported("tableIndexConfig.noDictionaryCardinalityRatioThreshold");
+    }
+    rejectIfPresent("tableIndexConfig.segmentNameGeneratorType",
+        indexingConfig.getSegmentNameGeneratorType());
+    rejectIfPresent("tableIndexConfig.columnMinMaxValueGeneratorMode",
+        indexingConfig.getColumnMinMaxValueGeneratorMode());
+    if (indexingConfig.isAutoGeneratedInvertedIndex()) {
+      rejectUnsupported("tableIndexConfig.autoGeneratedInvertedIndex");
+    }
+    if (indexingConfig.isCreateInvertedIndexDuringSegmentGeneration()) {
+      rejectUnsupported("tableIndexConfig.createInvertedIndexDuringSegmentGeneration");
+    }
+  }
+
+  private static void rejectIfPresent(String field, @Nullable String value) {
+    if (value != null && !value.isEmpty()) {
+      rejectUnsupported(field);
+    }
+  }
+
+  private static void rejectIfNotEmpty(String field, @Nullable Object value) {
+    if (value instanceof Collection && !((Collection) value).isEmpty()) {
+      rejectUnsupported(field);
+    }
+    if (value instanceof Map && !((Map) value).isEmpty()) {
+      rejectUnsupported(field);
+    }
+  }
+
+  private static void rejectUnsupported(String field) {
+    throw new IllegalArgumentException("SHOW CREATE TABLE cannot emit field '" + field
+        + "' in canonical DDL yet; replaying the emitted DDL would silently drop that setting. "
+        + "Use the JSON table config API for this table until the DDL grammar supports it.");
+  }
+
   private static void extractValidation(@Nullable SegmentsValidationAndRetentionConfig v,
       Map props) {
     if (v == null) {
@@ -229,6 +335,7 @@ private static void extractTopLevel(TableConfig config, Map prop
     putIfPresent(props, "description", config.getDescription());
     List tags = config.getTags();
     if (tags != null && !tags.isEmpty()) {
+      rejectCommaBearingValues("tags", tags);
       props.put("tags", String.join(",", tags));
     }
   }
@@ -264,12 +371,25 @@ private static void putCsvIfPresent(Map props, String key,
       return;
     }
     if (firstOnly) {
+      rejectCommaBearingValues(key, list);
       props.put(key, list.get(0));
     } else {
+      rejectCommaBearingValues(key, list);
       props.put(key, String.join(",", list));
     }
   }
 
+  private static void rejectCommaBearingValues(String key, List list) {
+    for (String value : list) {
+      if (value != null && value.contains(",")) {
+        throw new IllegalArgumentException("SHOW CREATE TABLE cannot emit property '" + key
+            + "' as comma-separated DDL because value '" + value + "' contains a comma; "
+            + "replaying the emitted DDL would split it into multiple values. Use the JSON "
+            + "table config API for this table until the DDL grammar supports escaped CSV values.");
+      }
+    }
+  }
+
   private static void putJsonIfPresent(Map props, String key,
       @Nullable Object obj) {
     if (obj == null) {
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
index cd2799d87247..cbdcc5ed3e22 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
@@ -34,26 +34,22 @@
 import org.apache.pinot.spi.utils.BytesUtils;
 
 
-/**
- * Emits a list of canonical column-declaration strings from a Pinot {@link Schema}. Each
- * declaration is formatted to match the grammar consumed by {@code SqlPinotColumnDeclaration}.
- *
- * 
    Column ordering follows the schema's natural ordering: dimensions first (insertion order),
- * then metrics, then date-time columns. This mirrors what users see in JSON config and is stable
- * across emit/parse round-trips.
- */
+/// Emits a list of canonical column-declaration strings from a Pinot [Schema]. Each
+/// declaration is formatted to match the grammar consumed by `SqlPinotColumnDeclaration`.
+///
+/// Column ordering follows the schema's natural ordering: dimensions first (insertion order),
+/// then metrics, then date-time columns. This mirrors what users see in JSON config and is stable
+/// across emit/parse round-trips.
 final class SchemaEmitter {
 
   private SchemaEmitter() {
   }
 
-  /**
-   * Returns canonical column declarations for all fields in {@code schema}.
-   *
-   * 
    Fails fast with {@link IllegalArgumentException} for MAP/LIST/STRUCT columns because those
-   * types have no DDL representation yet; emitting incomplete DDL for them would produce a
-   * statement that silently drops part of the schema on replay.
-   */
+  /// Returns canonical column declarations for all fields in `schema`.
+  ///
+  /// Fails fast with [IllegalArgumentException] for MAP/LIST/STRUCT columns because those
+  /// types have no DDL representation yet; emitting incomplete DDL for them would produce a
+  /// statement that silently drops part of the schema on replay.
   static List emitColumns(Schema schema) {
     List out = new ArrayList<>();
     for (DimensionFieldSpec dim : schema.getDimensionFieldSpecs()) {
@@ -163,11 +159,9 @@ private static String emitColumn(FieldSpec spec) {
     return sb.toString();
   }
 
-  /**
-   * Maps a Pinot {@link DataType} to the canonical SQL keyword we emit. We pick the
-   * Pinot-native names (LONG, BIG_DECIMAL, STRING, BYTES) where they exist so the round-trip
-   * matches what a human would write.
-   */
+  /// Maps a Pinot [DataType] to the canonical SQL keyword we emit. We pick the
+  /// Pinot-native names (LONG, BIG_DECIMAL, STRING, BYTES) where they exist so the round-trip
+  /// matches what a human would write.
   private static String emitDataType(DataType dt) {
     switch (dt) {
       case INT:
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SqlIdentifiers.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SqlIdentifiers.java
index ec9bd2176d35..85cde1e26ad5 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SqlIdentifiers.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SqlIdentifiers.java
@@ -24,16 +24,14 @@
 import java.util.regex.Pattern;
 
 
-/** Quoting helpers used by the canonical DDL emitter. */
+/// Quoting helpers used by the canonical DDL emitter.
 final class SqlIdentifiers {
 
-  /**
-   * SQL identifiers that need double-quoting because they conflict with the Pinot DDL grammar
-   * (reserved or context-sensitive keywords) or because they are not safe bare identifiers.
-   *
-   * 
    This set is intentionally over-cautious: a few extra quotes never break round-trip,
-   * but a missing quote would cause a re-parse to misinterpret the identifier as a keyword.
-   */
+  /// SQL identifiers that need double-quoting because they conflict with the Pinot DDL grammar
+  /// (reserved or context-sensitive keywords) or because they are not safe bare identifiers.
+  ///
+  /// This set is intentionally over-cautious: a few extra quotes never break round-trip,
+  /// but a missing quote would cause a re-parse to misinterpret the identifier as a keyword.
   private static final Set ALWAYS_QUOTE = ImmutableSet.of(
       // grammar keywords introduced by Pinot DDL that could appear as user identifiers
       "DIMENSION", "METRIC", "DATETIME", "FORMAT", "GRANULARITY", "OFFLINE", "REALTIME",
@@ -53,10 +51,8 @@ final class SqlIdentifiers {
   private SqlIdentifiers() {
   }
 
-  /**
-   * Returns {@code identifier} ready to embed in canonical DDL: double-quoted if required,
-   * bare otherwise. Embedded double quotes are escaped per SQL convention ({@code "} → {@code ""}).
-   */
+  /// Returns `identifier` ready to embed in canonical DDL: double-quoted if required,
+  /// bare otherwise. Embedded double quotes are escaped per SQL convention (`"` → `""`).
   static String quote(String identifier) {
     if (mustQuote(identifier)) {
       return "\"" + identifier.replace("\"", "\"\"") + "\"";
@@ -64,10 +60,8 @@ static String quote(String identifier) {
     return identifier;
   }
 
-  /**
-   * Returns a single-quoted SQL string literal for {@code value}; embedded single quotes are
-   * doubled per SQL convention.
-   */
+  /// Returns a single-quoted SQL string literal for `value`; embedded single quotes are
+  /// doubled per SQL convention.
   static String quoteString(String value) {
     return "'" + value.replace("'", "''") + "'";
   }
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
index 5c10e84eb867..8b880e52a1df 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
@@ -37,7 +37,7 @@
 import static org.testng.Assert.expectThrows;
 
 
-/** End-to-end compiler tests: SQL → CompiledDdl → Schema + TableConfig. */
+/// End-to-end compiler tests: SQL → CompiledDdl → Schema + TableConfig.
 public class DdlCompilerTest {
 
   // -------------------------------------------------------------------------------------------
@@ -167,14 +167,17 @@ public void streamPropertiesRoutedToStreamConfigsForRealtime() {
             + "  ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'"
             + ") TABLE_TYPE = REALTIME PROPERTIES ("
             + "  'timeColumnName' = 'ts',"
+            + "  'streamtype' = 'kafka',"
             + "  'stream.kafka.topic.name' = 'orders',"
             + "  'stream.kafka.consumer.factory.class.name' = 'KafkaConsumerFactory',"
             + "  'realtime.segment.flush.threshold.rows' = '500000'"
             + ")");
     Map stream = c.getTableConfig().getIndexingConfig().getStreamConfigs();
     assertNotNull(stream);
-    // Both "stream.*" and "realtime.*" prefixes route to streamConfigs because that is where
+    // "streamType", "stream.*", and "realtime.*" route to streamConfigs because that is where
     // Pinot actually reads them; routing elsewhere would make them silently inert.
+    assertEquals(stream.get("streamType"), "kafka");
+    assertFalse(stream.containsKey("streamtype"));
     assertEquals(stream.get("stream.kafka.topic.name"), "orders");
     assertEquals(stream.get("stream.kafka.consumer.factory.class.name"), "KafkaConsumerFactory");
     assertEquals(stream.get("realtime.segment.flush.threshold.rows"), "500000");
@@ -198,12 +201,10 @@ public void stringDefaultDoesNotLeakSqlQuotes() {
     assertEquals(defaultValue, "unknown");
   }
 
-  /**
-   * TABLE_TYPE parsing is case-insensitive (parseTableType uses equalsIgnoreCase), but the
-   * compiled TableConfig always stores the canonical uppercase form. Lock in the
-   * lowercase-input → uppercase-output behavior so a future grammar tightening cannot silently
-   * regress to case-sensitive matching.
-   */
+  /// TABLE_TYPE parsing is case-insensitive (parseTableType uses equalsIgnoreCase), but the
+  /// compiled TableConfig always stores the canonical uppercase form. Lock in the
+  /// lowercase-input → uppercase-output behavior so a future grammar tightening cannot silently
+  /// regress to case-sensitive matching.
   @Test
   public void lowercaseTableTypeAcceptedAndCanonicalized() {
     CompiledCreateTable lowerCase = compileCreate(
@@ -215,11 +216,9 @@ public void lowercaseTableTypeAcceptedAndCanonicalized() {
     assertEquals(mixedCase.getTableConfig().getTableType(), TableType.REALTIME);
   }
 
-  /**
-   * DEFAULT literals must be compatible with the column's declared data type. Non-numeric
-   * defaults on numeric columns must be rejected at compile time with a clear error rather
-   * than failing at first ingestion with a downstream-layer error.
-   */
+  /// DEFAULT literals must be compatible with the column's declared data type. Non-numeric
+  /// defaults on numeric columns must be rejected at compile time with a clear error rather
+  /// than failing at first ingestion with a downstream-layer error.
   @Test
   public void defaultLiteralWrongTypeRejected() {
     DdlCompilationException ex = expectThrows(DdlCompilationException.class, () -> compileCreate(
@@ -230,11 +229,9 @@ public void defaultLiteralWrongTypeRejected() {
         "expected error to name the column, got: " + ex.getMessage());
   }
 
-  /**
-   * SMALLINT and TINYINT are explicitly rejected to keep the type contract narrow: silently
-   * widening to INT today would lock those DDLs into INT semantics if Pinot later adds
-   * INT8/INT16. Rejection at the boundary is reversible; silent promotion is not.
-   */
+  /// SMALLINT and TINYINT are explicitly rejected to keep the type contract narrow: silently
+  /// widening to INT today would lock those DDLs into INT semantics if Pinot later adds
+  /// INT8/INT16. Rejection at the boundary is reversible; silent promotion is not.
   @Test
   public void smallintTinyintRejectedExplicitly() {
     DdlCompilationException ex1 = expectThrows(DdlCompilationException.class, () -> compileCreate(
@@ -247,12 +244,10 @@ public void smallintTinyintRejectedExplicitly() {
         "expected error to name TINYINT, got: " + ex2.getMessage());
   }
 
-  /**
-   * DEFAULT NULL is semantically meaningless for Pinot's "default null value" concept (the
-   * value used when the source row is null). A user writing it would get silently no-op
-   * behavior under the previous implementation; we now reject explicitly so the user sees a
-   * clear error and corrects their DDL.
-   */
+  /// DEFAULT NULL is semantically meaningless for Pinot's "default null value" concept (the
+  /// value used when the source row is null). A user writing it would get silently no-op
+  /// behavior under the previous implementation; we now reject explicitly so the user sees a
+  /// clear error and corrects their DDL.
   @Test
   public void defaultNullRejectedExplicitly() {
     DdlCompilationException ex = expectThrows(DdlCompilationException.class, () -> compileCreate(
@@ -331,9 +326,28 @@ public void duplicatePropertyRejected() {
   }
 
   @Test
-  public void metricRoleRequiresNumericType() {
-    expectThrows(DdlCompilationException.class, () -> compileCreate(
+  public void queryOptionsRejected() {
+    DdlCompilationException setOption = expectThrows(DdlCompilationException.class, () -> DdlCompiler.compile(
+        "SET timeoutMs = '1'; CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE"));
+    assertTrue(setOption.getMessage().contains("query options"), setOption.getMessage());
+
+    DdlCompilationException legacyOption = expectThrows(DdlCompilationException.class, () -> DdlCompiler.compile(
+        "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE OPTION(timeoutMs = '1')"));
+    assertTrue(legacyOption.getMessage().contains("query options"), legacyOption.getMessage());
+  }
+
+  @Test
+  public void metricRoleRequiresMetricCompatibleType() {
+    DdlCompilationException e = expectThrows(DdlCompilationException.class, () -> compileCreate(
         "CREATE TABLE t (s STRING METRIC) TABLE_TYPE = OFFLINE"));
+    assertTrue(e.getMessage().contains("metric-compatible"), e.getMessage());
+  }
+
+  @Test
+  public void bytesMetricRoleIsAccepted() {
+    CompiledCreateTable c = compileCreate("CREATE TABLE t (digest BYTES METRIC) TABLE_TYPE = OFFLINE");
+    assertTrue(c.getSchema().getFieldSpecFor("digest") instanceof MetricFieldSpec);
+    assertEquals(c.getSchema().getFieldSpecFor("digest").getDataType(), DataType.BYTES);
   }
 
   @Test
@@ -349,6 +363,12 @@ public void streamPropertyOnOfflineTableRejected() {
             + "  'stream.kafka.topic.name' = 'orders')"));
   }
 
+  @Test
+  public void streamTypePropertyOnOfflineTableRejected() {
+    expectThrows(DdlCompilationException.class, () -> compileCreate(
+        "CREATE TABLE t (id INT) TABLE_TYPE = OFFLINE PROPERTIES ('streamType' = 'kafka')"));
+  }
+
   @Test
   public void invalidTaskPropertyShapeRejected() {
     expectThrows(DdlCompilationException.class, () -> compileCreate(
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
index 56dc1145ec5c..afe598a59d5b 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
@@ -39,11 +39,9 @@
 import static org.testng.Assert.assertTrue;
 
 
-/**
- * Golden-output unit tests for {@link CanonicalDdlEmitter}. Every test asserts the exact emitted
- * string so the canonical form is locked in: any drift requires updating both the emitter and
- * the golden expectation in the same PR.
- */
+/// Golden-output unit tests for [CanonicalDdlEmitter]. Every test asserts the exact emitted
+/// string so the canonical form is locked in: any drift requires updating both the emitter and
+/// the golden expectation in the same PR.
 public class CanonicalDdlEmitterTest {
 
   @Test
@@ -120,6 +118,7 @@ public void streamConfigsRoundTripWithOriginalKeys() {
         .addDateTime("ts", DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
         .build();
     Map streamCfgs = new LinkedHashMap<>();
+    streamCfgs.put("streamType", "kafka");
     streamCfgs.put("stream.kafka.topic.name", "click_events");
     streamCfgs.put("stream.kafka.consumer.factory.class.name", "KafkaConsumerFactory");
     streamCfgs.put("realtime.segment.flush.threshold.rows", "500000");
@@ -130,6 +129,7 @@ public void streamConfigsRoundTripWithOriginalKeys() {
         .build();
 
     String emitted = CanonicalDdlEmitter.emit(schema, config);
+    assertTrue(emitted.contains("'streamType' = 'kafka'"), emitted);
     assertTrue(emitted.contains("'stream.kafka.topic.name' = 'click_events'"), emitted);
     assertTrue(emitted.contains(
         "'stream.kafka.consumer.factory.class.name' = 'KafkaConsumerFactory'"), emitted);
@@ -152,11 +152,9 @@ public void noDefaultsEmittedWhenAtNaturalDefault() {
     assertFalse(emitted.contains("'loadMode'"), emitted);
   }
 
-  /**
-   * Regression: comparing default value vs natural default with Object.equals does reference
-   * equality on byte[], so a BYTES column at its natural default would always emit a redundant
-   * DEFAULT '' clause. The fix uses DataType.equals which delegates to Arrays.equals.
-   */
+  /// Regression: comparing default value vs natural default with Object.equals does reference
+  /// equality on byte[], so a BYTES column at its natural default would always emit a redundant
+  /// DEFAULT '' clause. The fix uses DataType.equals which delegates to Arrays.equals.
   @Test
   public void noDefaultEmittedForBytesAtNaturalDefault() {
     Schema schema = new Schema();
@@ -170,10 +168,8 @@ public void noDefaultEmittedForBytesAtNaturalDefault() {
         "BYTES column at natural default must not emit a DEFAULT clause; got:\n" + emitted);
   }
 
-  /**
-   * Regression: BigDecimal.toString() can emit scientific notation (1E+30) which Calcite's
-   * Literal() rule does not accept. The fix routes BIG_DECIMAL defaults through toPlainString().
-   */
+  /// Regression: BigDecimal.toString() can emit scientific notation (1E+30) which Calcite's
+  /// Literal() rule does not accept. The fix routes BIG_DECIMAL defaults through toPlainString().
   @Test
   public void bigDecimalDefaultEmitsPlainString() {
     Schema schema = new Schema();
@@ -190,11 +186,9 @@ public void bigDecimalDefaultEmitsPlainString() {
         "expected plain-string form of 1E+30; got:\n" + emitted);
   }
 
-  /**
-   * Regression: BIG_DECIMAL natural-default check must use compareTo rather than equals,
-   * so a stored BigDecimal("0.0") (scale 1) is treated as equivalent to BigDecimal.ZERO and
-   * canonical DDL elides the redundant DEFAULT clause.
-   */
+  /// Regression: BIG_DECIMAL natural-default check must use compareTo rather than equals,
+  /// so a stored BigDecimal("0.0") (scale 1) is treated as equivalent to BigDecimal.ZERO and
+  /// canonical DDL elides the redundant DEFAULT clause.
   @Test
   public void bigDecimalAtNaturalDefaultDoesNotEmitDefault() {
     Schema schema = new Schema();
@@ -209,10 +203,8 @@ public void bigDecimalAtNaturalDefaultDoesNotEmitDefault() {
         "BIG_DECIMAL at scale-shifted natural default must not emit DEFAULT; got:\n" + emitted);
   }
 
-  /**
-   * Regression: BOOLEAN columns store defaults internally as Integer 0/1; canonical DDL
-   * must emit the SQL literal form (TRUE/FALSE) so the output is grammar-standard.
-   */
+  /// Regression: BOOLEAN columns store defaults internally as Integer 0/1; canonical DDL
+  /// must emit the SQL literal form (TRUE/FALSE) so the output is grammar-standard.
   @Test
   public void booleanDefaultEmittedAsSqlLiteral() {
     Schema schema = new Schema();
@@ -229,10 +221,8 @@ public void booleanDefaultEmittedAsSqlLiteral() {
         "must not emit raw integer encoding (DEFAULT 1); got:\n" + emitted);
   }
 
-  /**
-   * Regression: BYTES non-natural-default emit must hex-encode the byte[] rather than fall
-   * through to value.toString() which would emit "[B@" identity-hash garbage.
-   */
+  /// Regression: BYTES non-natural-default emit must hex-encode the byte[] rather than fall
+  /// through to value.toString() which would emit "[B@" identity-hash garbage.
   @Test
   public void bytesNonNaturalDefaultEmittedAsHex() {
     Schema schema = new Schema();
@@ -249,11 +239,9 @@ public void bytesNonNaturalDefaultEmittedAsHex() {
         "BYTES default must not leak the JVM byte[] identity-hash form; got:\n" + emitted);
   }
 
-  /**
-   * Regression: TIMESTAMP DEFAULT emission must use UTC ISO-8601 form (Instant.toString)
-   * rather than java.sql.Timestamp.toString, which formats in the JVM's default time zone
-   * and would make canonical DDL emit different strings on different controllers.
-   */
+  /// Regression: TIMESTAMP DEFAULT emission must use UTC ISO-8601 form (Instant.toString)
+  /// rather than java.sql.Timestamp.toString, which formats in the JVM's default time zone
+  /// and would make canonical DDL emit different strings on different controllers.
   @Test
   public void timestampDefaultEmittedInUtcIso() {
     Schema schema = new Schema();
@@ -392,8 +380,8 @@ public void customConfigKeyShadowingPromotedKeyRejected() {
 
   @Test
   public void customConfigKeyShadowingTaskPrefixRejected() {
-    // Same root-cause as above but for the prefix-routed paths: task.., stream.*,
-    // and realtime.* are all consumed by PropertyMapping's prefix routing on re-parse, so
+    // Same root-cause as above but for the prefix-routed paths: task.., streamType,
+    // stream.*, and realtime.* are all consumed by PropertyMapping's routing on re-parse, so
     // a TableCustomConfig entry under any of those prefixes would silently land in the wrong
     // TableConfig field. Reject so the user knows to rename.
     Schema schema = new Schema.SchemaBuilder()
@@ -413,6 +401,25 @@ public void customConfigKeyShadowingTaskPrefixRejected() {
     }
   }
 
+  @Test
+  public void customConfigKeyShadowingStreamTypeRejected() {
+    Schema schema = new Schema.SchemaBuilder()
+        .setSchemaName("t")
+        .addSingleValueDimension("id", DataType.INT)
+        .build();
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
+        .setTableName("t")
+        .setCustomConfig(new TableCustomConfig(
+            Collections.singletonMap("streamType", "not-a-stream-config")))
+        .build();
+    try {
+      CanonicalDdlEmitter.emit(schema, config);
+      org.testng.Assert.fail("Expected IllegalArgumentException for shadowing streamType key");
+    } catch (IllegalArgumentException expected) {
+      assertTrue(expected.getMessage().contains("streamType"), expected.getMessage());
+    }
+  }
+
   @Test
   public void deterministicOutputAcrossRepeatedCalls() {
     Schema schema = new Schema.SchemaBuilder()
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
index ccac382dc3f3..b4bfe9380817 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
@@ -21,6 +21,7 @@
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.node.ObjectNode;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.Iterator;
 import java.util.LinkedHashMap;
 import java.util.Map;
@@ -48,23 +49,22 @@
 import static org.testng.Assert.assertFalse;
 import static org.testng.Assert.assertNotNull;
 import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
 
 
-/**
- * Round-trip suite: original (Schema, TableConfig) → canonical DDL → re-parse → re-compile →
- * round-tripped (Schema, TableConfig). Each test asserts the round-tripped pair is semantically
- * equivalent to the original — same Schema (column shape, datetime format, primary keys) and
- * same TableConfig fields.
- *
- * 
    Semantic equivalence is computed by comparing JSON serializations rather than direct
- * .equals(): TableConfig and Schema do not implement equals() reliably across all nested
- * configs, but their JSON representations are what eventually persist to ZK and what callers
- * actually compare.
- *
- * 
    Fixtures here are synthetic so the test is hermetic and does not depend on examples in
- * other modules. The set deliberately exercises every routing rule
- * ({@code stream.*}, {@code task.*}, JSON blob, custom config, promoted scalar, CSV list).
- */
+/// Round-trip suite: original (Schema, TableConfig) → canonical DDL → re-parse → re-compile →
+/// round-tripped (Schema, TableConfig). Each test asserts the round-tripped pair is semantically
+/// equivalent to the original — same Schema (column shape, datetime format, primary keys) and
+/// same TableConfig fields.
+///
+/// Semantic equivalence is computed by comparing JSON serializations rather than direct
+/// .equals(): TableConfig and Schema do not implement equals() reliably across all nested
+/// configs, but their JSON representations are what eventually persist to ZK and what callers
+/// actually compare.
+///
+/// Fixtures here are synthetic so the test is hermetic and does not depend on examples in
+/// other modules. The set deliberately exercises every routing rule
+/// (`streamType`, `stream.*`, `task.*`, JSON blob, custom config, promoted scalar, CSV list).
 public class RoundTripTest {
 
   // -------------------------------------------------------------------------------------------
@@ -131,6 +131,7 @@ public void realtimeTableWithStreamConfigs() {
         .addDateTime("ts", DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
         .build();
     Map streamCfgs = new LinkedHashMap<>();
+    streamCfgs.put("streamType", "kafka");
     streamCfgs.put("stream.kafka.topic.name", "click_events");
     streamCfgs.put("stream.kafka.consumer.factory.class.name", "KafkaConsumerFactory");
     streamCfgs.put("realtime.segment.flush.threshold.rows", "500000");
@@ -214,6 +215,15 @@ public void columnDefaultsRoundTrip() {
     assertRoundTrip(schema, config);
   }
 
+  @Test
+  public void bytesMetricRoundTrips() {
+    Schema schema = new Schema();
+    schema.setSchemaName("t");
+    schema.addField(new MetricFieldSpec("digest", DataType.BYTES));
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE).setTableName("t").build();
+    assertRoundTrip(schema, config);
+  }
+
   @Test
   public void identifiersWithReservedNamesRoundTrip() {
     // Column named "metric" requires quoting on emit; round-trip must preserve the name.
@@ -373,11 +383,88 @@ public void replicasPerPartitionRoundTrip() {
     assertRoundTrip(schema, config);
   }
 
+  @Test
+  public void unsupportedValidationConfigFieldFailsFastInsteadOfSilentLoss() {
+    Schema schema = new Schema.SchemaBuilder()
+        .setSchemaName("events")
+        .addSingleValueDimension("id", DataType.INT)
+        .build();
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
+        .setTableName("events")
+        .build();
+    config.getValidationConfig().setReplacedSegmentsRetentionPeriod("3d");
+    assertUnsupportedShowCreate(schema, config, "segmentsConfig.replacedSegmentsRetentionPeriod");
+  }
+
+  @Test
+  public void unsupportedIndexingConfigFieldFailsFastInsteadOfSilentLoss() {
+    Schema schema = new Schema.SchemaBuilder()
+        .setSchemaName("events")
+        .addSingleValueDimension("id", DataType.INT)
+        .build();
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
+        .setTableName("events")
+        .build();
+    config.getIndexingConfig().setNoDictionaryConfig(Collections.singletonMap("id", "RAW"));
+    assertUnsupportedShowCreate(schema, config, "tableIndexConfig.noDictionaryConfig");
+  }
+
+  @Test
+  public void unsupportedSchemaMetadataFailsFastInsteadOfSilentLoss() {
+    Schema schema = new Schema.SchemaBuilder()
+        .setSchemaName("events")
+        .addSingleValueDimension("id", DataType.INT)
+        .setEnableColumnBasedNullHandling(true)
+        .build();
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
+        .setTableName("events")
+        .build();
+    assertUnsupportedShowCreate(schema, config, "enableColumnBasedNullHandling");
+  }
+
+  @Test
+  public void csvPropertyValueWithCommaFailsFastInsteadOfSilentSplit() {
+    Schema schema = new Schema.SchemaBuilder()
+        .setSchemaName("events")
+        .addSingleValueDimension("country,city", DataType.STRING)
+        .build();
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
+        .setTableName("events")
+        .setSortedColumn("country,city")
+        .build();
+    assertUnsupportedShowCreate(schema, config, "sortedColumn");
+  }
+
+  @Test
+  public void tagValueWithCommaFailsFastInsteadOfSilentSplit() {
+    Schema schema = new Schema.SchemaBuilder()
+        .setSchemaName("events")
+        .addSingleValueDimension("id", DataType.INT)
+        .build();
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
+        .setTableName("events")
+        .setTags(Arrays.asList("team,one", "critical"))
+        .build();
+    assertUnsupportedShowCreate(schema, config, "tags");
+  }
+
   // -------------------------------------------------------------------------------------------
   // Equivalence machinery
   // -------------------------------------------------------------------------------------------
 
-  /** Asserts that emit -> parse -> compile produces a semantically equivalent (schema, config). */
+  private static void assertUnsupportedShowCreate(Schema schema, TableConfig config,
+      String expectedField) {
+    try {
+      CanonicalDdlEmitter.emit(schema, config);
+      fail("Expected SHOW CREATE TABLE emission to reject unsupported field " + expectedField);
+    } catch (IllegalArgumentException expected) {
+      assertTrue(expected.getMessage().contains(expectedField),
+          "Expected unsupported-field error to mention " + expectedField + " but got: "
+              + expected.getMessage());
+    }
+  }
+
+  /// Asserts that emit -> parse -> compile produces a semantically equivalent (schema, config).
   private static void assertRoundTrip(Schema originalSchema, TableConfig originalConfig) {
     String ddl = CanonicalDdlEmitter.emit(originalSchema, originalConfig);
     CompiledCreateTable round = (CompiledCreateTable) DdlCompiler.compile(ddl);
@@ -404,11 +491,9 @@ private static void assertTableConfigEquivalent(TableConfig a, TableConfig b, St
         + "\nExpected: " + aJson + "\nActual:   " + bJson);
   }
 
-  /**
-   * Removes fields that are not meaningful for semantic comparison. Empty maps in TableCustomConfig
-   * compare-equal whether the field is null, missing, or {}, so we strip them. Same for
-   * empty lists added by builders that are not user-meaningful.
-   */
+  /// Removes fields that are not meaningful for semantic comparison. Empty maps in TableCustomConfig
+  /// compare-equal whether the field is null, missing, or {}, so we strip them. Same for
+  /// empty lists added by builders that are not user-meaningful.
   private static JsonNode stripVolatile(JsonNode node) {
     if (node == null || !node.isObject()) {
       return node;

From 4bc6aaa5f795d6d50db2030f87ca0b93bc6de97d Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Thu, 14 May 2026 21:55:15 -0700
Subject: [PATCH 28/32] Prefer TIMESTAMP over LONG/EPOCH for DDL time-column
 examples

Switch the README quickstart and the tests that specifically validate
DateTimeFieldSpec format normalization (compiler's
columnRolesProduceCorrectFieldSpecs, emitter's allColumnRolesAndDatetime,
parser's createTableAllColumnModifiers) to use TIMESTAMP as the time-column
data type, since TIMESTAMP is the recommended shape for new tables.

Behavior surfaced: DateTimeFieldSpec silently rewrites the FORMAT string to
the bare token "TIMESTAMP" whenever the column data type is TIMESTAMP
(see pinot-spi DateTimeFieldSpec lines 78-81). The README therefore uses
the canonical short form FORMAT 'TIMESTAMP' that SHOW CREATE TABLE will
emit, so what users copy matches what the tool produces. Test comments
document the silent rewrite.

Coverage preserved on the historical LONG/EPOCH path:
- Parser negative cases (datetimeWithoutFormatFails,
  datetimeWithoutGranularityFails) and createTableRealtime still use
  LONG/EPOCH.
- Compiler property-routing tests (promotedPropertiesMapToTableConfigFields,
  streamPropertiesRoutedToStreamConfigsForRealtime) still use LONG/EPOCH.
- Emitter streamConfigsRoundTripWithOriginalKeys and the dedicated
  datetimeFieldRoundTripsFormatAndGranularity (SIMPLE_DATE_FORMAT) keep
  the LONG path.
- All 5 original RoundTripTest cases keep LONG/EPOCH.
- Controller integration tests keep LONG/EPOCH.

Added one new round-trip test (offlineTableWithTimestampTimeColumn) so the
TIMESTAMP emit -> parse -> compile -> idempotency loop is locked under
positive regression coverage.

Co-Authored-By: Claude Opus 4.7 
---
 .../pinot/sql/parsers/PinotDdlParserTest.java  |  7 +++++--
 pinot-sql-ddl/README.md                        |  2 +-
 .../pinot/sql/ddl/compile/DdlCompilerTest.java |  7 +++++--
 .../ddl/reverse/CanonicalDdlEmitterTest.java   |  7 +++++--
 .../pinot/sql/ddl/roundtrip/RoundTripTest.java | 18 ++++++++++++++++++
 5 files changed, 34 insertions(+), 7 deletions(-)

diff --git a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
index b944db6c0d8d..f497d0c2bf72 100644
--- a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
@@ -90,7 +90,7 @@ public void createTableAllColumnModifiers() {
             + "  d2 INT NOT NULL DIMENSION,"
             + "  m1 LONG METRIC,"
             + "  m2 DOUBLE NOT NULL DEFAULT 0.0 METRIC,"
-            + "  ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'"
+            + "  ts TIMESTAMP DATETIME FORMAT '1:MILLISECONDS:TIMESTAMP' GRANULARITY '1:MILLISECONDS'"
             + ") TABLE_TYPE = OFFLINE");
     assertEquals(node.getColumns().size(), 5);
     SqlPinotColumnDeclaration d2 = (SqlPinotColumnDeclaration) node.getColumns().get(1);
@@ -104,7 +104,10 @@ public void createTableAllColumnModifiers() {
     SqlPinotColumnDeclaration ts = (SqlPinotColumnDeclaration) node.getColumns().get(4);
     assertEquals(ts.getRole(), "DATETIME");
     assertNotNull(ts.getDateTimeFormat());
-    assertEquals(ts.getDateTimeFormat().toValue(), "1:MILLISECONDS:EPOCH");
+    // Parser captures the FORMAT literal verbatim; the post-compile DateTimeFieldSpec
+    // normalizes this to "TIMESTAMP" for TIMESTAMP data type, but the AST faithfully
+    // carries the original token.
+    assertEquals(ts.getDateTimeFormat().toValue(), "1:MILLISECONDS:TIMESTAMP");
     assertEquals(ts.getDateTimeGranularity().toValue(), "1:MILLISECONDS");
   }
 
diff --git a/pinot-sql-ddl/README.md b/pinot-sql-ddl/README.md
index 7f2872fa0d91..402b9aa2c838 100644
--- a/pinot-sql-ddl/README.md
+++ b/pinot-sql-ddl/README.md
@@ -31,7 +31,7 @@ CREATE TABLE events (
   id    INT NOT NULL DIMENSION,
   city  STRING DIMENSION,
   amount DOUBLE METRIC,
-  ts    LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'
+  ts    TIMESTAMP DATETIME FORMAT 'TIMESTAMP' GRANULARITY '1:MILLISECONDS'
 )
 TABLE_TYPE = OFFLINE
 PROPERTIES (
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
index 8b880e52a1df..180d4417b0ce 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/compile/DdlCompilerTest.java
@@ -73,7 +73,7 @@ public void columnRolesProduceCorrectFieldSpecs() {
         "CREATE TABLE t ("
             + "  d1 STRING DIMENSION,"
             + "  m1 LONG METRIC,"
-            + "  ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'"
+            + "  ts TIMESTAMP DATETIME FORMAT '1:MILLISECONDS:TIMESTAMP' GRANULARITY '1:MILLISECONDS'"
             + ") TABLE_TYPE = OFFLINE");
 
     Schema s = c.getSchema();
@@ -83,7 +83,10 @@ public void columnRolesProduceCorrectFieldSpecs() {
     assertTrue(d1 instanceof DimensionFieldSpec);
     assertTrue(m1 instanceof MetricFieldSpec);
     assertTrue(ts instanceof DateTimeFieldSpec);
-    assertEquals(((DateTimeFieldSpec) ts).getFormat(), "1:MILLISECONDS:EPOCH");
+    // DateTimeFieldSpec normalizes the format string to "TIMESTAMP" when the column
+    // data type is TIMESTAMP, regardless of the user-supplied format. See
+    // DateTimeFieldSpec#setFormat — the format is implicit in the data type.
+    assertEquals(((DateTimeFieldSpec) ts).getFormat(), "TIMESTAMP");
     assertEquals(((DateTimeFieldSpec) ts).getGranularity(), "1:MILLISECONDS");
   }
 
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
index afe598a59d5b..ed2d803c0ff5 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/reverse/CanonicalDdlEmitterTest.java
@@ -70,7 +70,7 @@ public void allColumnRolesAndDatetime() {
         .setSchemaName("events")
         .addSingleValueDimension("dim", DataType.STRING)
         .addMetric("sum", DataType.LONG)
-        .addDateTime("ts", DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+        .addDateTime("ts", DataType.TIMESTAMP, "1:MILLISECONDS:TIMESTAMP", "1:MILLISECONDS")
         .build();
     TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
         .setTableName("events")
@@ -80,8 +80,11 @@ public void allColumnRolesAndDatetime() {
     String emitted = CanonicalDdlEmitter.emit(schema, config);
     assertTrue(emitted.contains("dim STRING DIMENSION"), emitted);
     assertTrue(emitted.contains("sum LONG METRIC"), emitted);
+    // DateTimeFieldSpec normalizes the format to "TIMESTAMP" when the column data type is
+    // TIMESTAMP, so the canonical emission carries the short form regardless of what the
+    // caller passed to addDateTime.
     assertTrue(emitted.contains(
-        "ts LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' GRANULARITY '1:MILLISECONDS'"), emitted);
+        "ts TIMESTAMP DATETIME FORMAT 'TIMESTAMP' GRANULARITY '1:MILLISECONDS'"), emitted);
     assertTrue(emitted.contains("'timeColumnName' = 'ts'"), emitted);
   }
 
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
index b4bfe9380817..2c940aefcc8c 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
@@ -104,6 +104,24 @@ public void offlineTableWithRetentionAndTenants() {
     assertRoundTrip(schema, config);
   }
 
+  /// Regression: TIMESTAMP-typed time columns must round-trip through canonical emit, parse, and
+  /// re-compile. DateTimeFieldSpec normalizes the format string to "TIMESTAMP" on construction
+  /// (the format is implicit in the data type), so the emitted DDL carries `FORMAT 'TIMESTAMP'`
+  /// and the re-parsed schema must reproduce the same normalized form.
+  @Test
+  public void offlineTableWithTimestampTimeColumn() {
+    Schema schema = new Schema.SchemaBuilder()
+        .setSchemaName("events")
+        .addSingleValueDimension("id", DataType.INT)
+        .addDateTime("ts", DataType.TIMESTAMP, "1:MILLISECONDS:TIMESTAMP", "1:MILLISECONDS")
+        .build();
+    TableConfig config = new TableConfigBuilder(TableType.OFFLINE)
+        .setTableName("events")
+        .setTimeColumnName("ts")
+        .build();
+    assertRoundTrip(schema, config);
+  }
+
   @Test
   public void offlineTableWithIndexingConfig() {
     Schema schema = new Schema.SchemaBuilder()

From 89a899e68069e633d816b38829f594c370587ce8 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Thu, 14 May 2026 22:23:51 -0700
Subject: [PATCH 29/32] Address review feedback: BOOLEAN default round-trip
 test + Swagger 404 + rolling-upgrade doc + FQCN cleanup
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Multi-domain review found 0 critical and 9 major findings. This commit
addresses the actionable major findings and the low-risk minor nits.

- RoundTripTest: add offlineTableWithBooleanDefault to lock in BOOLEAN
  DEFAULT TRUE end-to-end (parse -> compile -> emit -> re-parse -> emit
  idempotency) and to pin the natural-default elision behavior for
  DEFAULT FALSE. Calcite's SqlLiteral.toValue() returns "TRUE"/"FALSE";
  FieldSpec.setDefaultNullValue routes those through
  DataType.BOOLEAN.convert (BooleanUtils.toInt) to Integer 0/1. Since
  the BOOLEAN dimension natural default is Integer 0, the canonical
  emitter correctly elides DEFAULT FALSE.

- PinotDdlRestletResource @ApiResponses: add 404 (DROP without IF
  EXISTS, SHOW CREATE on missing table) and clarify 400, 409, 500
  preconditions so Swagger-generated clients know to handle the full
  failure surface.

- DESIGN.md: expand Section 8 with an explicit 8.1 "Rolling-upgrade
  safety" subsection covering mixed-version ordering, ZK schema
  immutability, pre-DDL controller behavior, rollback safety, and why
  the new SQL keywords cannot break existing DQL. Makes the
  rolling-upgrade story auditable.

- SchemaEmitter: replace inline FQCN
  org.apache.pinot.spi.data.TimeGranularitySpec with a proper import.
  Matches the project's "imports over fully qualified class names"
  convention.

- PinotDdlParserTest: replace inline java.util.Locale.ROOT FQCN with a
  Locale import. Same convention.

Skipped findings (not actionable as stated):
- "RoundTripTest doesn't structurally compare JSON" — actually does;
  assertRoundTrip uses JsonUtils.objectToJsonNode and JsonNode.equals,
  which is structural, not byte-level.
- "applyJsonBlob masks Jackson exception detail" — the catch already
  attaches the cause AND embeds e.getMessage() (which carries the
  Jackson [Source: ...; line: N, column: N] suffix) in the
  DdlCompilationException message. The 400 response surfaces that.

Tests: 31 parser + 86 sql-ddl (+1 BOOLEAN round-trip) + 35 controller
DDL tests pass. Spotless / checkstyle / license clean.

Co-Authored-By: Claude Opus 4.7 
---
 .../pinot/sql/parsers/PinotDdlParserTest.java |  3 +-
 .../resources/PinotDdlRestletResource.java    |  9 +++--
 pinot-sql-ddl/DESIGN.md                       | 29 ++++++++++++++--
 .../pinot/sql/ddl/reverse/SchemaEmitter.java  |  3 +-
 .../sql/ddl/roundtrip/RoundTripTest.java      | 33 +++++++++++++++++++
 5 files changed, 70 insertions(+), 7 deletions(-)

diff --git a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
index f497d0c2bf72..0e7c3202078a 100644
--- a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
+++ b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/PinotDdlParserTest.java
@@ -18,6 +18,7 @@
  */
 package org.apache.pinot.sql.parsers;
 
+import java.util.Locale;
 import org.apache.calcite.sql.SqlIdentifier;
 import org.apache.calcite.sql.SqlNode;
 import org.apache.pinot.sql.parsers.parser.SqlPinotColumnDeclaration;
@@ -329,7 +330,7 @@ public void createTableMissingPrimaryKeyParensRejected() {
         () -> CalciteSqlParser.compileToSqlNodeAndOptions(
             "CREATE TABLE t (id INT) PRIMARY KEY id TABLE_TYPE = OFFLINE"));
     String message = ex.getMessage() == null ? "" : ex.getMessage();
-    assertTrue(message.contains("(") || message.toUpperCase(java.util.Locale.ROOT).contains("LPAREN"),
+    assertTrue(message.contains("(") || message.toUpperCase(Locale.ROOT).contains("LPAREN"),
         "expected error to indicate the missing LPAREN, got: " + message);
   }
 
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 98ad3a12204e..735bdf52807a 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -156,9 +156,12 @@ public class PinotDdlRestletResource {
   @ApiResponses(value = {
       @ApiResponse(code = 200, message = "Success (DROP, SHOW TABLES, SHOW CREATE TABLE, dry-run, IF NOT EXISTS)"),
       @ApiResponse(code = 201, message = "Table created"),
-      @ApiResponse(code = 400, message = "Bad request (parse/semantic error)"),
-      @ApiResponse(code = 409, message = "Table already exists"),
-      @ApiResponse(code = 500, message = "Internal server error")
+      @ApiResponse(code = 400, message = "Bad request (parse error, semantic error, oversize input, "
+          + "unsupported emitter type, or type-incompatible default value)"),
+      @ApiResponse(code = 404, message = "Table or schema not found (DROP without IF EXISTS, SHOW CREATE)"),
+      @ApiResponse(code = 409, message = "Conflict (duplicate CREATE without IF NOT EXISTS, logical-table "
+          + "reference blocking DROP, or race lost to a concurrent writer)"),
+      @ApiResponse(code = 500, message = "Internal server error (Helix/ZK inconsistency or controller defect)")
   })
   public Response executeDdl(
       @ApiParam(value = "DDL request body with 'sql' field", required = true)
diff --git a/pinot-sql-ddl/DESIGN.md b/pinot-sql-ddl/DESIGN.md
index c03028b7d67f..2aafd43b3f68 100644
--- a/pinot-sql-ddl/DESIGN.md
+++ b/pinot-sql-ddl/DESIGN.md
@@ -441,8 +441,33 @@ emit canonical DDL
   `PROPERTIES`, `TABLES`, `TABLE_TYPE`, `IF`) are added as **non-reserved**, so existing
   identifier usage continues to parse. A column named `metric` works in DQL exactly as
   before; on canonical-DDL emission it is double-quoted.
-- A pre-DDL controller serving the same cluster simply 404s on `POST /sql/ddl` —
-  rolling-upgrade safe.
+
+### 8.1 Rolling-upgrade safety
+
+The DDL surface is safe under any rolling-upgrade ordering because the only persisted
+artifacts it produces (Schema + TableConfig in ZK) are the same shape that
+`POST /tables` and `POST /schemas` already produce. Concretely:
+
+- **No ZK schema change.** `DdlCompiler` returns a stock `(Schema, TableConfig)` pair
+  and persists them through the existing `PinotHelixResourceManager` paths. There are
+  no new ZK property-store paths, no new fields on `TableConfig`/`Schema`, and no
+  versioning bumps. A new-controller CREATE produces a ZK record indistinguishable
+  from one written by an old controller via the JSON API.
+- **Old controllers, brokers, and servers see no difference at runtime.** They read
+  the same `TableConfig`/`Schema` JSON they always read. The DDL-created table is a
+  regular Pinot table from their perspective.
+- **Pre-DDL controllers simply 404 on `POST /sql/ddl`.** Clients that probe the new
+  endpoint during the rolling window get a clean 404 from the old binary and a
+  normal response from the new binary. There is no in-between state in which the
+  endpoint half-exists.
+- **Direction of rollout is irrelevant.** Upgrading controllers first, brokers
+  first, servers first, or any interleaving all produce the same on-disk
+  representation. A subsequent rollback to a pre-DDL controller continues to serve
+  tables created via DDL identically to JSON-API-created tables.
+- **New SQL keywords do not break DQL.** Because every new keyword is added under
+  `nonReservedKeywordsToAdd` in `config.fmpp`, existing user queries that use words
+  like `dimension`, `metric`, `format`, or `granularity` as identifiers continue to
+  parse on the new binary. A pre-DDL binary never saw the tokens at all.
 
 ## 9. Concurrency and consistency
 
diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
index cbdcc5ed3e22..24a6e1e436e9 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/reverse/SchemaEmitter.java
@@ -31,6 +31,7 @@
 import org.apache.pinot.spi.data.MetricFieldSpec;
 import org.apache.pinot.spi.data.Schema;
 import org.apache.pinot.spi.data.TimeFieldSpec;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
 import org.apache.pinot.spi.utils.BytesUtils;
 
 
@@ -101,7 +102,7 @@ private static String emitTimeColumn(TimeFieldSpec spec) {
     // Emit the legacy time column as a DATETIME column so it survives a round-trip.
     // Use the outgoing granularity spec to derive format and granularity strings matching
     // the DateTimeFieldSpec convention ({size}:{unit}:{format}).
-    org.apache.pinot.spi.data.TimeGranularitySpec tgs = spec.getOutgoingGranularitySpec();
+    TimeGranularitySpec tgs = spec.getOutgoingGranularitySpec();
     String format = tgs.getTimeUnitSize() + ":" + tgs.getTimeType().name() + ":"
         + tgs.getTimeFormat();
     String granularity = tgs.getTimeUnitSize() + ":" + tgs.getTimeType().name();
diff --git a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
index 2c940aefcc8c..61e340dce87a 100644
--- a/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
+++ b/pinot-sql-ddl/src/test/java/org/apache/pinot/sql/ddl/roundtrip/RoundTripTest.java
@@ -122,6 +122,39 @@ public void offlineTableWithTimestampTimeColumn() {
     assertRoundTrip(schema, config);
   }
 
+  /// Regression: BOOLEAN columns store defaults internally as Integer 0/1; the SQL literal
+  /// `DEFAULT TRUE` must round-trip end-to-end. Calcite's `SqlLiteral.toValue()` returns the
+  /// upper-case token "TRUE" for `BOOLEAN_TRUE`; `FieldSpec.setDefaultNullValue` then routes
+  /// it through `DataType.BOOLEAN.convert(...)` (which calls `BooleanUtils.toInt`) to land
+  /// the stored value as Integer 1. The emitter writes the SQL literal form back. This test
+  /// locks in the full DDL → compile → emit → re-parse loop and also pins the natural-default
+  /// elision behaviour for `DEFAULT FALSE` (BOOLEAN's stored natural default is Integer 0, so
+  /// the canonical emitter intentionally omits the DEFAULT clause for `FALSE`).
+  @Test
+  public void offlineTableWithBooleanDefault() {
+    String ddl = "CREATE TABLE flags (\n"
+        + "  flag BOOLEAN NOT NULL DEFAULT TRUE DIMENSION,\n"
+        + "  inactive BOOLEAN DEFAULT FALSE DIMENSION\n"
+        + ")\n"
+        + "TABLE_TYPE = OFFLINE;\n";
+    CompiledCreateTable first = (CompiledCreateTable) DdlCompiler.compile(ddl);
+    String firstEmit = CanonicalDdlEmitter.emit(first.getSchema(), first.getTableConfig());
+    assertTrue(firstEmit.contains("flag BOOLEAN NOT NULL DEFAULT TRUE DIMENSION"),
+        "Initial emit must preserve BOOLEAN NOT NULL DEFAULT TRUE; got:\n" + firstEmit);
+    // DEFAULT FALSE is the natural default for BOOLEAN dimensions (stored as Integer 0), so the
+    // emitter elides the DEFAULT clause. The column itself still appears.
+    assertTrue(firstEmit.contains("inactive BOOLEAN DIMENSION"),
+        "BOOLEAN column at natural default FALSE must emit without a DEFAULT clause; got:\n"
+            + firstEmit);
+    assertFalse(firstEmit.contains("inactive BOOLEAN DEFAULT FALSE"),
+        "BOOLEAN natural default FALSE must not be emitted; got:\n" + firstEmit);
+    // Idempotency: emit -> parse -> emit must produce the same canonical text.
+    CompiledCreateTable second = (CompiledCreateTable) DdlCompiler.compile(firstEmit);
+    String secondEmit = CanonicalDdlEmitter.emit(second.getSchema(), second.getTableConfig());
+    assertEquals(secondEmit, firstEmit,
+        "BOOLEAN DEFAULT canonical DDL must be idempotent across re-emit:\n" + firstEmit);
+  }
+
   @Test
   public void offlineTableWithIndexingConfig() {
     Schema schema = new Schema.SchemaBuilder()

From 436f1a74300c21b66b8606d861b90bb5cb945daf Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Thu, 14 May 2026 23:59:38 -0700
Subject: [PATCH 30/32] Polish DDL response shape and document control-plane
 races

Final-round review identified three actionable polish items.

- DdlExecutionResponse._dryRun: change primitive boolean to boxed
  Boolean so @JsonInclude(NON_NULL) elides it from SHOW TABLES /
  SHOW CREATE TABLE responses where dry-run semantics do not apply.
  Previously those responses always emitted "dryRun":false, which
  is misleading. CREATE and DROP responses still carry the field
  (both have a working dry-run path). Class Javadoc updated to
  document the per-operation visibility.

- PinotDdlRestletResource.executeCreate: expand the
  TableAlreadyExistsException race comment to acknowledge the
  second-order race where a third caller DROPs the table between
  the 409 throw and the IF-NOT-EXISTS re-check, causing a 409
  return even though IF NOT EXISTS would normally be satisfied.
  Recovery is a retry; closing the window requires a version-checked
  create-or-no-op primitive on PinotHelixResourceManager that does
  not currently exist.

- PinotDdlRestletResource.assertNoLogicalTableReferences: add
  Javadoc covering the O(L) ZK-read cost per DROP (L = cluster
  logical-table count). Matches the existing DELETE /tables/{name}
  contract; documented so bulk-drop callers know what to expect.

Tests: 35 controller DDL tests pass. Spotless / checkstyle / license
clean.

Co-Authored-By: Claude Opus 4.7 
---
 .../api/resources/PinotDdlRestletResource.java   | 16 ++++++++++++++++
 .../api/resources/ddl/DdlExecutionResponse.java  | 15 +++++++++++----
 2 files changed, 27 insertions(+), 4 deletions(-)

diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
index 735bdf52807a..857bf72d5c00 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotDdlRestletResource.java
@@ -340,6 +340,13 @@ private Response executeCreate(CompiledCreateTable create, String database,
       // (legitimate hybrid-pair pattern), and a hasTable re-check followed by deleteSchema is
       // racy in the same way the generic-failure branch below is. Stale schemas can be removed
       // via DELETE /schemas/{name} if needed.
+      //
+      // Second-order race: between TableAlreadyExistsException and the IF-NOT-EXISTS hasTable
+      // re-check, a third caller may have DROPped the table. In that narrow window the
+      // re-check returns false and we fall through to throw 409 even though IF NOT EXISTS
+      // would normally be satisfied. The user can retry; Pinot's `addTable` does not currently
+      // expose a version-checked create-or-no-op primitive that would close this window. If
+      // one is added, switch this branch to use it.
       if (create.isIfNotExists() && _pinotHelixResourceManager.hasTable(tableNameWithType)) {
         response.setMessage("Table " + tableNameWithType
             + " already exists; CREATE IF NOT EXISTS is a no-op.");
@@ -648,6 +655,15 @@ private DdlExecutionResponse executeDrop(CompiledDropTable drop, String database
     return response;
   }
 
+  /// Rejects the DROP when any target physical table is currently referenced by a logical table.
+  /// Matches the same safeguard enforced by the existing `DELETE /tables/{name}` endpoint.
+  ///
+  /// Cost: O(L) ZK reads to fetch every logical-table config, plus O(L × T) reference checks
+  /// where L is the cluster's logical-table count and T is the DROP target count (1 or 2). For
+  /// clusters with hundreds of logical tables, a tight loop of DDL DROPs can produce a
+  /// noticeable controller-side ZK read load; callers performing bulk drops may prefer the
+  /// existing JSON `DELETE` endpoint, which has the same cost per call but is friendlier to
+  /// concurrent batch tooling.
   private void assertNoLogicalTableReferences(List targets) {
     List allLogicalTableConfigs =
         ZKMetadataProvider.getAllLogicalTableConfigs(_pinotHelixResourceManager.getPropertyStore());
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java
index c794fbd8966e..00a0d848a661 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ddl/DdlExecutionResponse.java
@@ -30,14 +30,21 @@
 /// The shape of the response varies by operation. [JsonInclude.Include#NON_NULL] keeps
 /// the wire payload focused on the fields that actually apply to the operation that ran.
 ///
-/// - CREATE_TABLE: `tableName, tableType, schema, tableConfig, ifNotExists, warnings`
-/// - DROP_TABLE: `tableName, tableType, deletedTables, ifExists`
+/// - CREATE_TABLE: `tableName, tableType, schema, tableConfig, ifNotExists, warnings, dryRun`
+/// - DROP_TABLE: `tableName, tableType, deletedTables, ifExists, dryRun`
 /// - SHOW_TABLES: `tableNames`
 /// - SHOW_CREATE_TABLE: `tableName, tableType, ddl`
+///
+/// `dryRun` is emitted only for operations that have dry-run semantics (CREATE, DROP); it is
+/// absent from SHOW responses where the concept does not apply.
 @JsonInclude(JsonInclude.Include.NON_NULL)
 public class DdlExecutionResponse {
   private DdlOperation _operation;
-  private boolean _dryRun;
+  /// Boxed Boolean so that CREATE / DROP responses can carry the dry-run flag while SHOW
+  /// TABLES and SHOW CREATE TABLE responses elide it via {@link JsonInclude.Include#NON_NULL}
+  /// — those operations have no dry-run semantics and a serialized `"dryRun": false` on them
+  /// is meaningless to the caller.
+  private Boolean _dryRun;
   private String _databaseName;
   private String _tableName;
   private String _tableType;
@@ -60,7 +67,7 @@ public DdlExecutionResponse setOperation(DdlOperation operation) {
     return this;
   }
 
-  public boolean isDryRun() {
+  public Boolean isDryRun() {
     return _dryRun;
   }
 

From e59e49c90477a3913c4d8085a4ae9addf0191c64 Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Fri, 15 May 2026 01:28:11 -0700
Subject: [PATCH 31/32] Add cluster-level integration test for TIMESTAMP DDL
 time column
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Before this commit, TIMESTAMP-as-time-column was covered at the
parser / compiler / emitter / round-trip layers but never at the
controller integration layer — the in-process layers exercise
DdlCompiler and CanonicalDdlEmitter in isolation but skip
PinotHelixResourceManager.addTable / addSchema and the validation
pipeline.

Add createWithTimestampTimeColumnRoundTrips to
PinotDdlRestletResourceTest that:
1. Issues CREATE TABLE with `ts TIMESTAMP DATETIME FORMAT
   '1:MILLISECONDS:TIMESTAMP' GRANULARITY '1:MILLISECONDS'`
   against a real ControllerTest cluster.
2. Verifies the persisted Schema's DateTimeFieldSpec normalizes the
   user-supplied format to the short token "TIMESTAMP" (the
   DateTimeFieldSpec rewrite the JSON API also performs).
3. Calls SHOW CREATE TABLE and asserts the emitted DDL renders
   "ts TIMESTAMP DATETIME FORMAT 'TIMESTAMP' GRANULARITY
   '1:MILLISECONDS'" — closing the round-trip loop at the wire
   level, not just the in-process emitter level.

Cleans up via DROP TABLE so subsequent tests are unaffected.

Tests: 18 integration tests pass (was 17).

Co-Authored-By: Claude Opus 4.7 
---
 .../api/PinotDdlRestletResourceTest.java      | 46 +++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
index 7a8b7e449003..9dc5fd99269f 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotDdlRestletResourceTest.java
@@ -209,6 +209,52 @@ public void showCreateRendersCanonicalDdlAndRoundTrips()
         "Properties not in lex order:\n" + ddl);
   }
 
+  /// Cluster-level regression for the recommended TIMESTAMP-typed time column: CREATE persists,
+  /// SHOW CREATE renders the canonical short-form FORMAT 'TIMESTAMP' (DateTimeFieldSpec normalizes
+  /// the format token implicitly for TIMESTAMP), and the re-issued DDL compiles on the same
+  /// controller. This pins the end-to-end path against PinotHelixResourceManager.addTable /
+  /// addSchema, the validation pipeline, and the reverse emitter — the layers that the
+  /// in-process compiler/emitter unit tests do not exercise.
+  @Test
+  public void createWithTimestampTimeColumnRoundTrips()
+      throws IOException {
+    String tbl = "ddlTimestampTimeRoundtrip";
+    String createSql = "CREATE TABLE " + tbl + " ("
+        + "  id INT NOT NULL DIMENSION,"
+        + "  ts TIMESTAMP DATETIME FORMAT '1:MILLISECONDS:TIMESTAMP' GRANULARITY '1:MILLISECONDS'"
+        + ") TABLE_TYPE = OFFLINE PROPERTIES ("
+        + "  'timeColumnName' = 'ts',"
+        + "  'replication' = '1'"
+        + ")";
+    JsonNode createResp = postDdl(createSql, false);
+    assertEquals(createResp.get("operation").asText(), "CREATE_TABLE");
+    assertEquals(createResp.get("tableName").asText(), tbl + "_OFFLINE");
+    // The stored Schema must carry the normalized TIMESTAMP format token, not the verbose
+    // 1:MILLISECONDS:TIMESTAMP form the user typed — DateTimeFieldSpec rewrites it because the
+    // format is implicit for TIMESTAMP.
+    JsonNode schema = createResp.get("schema");
+    JsonNode tsField = null;
+    for (JsonNode f : schema.get("dateTimeFieldSpecs")) {
+      if ("ts".equals(f.get("name").asText())) {
+        tsField = f;
+        break;
+      }
+    }
+    assertNotNull(tsField, "Schema must declare ts as a DateTimeFieldSpec: " + schema);
+    assertEquals(tsField.get("dataType").asText(), "TIMESTAMP");
+    assertEquals(tsField.get("format").asText(), "TIMESTAMP",
+        "DateTimeFieldSpec must normalize TIMESTAMP format to short form; got " + tsField);
+    assertEquals(tsField.get("granularity").asText(), "1:MILLISECONDS");
+
+    // SHOW CREATE TABLE must render the short form and be idempotent on re-compile.
+    JsonNode showResp = postDdl("SHOW CREATE TABLE " + tbl, false);
+    String ddl = showResp.get("ddl").asText();
+    assertTrue(ddl.contains("ts TIMESTAMP DATETIME FORMAT 'TIMESTAMP' GRANULARITY '1:MILLISECONDS'"),
+        "Canonical DDL must emit TIMESTAMP short-form format; got:\n" + ddl);
+    // DROP cleans up so the next test starts fresh.
+    postDdl("DROP TABLE " + tbl, false);
+  }
+
   @Test
   public void showCreateOnMissingTableReturns404()
       throws IOException {

From 11dd97255acf74aaa26356221da35592f5ab642c Mon Sep 17 00:00:00 2001
From: Xiang Fu 
Date: Fri, 15 May 2026 16:21:38 -0700
Subject: [PATCH 32/32] Document BOOLEAN silent-coercion semantics in DDL
 DEFAULT validation
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

DataType.BOOLEAN.convert("garbage") returns 0 via BooleanUtils.toInt
rather than throwing — non-true/false strings map silently to false.
This matches the JSON /tables endpoint with
`"defaultNullValue": ""` so DDL behavior is consistent with
the rest of Pinot, but the silent coercion is non-obvious to a reader
of DdlCompiler.toFieldSpec who would expect convert() to throw the
same way INT does for non-numeric strings.

Add a "Caveat for BOOLEAN" note where the validation lives so the
behavior is locally discoverable. No semantic change.

Co-Authored-By: Claude Opus 4.7 
---
 .../java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java  | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
index cc5181dc4857..a425f5e70999 100644
--- a/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
+++ b/pinot-sql-ddl/src/main/java/org/apache/pinot/sql/ddl/compile/DdlCompiler.java
@@ -340,6 +340,12 @@ private static FieldSpec toFieldSpec(ResolvedColumnDefinition col) {
       // the string lazily; without this check, "INT col DEFAULT 'abc'" would compile cleanly
       // and then fail at first ingestion with a less-specific error from the segment generator.
       // Failing here gives the user a 400 with the column name and the offending literal.
+      //
+      // Caveat for BOOLEAN: DataType.BOOLEAN.convert delegates to BooleanUtils.toInt which maps
+      // any non-true/false string to 0 (false) silently rather than throwing. This matches what
+      // happens for the JSON /tables endpoint with `"defaultNullValue": ""`, so DDL
+      // behavior is consistent with the rest of Pinot. A user writing
+      // `BOOLEAN col DEFAULT 'maybe'` will see no compile error and the column will ingest 0.
       try {
         col.getDataType().convert(col.getDefaultValue());
       } catch (RuntimeException e) {