Add SQL DDL support for CREATE/DROP TABLE and SHOW CREATE TABLE

[xiangfu0]

Summary

Adds a new SQL DDL surface to Pinot — CREATE TABLE, DROP TABLE, SHOW TABLES, SHOW CREATE TABLE — exposed via a single controller endpoint POST /sql/ddl. DDL statements compile into the existing Schema + TableConfig pair and persist through the same PinotHelixResourceManager paths as the JSON /tables and /schemas endpoints, sharing the validation pipeline (TableConfigValidationUtils) so DDL-created tables are subject to the same min-replica, storage-quota, hybrid-pair, instance-assignment, and tenant-tag checks as JSON-API tables.

The implementation is split into three layers:

• Parser (pinot-common): Calcite FMPP/JavaCC grammar additions + 6 AST node classes.
• Compiler (pinot-sql-ddl — new module): DdlCompiler translates AST → (Schema, TableConfig); CanonicalDdlEmitter does the reverse for SHOW CREATE TABLE. Round-trip is idempotent.
• REST endpoint (pinot-controller): POST /sql/ddl dispatches by operation, applies tuner configs before validation, supports dryRun=true for compile-without-persist.

---

Quickstart examples

Create an OFFLINE table

CREATE TABLE events (
  id    INT NOT NULL DIMENSION,
  city  STRING DIMENSION,
  amount DOUBLE METRIC,
  ts    TIMESTAMP DATETIME FORMAT 'TIMESTAMP' GRANULARITY '1:MILLISECONDS'
)
TABLE_TYPE = OFFLINE
PROPERTIES (
  'timeColumnName' = 'ts',
  'replication' = '3',
  'brokerTenant' = 'DefaultTenant',
  'serverTenant' = 'DefaultTenant'
);

TIMESTAMP is the recommended type for new time columns; LONG DATETIME FORMAT '1:MILLISECONDS:EPOCH' is still accepted for tables migrating from existing JSON configs. For a TIMESTAMP-typed column, Pinot's DateTimeFieldSpec treats the format as implicit and stores it as the short token 'TIMESTAMP' regardless of what the user supplied — the canonical DDL emitted by SHOW CREATE TABLE reflects that.

Create a REALTIME table with Kafka stream config

CREATE TABLE clicks (
  user_id STRING DIMENSION,
  url     STRING DIMENSION,
  ts      TIMESTAMP DATETIME FORMAT 'TIMESTAMP' GRANULARITY '1:MILLISECONDS'
)
TABLE_TYPE = REALTIME
PROPERTIES (
  'timeColumnName' = 'ts',
  'replication' = '2',
  'stream.kafka.topic.name' = 'click_events',
  'stream.kafka.consumer.factory.class.name' = 'org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory',
  'stream.kafka.broker.list' = 'kafka-broker:9092',
  'realtime.segment.flush.threshold.rows' = '500000'
);

Multi-value (array) dimension

CREATE TABLE products (
  id   INT DIMENSION,
  tags STRING DIMENSION ARRAY
)
TABLE_TYPE = OFFLINE;

Idempotent re-creation with IF NOT EXISTS

CREATE TABLE IF NOT EXISTS events (id INT) TABLE_TYPE = OFFLINE;
-- Returns 200 if the table already exists; 201 on first creation.

Defaults and NOT NULL

CREATE TABLE users (
  id    INT NOT NULL DIMENSION,
  name  STRING NOT NULL DEFAULT 'unknown' DIMENSION,
  score DOUBLE DEFAULT 0.0 METRIC
)
TABLE_TYPE = OFFLINE;

Upsert table with PRIMARY KEY

CREATE TABLE upsertOrders (
  orderId INT NOT NULL DIMENSION,
  userId  STRING NOT NULL DIMENSION,
  amount  DOUBLE METRIC,
  ts      TIMESTAMP DATETIME FORMAT 'TIMESTAMP' GRANULARITY '1:MILLISECONDS'
) PRIMARY KEY (orderId)
TABLE_TYPE = REALTIME
PROPERTIES (
  'timeColumnName' = 'ts',
  'replication' = '2',
  'upsertConfig' = '{"mode":"FULL"}',
  'stream.kafka.topic.name' = 'orders'
);

DB-qualified table

CREATE TABLE analytics.events (id INT) TABLE_TYPE = OFFLINE;
SHOW CREATE TABLE analytics.events;
DROP TABLE analytics.events;

(The Database: HTTP header is also honored; the SQL-side db. qualifier wins when both are present.)

DROP TABLE

DROP TABLE events;                          -- drops both OFFLINE and REALTIME variants
DROP TABLE events TYPE OFFLINE;             -- drops a single variant
DROP TABLE IF EXISTS analytics.events;      -- 200 OK whether the table exists or not

SHOW

SHOW TABLES;                       -- all tables in the database from the Database header
SHOW TABLES FROM analytics;        -- override the database
SHOW CREATE TABLE events;          -- canonical DDL for the table
SHOW CREATE TABLE events TYPE REALTIME;

---

REST endpoint contract

All DDL operations are sent to a single endpoint:

POST /sql/ddl[?dryRun=true|false]
Content-Type: application/json
Authorization: <Bearer | Basic …>
Database: <optional database name>

{ "sql": "<single DDL statement>" }

Response codes

Code	Operation	Meaning
200 OK	DROP / SHOW / dry-run / IF [NOT] EXISTS no-op	Operation succeeded; body has the operation outcome.
201 Created	CREATE	Table was created and persisted to ZK.
400 Bad Request	All	Parse error, semantic error, oversize input (>256K chars), unsupported emitter type, or type-incompatible default value.
404 Not Found	DROP / SHOW CREATE	Table or schema not found (DROP without IF EXISTS, or SHOW CREATE on a missing table).
409 Conflict	CREATE / DROP	Duplicate CREATE without IF NOT EXISTS; logical-table reference blocking DROP; race lost to a concurrent writer.
500 Internal Server Error	All	Helix/ZK inconsistency, or controller defect.

Curl example

curl -X POST 'http://controller:9000/sql/ddl' \
  -H 'Content-Type: application/json' \
  -H 'Database: analytics' \
  -d '{"sql": "CREATE TABLE events (id INT) TABLE_TYPE = OFFLINE"}'

Dry-run (compile and validate without persisting)

curl -X POST 'http://controller:9000/sql/ddl?dryRun=true' \
  -H 'Content-Type: application/json' \
  -d '{"sql": "CREATE TABLE events (id INT) TABLE_TYPE = OFFLINE"}'

A dry-run produces the full compiled Schema + TableConfig payload and runs the entire validation pipeline; it persists nothing. Use it to validate DDL before applying.

Response shape (CREATE)

{
  "operation": "CREATE_TABLE",
  "tableName": "analytics.events_OFFLINE",
  "tableType": "OFFLINE",
  "databaseName": "analytics",
  "dryRun": false,
  "ifNotExists": false,
  "schema":      { /* fully-resolved Schema JSON */ },
  "tableConfig": { /* fully-resolved post-tuner TableConfig JSON */ },
  "warnings":    [ /* non-fatal compile-time warnings */ ],
  "message":     "Successfully created table analytics.events_OFFLINE"
}

Field absence semantics: every field uses @JsonInclude(NON_NULL). dryRun is emitted on CREATE and DROP responses (both have working dry-run paths) and elided on SHOW responses where the concept does not apply. ifNotExists, ifExists, tableType, warnings, deletedTables, tableNames, ddl, databaseName are absent when not relevant for the operation.

---

Property routing reference

The DDL PROPERTIES (...) clause is the escape hatch for everything not expressed by first-class column attributes (NOT NULL, DEFAULT, role suffix). Routing rules — applied in order:

Rule	Pattern	Routes to	Examples
1. Promoted scalar	known builder field	TableConfigBuilder.setX(…)	replication, brokerTenant, serverTenant, timeColumnName, retentionTimeUnit, retentionTimeValue, loadMode, sortedColumn, nullHandlingEnabled, aggregateMetrics, peerSegmentDownloadScheme, crypterClassName, segmentVersion, description, tags, isDimTable, replicasPerPartition, invertedIndexColumns, noDictionaryColumns, bloomFilterColumns, rangeIndexColumns, jsonIndexColumns, varLengthDictionaryColumns, onHeapDictionaryColumns, deletedSegmentsRetentionPeriod
2. JSON-blob	known nested config	TableConfigBuilder.setX(JSON.parse(value))	ingestionConfig, upsertConfig, dedupConfig, dimensionTableConfig, routingConfig, queryConfig, quotaConfig, tierConfigs, tunerConfigs, fieldConfigs, instanceAssignmentConfigMap, tagOverrideConfig, replicaGroupStrategyConfig, completionConfig, starTreeIndexConfigs, segmentPartitionConfig, multiColumnTextIndexConfig, jsonIndexConfigs, instancePartitionsMap, segmentAssignmentConfigMap, tableSamplers, tierOverwrites
3. Stream prefix	stream.*, realtime.*	IndexingConfig.streamConfigs (verbatim, prefix preserved). REALTIME-only.	stream.kafka.topic.name, stream.kafka.consumer.factory.class.name, realtime.segment.flush.threshold.rows
4. Task prefix	task.<taskType>.<key>	TableTaskConfig.taskTypeConfigsMap[taskType][key]	task.RealtimeToOfflineSegmentsTask.bucketTimePeriod
5. Custom (fallback)	anything else	TableCustomConfig.customConfigs	ourTeam.flag, internal.config.X

CSV list values (rule 1 with list-typed builder fields like invertedIndexColumns) are split on commas. The free-form pass-through (rules 2-5) is the forward-compat hook: stream, minion-task, and nested-config schemas can evolve without grammar changes.

JSON-blob example: ingestion config

CREATE TABLE events (
  id INT DIMENSION,
  ts TIMESTAMP DATETIME FORMAT 'TIMESTAMP' GRANULARITY '1:MILLISECONDS'
)
TABLE_TYPE = OFFLINE
PROPERTIES (
  'timeColumnName' = 'ts',
  'ingestionConfig' = '{"batchIngestionConfig":{"segmentIngestionType":"APPEND","segmentIngestionFrequency":"DAILY"}}'
);

Task config example

CREATE TABLE events (id INT) TABLE_TYPE = OFFLINE
PROPERTIES (
  'task.RealtimeToOfflineSegmentsTask.bucketTimePeriod' = '1d',
  'task.RealtimeToOfflineSegmentsTask.maxNumRecordsPerSegment' = '5000000',
  'task.SegmentRefreshTask.tableMaxNumTasks' = '5'
);

---

Data types

DDL alias	Pinot type	Notes
INT, INTEGER	INT
BIGINT, LONG	LONG
FLOAT, REAL	FLOAT
DOUBLE	DOUBLE
DECIMAL, NUMERIC, BIG_DECIMAL	BIG_DECIMAL	DECIMAL(p,s) accepted but precision/scale are not enforced — a compile-time warning is emitted.
BOOLEAN	BOOLEAN	Stored as INT 0/1 internally; canonical DDL emits TRUE/FALSE.
TIMESTAMP	TIMESTAMP	Recommended for time columns. Format is implicit ('TIMESTAMP'); defaults emit as quoted UTC ISO-8601 ('2023-11-14T22:13:20Z').
VARCHAR, CHAR, STRING	STRING
VARBINARY, BINARY, BYTES	BYTES	Defaults emit as quoted hex ('deadbeef').
JSON	JSON
SMALLINT, TINYINT	(rejected)	Intentionally rejected to keep room for future narrow integer types.

Column roles: DIMENSION (default), DIMENSION ARRAY (multi-value), METRIC (numeric only), DATETIME FORMAT '<fmt>' GRANULARITY '<gran>'.

---

Authorization

• DDL operations are authorized after database-name translation, against the post-translation db.tableName_TYPE resource. This prevents a Database: header from substituting the resource that auth was checked against (cross-DB privilege escalation).
• DROP TABLE and SHOW CREATE TABLE without a TYPE clause require permission on both OFFLINE and REALTIME variants — the no-fingerprinting contract under per-type ACL plugins. Partial-permission callers must use TYPE OFFLINE | REALTIME to read or drop a single variant.

---

Forward and reverse round-trip

SHOW CREATE TABLE produces a deterministic canonical DDL string with:

• Properties emitted in lexicographic order.
• BIG_DECIMAL defaults via toPlainString() (no scientific notation).
• BOOLEAN defaults as TRUE/FALSE.
• TIMESTAMP defaults as UTC ISO-8601.
• BYTES defaults as quoted hex.
• Identifiers that collide with reserved/data-type/DDL keywords double-quoted (e.g. a column named int or metric is emitted as "int" / "metric").
• Custom-config keys that would shadow promoted scalars or JSON-blob keys are rejected at emit time to prevent silent round-trip corruption.

emit → parse → compile → emit is idempotent: the RoundTripTest suite exercises every routing rule against this property, including the TIMESTAMP-typed time column and the BOOLEAN DEFAULT TRUE round-trip (where Pinot's natural-default elision intentionally omits DEFAULT FALSE).

---

Rolling-upgrade safety

The DDL feature is purely additive. No SPI signature, no enum, no TableConfig/Schema field, no wire format, and no ZK property-store path is renamed or removed. The persisted artifacts produced by POST /sql/ddl are shape-identical to those produced by the existing POST /tables and POST /schemas endpoints — old brokers, servers, and controllers read DDL-created tables exactly as they read JSON-API-created tables.

Concretely:

• Pre-DDL controllers return 404 on POST /sql/ddl. There is no half-state — the endpoint either exists or it does not.
• New SQL keywords (DIMENSION, METRIC, GRANULARITY, OFFLINE, REALTIME, PROPERTIES, TABLES, TABLE_TYPE, IF) are added under nonReservedKeywordsToAdd in config.fmpp. Existing DQL queries that use dimension, metric, format, etc. as identifiers continue to parse on the new binary. Pre-DDL binaries never saw these tokens at all.
• Rollout direction is irrelevant. Upgrading controllers first, brokers first, servers first, or any interleaving all produce the same on-disk representation. Rollback to a pre-DDL controller continues to serve DDL-created tables identically to JSON-API tables.

See pinot-sql-ddl/DESIGN.md §8.1 for the full mixed-version posture.

---

Test plan

• 31 parser tests (PinotDdlParserTest) — all syntax variants, IF NOT EXISTS, IF EXISTS, SHOW CREATE TABLE, SHOW TABLES FROM <db>, keyword-as-identifier, PRIMARY KEY parens regression, malformed-PK error message
• 41 compiler tests (DdlCompilerTest) — every property routing rule, all data type aliases, role inference, type compatibility for DEFAULT, negative cases (DEFAULT NULL, type mismatch, SMALLINT/TINYINT rejection, lowercase TABLE_TYPE, missing PK column), DECIMAL precision warning, stream/realtime/task routing, JSON-blob round-trip
• 21 emitter golden-output tests (CanonicalDdlEmitterTest) — canonical clause order, lexicographic property ordering, BIG_DECIMAL plain-string, BIG_DECIMAL scale-shifted natural-default elision, BOOLEAN TRUE/FALSE, TIMESTAMP UTC ISO-8601, BYTES non-natural-default hex, ComplexFieldSpec/MAP/LIST/STRUCT explicit rejection, custom-config key shadowing rejection
• 24 round-trip tests (RoundTripTest) — emit → parse → compile → emit idempotence across stream configs, task configs, custom configs, ingestion JSON-blob, MV dimensions, primary keys, identifiers needing quoting, kitchen-sink promoted-scalars regression, TIMESTAMP time column, BOOLEAN DEFAULT TRUE (with natural-default elision for FALSE)
• 18 unit tests (PinotDdlRestletResourceUnitTest) — describeColumnShapeMismatch covers data type, field type, single-value, NOT NULL, default-null-value, BYTES content equality, BIG_DECIMAL compareTo equivalence, DATETIME format/granularity
• 17 controller integration tests (PinotDdlRestletResourceTest) — CREATE/DROP/SHOW round-trip on a real ControllerTest cluster, dry-run, IF [NOT] EXISTS, 201/200/404/409/400 codes, DB-qualified DDL, oversize input rejection, parse-error and semantic-error 400s

All tests pass. ./mvnw spotless:apply checkstyle:check license:format license:check clean across pinot-common, pinot-sql-ddl, and pinot-controller.

---

Review history

This branch went through multiple review iterations driven by the code-review-orchestrator and code-reviewer agents across the eight defect domains (config-backcompat, concurrency-state, architecture, performance, correctness-nulls, testing, naming-api, process-scope). Headline issues addressed by regression tests: parser PRIMARY KEY LOOKAHEAD, BYTES typed comparison + emit, BIG_DECIMAL scale-sensitive equality, ComplexFieldSpec silent drop, validation divergence vs POST /tables, hybrid second-variant CREATE schema metadata, TIMESTAMP UTC emission, SHOW CREATE auth fingerprinting, DEFAULT NULL rejection, type-compatible default validation, BOOLEAN default round-trip, and the dryRun wire-shape asymmetry on SHOW responses.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 67.55181% with 501 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.72%. Comparing base (11545db) to head (11dd972).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
...troller/api/resources/PinotDdlRestletResource.java	57.00%	143 Missing and 41 partials ⚠️
.../apache/pinot/sql/ddl/compile/PropertyMapping.java	65.82%	72 Missing and 9 partials ⚠️
...pache/pinot/sql/ddl/reverse/PropertyExtractor.java	65.19%	33 Missing and 38 partials ⚠️
.../pinot/sql/parsers/parser/SqlPinotCreateTable.java	31.91%	32 Missing ⚠️
.../org/apache/pinot/sql/ddl/compile/DdlCompiler.java	84.50%	17 Missing and 14 partials ⚠️
...rg/apache/pinot/sql/ddl/reverse/SchemaEmitter.java	70.32%	14 Missing and 13 partials ⚠️
.../sql/parsers/parser/SqlPinotColumnDeclaration.java	46.34%	22 Missing ⚠️
...che/pinot/sql/ddl/reverse/CanonicalDdlEmitter.java	80.95%	6 Missing and 6 partials ⚠️
...he/pinot/sql/parsers/parser/SqlPinotDropTable.java	47.36%	10 Missing ⚠️
...che/pinot/sql/parsers/parser/SqlPinotProperty.java	46.66%	8 Missing ⚠️
... and 7 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18241      +/-   ##
============================================
+ Coverage     63.70%   63.72%   +0.02%     
- Complexity     1684     1932     +248     
============================================
  Files          3266     3292      +26     
  Lines        199898   201470    +1572     
  Branches      31048    31316     +268     
============================================
+ Hits         127338   128388    +1050     
- Misses        62409    62794     +385     
- Partials      10151    10288     +137     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-21	63.72% <67.55%> (+0.02%)	⬆️
temurin	63.72% <67.55%> (+0.02%)	⬆️
unittests	63.72% <67.55%> (+0.02%)	⬆️
unittests1	55.78% <40.90%> (-0.03%)	⬇️
unittests2	35.24% <67.55%> (+0.27%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Introduces SQL DDL support to Apache Pinot via a single POST /sql/ddl controller endpoint. Adds a new Calcite grammar extension and AST nodes in pinot-common, a new pinot-sql-ddl module that compiles AST → (Schema, TableConfig) and reverse-emits canonical DDL for SHOW CREATE TABLE, and wires the endpoint into pinot-controller reusing the existing validation/persistence pipeline. Existing /sql endpoint is updated to reject DDL with a clear redirect.

Changes:

• New pinot-sql-ddl module with DdlCompiler, PropertyMapping (forward routing of PROPERTIES entries to promoted scalars / stream / task / custom configs / JSON blobs) and CanonicalDdlEmitter (reverse path with shadow-key rejection, deterministic property ordering, BIG_DECIMAL/BYTES/BOOLEAN/TIMESTAMP-safe default emission).
• Grammar/AST additions in pinot-common (config.fmpp, parserImpls.ftl, 6 SqlPinot* nodes) and DDL classification in CalciteSqlParser.
• New PinotDdlRestletResource* controller resource with hybrid-pair schema compatibility check (describeColumnShapeMismatch), DDL-on-/sql rejection in PinotQueryResource, and DTOs DdlExecutionRequest/DdlExecutionResponse.

Reviewed changes

Copilot reviewed 43 out of 43 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
pom.xml, pinot-controller/pom.xml, pinot-sql-ddl/pom.xml	Register the new module and wire it into pinot-controller.
pinot-sql-ddl/README.md, package-info.java	Module overview and exception → HTTP-status contract docs.
pinot-common/src/main/codegen/config.fmpp	Adds DDL keywords as non-reserved tokens and registers the new statement parser methods.
pinot-common/src/main/codegen/includes/parserImpls.ftl	JavaCC productions for CREATE/DROP TABLE, SHOW TABLES, SHOW CREATE TABLE, with PRIMARY KEY, PROPERTIES, and DATETIME FORMAT … GRANULARITY ….
pinot-common/.../sql/parsers/parser/SqlPinot*.java	New AST nodes (CreateTable, DropTable, ShowTables, ShowCreateTable, ColumnDeclaration, Property).
pinot-common/.../CalciteSqlParser.java	Recognizes the new SqlPinot* nodes and classifies them as PinotSqlType.DDL; touches up some Javadoc to /// style.
pinot-common/.../PinotDdlParserTest.java	31 parser tests (syntax, IF EXISTS variants, identifier-as-keyword, PRIMARY KEY error path).
pinot-sql-ddl/.../compile/*.java	Forward compiler: DdlCompiler, PropertyMapping, DataTypeMapper, exception type, compiled-result hierarchy.
pinot-sql-ddl/.../resolved/*.java	Immutable typed IR between AST and compiled artifacts.
pinot-sql-ddl/.../reverse/*.java	Reverse emitter: CanonicalDdlEmitter, SchemaEmitter, PropertyExtractor, SqlIdentifiers.
pinot-sql-ddl/src/test/.../{compile,reverse}/*.java	41 compiler and 21 emitter golden-output tests.
pinot-controller/.../ddl/DdlExecutionRequest.java, DdlExecutionResponse.java	REST DTOs with operation-conditional field inclusion.
pinot-controller/.../PinotQueryResource.java	Returns a clear validation error directing DDL callers to /sql/ddl.
pinot-controller/.../PinotQueryResourceTest.java	Three new tests covering the DDL-on-/sql redirect across query-option permutations.
pinot-controller/.../PinotDdlRestletResourceUnitTest.java	18 unit tests for the column-shape comparator and DROP guards (logical-table refs, post-preflight task race, DB-header conflicts).
pinot-controller/.../PinotDdlRestletResourceTest.java	17 end-to-end controller integration tests on a real ControllerTest cluster.

[xiangfu0]

Opened docs PR pinot-contrib/pinot-docs#807 to document the new SQL table DDL endpoint () and its Querying & SQL / controller API docs surfaces.

[xiangfu0]

Opened docs PR pinot-contrib/pinot-docs#807 to document the new SQL table DDL endpoint (POST /sql/ddl) and its Querying & SQL / controller API docs surfaces.