Skip to content

CREATE TABLE support for existing drivers#195

Merged
jogrogan merged 4 commits intomainfrom
jogrogan/create
Mar 3, 2026
Merged

CREATE TABLE support for existing drivers#195
jogrogan merged 4 commits intomainfrom
jogrogan/create

Conversation

@jogrogan
Copy link
Collaborator

@jogrogan jogrogan commented Mar 3, 2026

This change adds CREATE TABLE support for all existing Drivers with some shared utilities. Each module now includes the ability to create and delete tables as well as incorporates custom validation checks depending on Driver type.

  1. Support new !describe command in Quidem to list out table schemas. This is similar to !describe in Sqlline or select ... from metadata."COLUMNS" but these both require loading all drivers then reconciling the query which the new !describe in Quidem doesn't do.
  2. Add support for CREATE TABLE for KafkaDriver topics with validation checks
  3. Refactor Venice CREATE TABLE + add validation checks
  4. Add support for CREATE TABLE for catalog level schemas (CATALOG.SCHEMA.TABLE semantics)
  5. Add support for CREATE TABLE for MySQL databases and tables
  6. Adds lots of tests

Copy link
Collaborator

@ryannedolan ryannedolan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Seems like we could get rid of nearly all of the controllers at this point.

return null;
}

HoptimatorJdbcSchema schema = subSchemaPlus.unwrap(HoptimatorJdbcSchema.class);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should add a properties field to HoptimatorJdbcSchema at some point.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea we could, doesn't really save us much besides maybe some weird DataSource URL unwrapping. Ideally we'd make HoptimatorJdbcSchema aware of which underlying Driver a table came from (e.g. KafkaDriver) then we'd be able to simplify a lot of this logic to just check table types and fetch properties associated with a table instead of reconstructing a lot of it like we need to do here in each Deployer.

/**
* Deployer for Kafka topics. Creates topics in the synchronous DDL hot path.
*
* <p>Implements {@link Validated} to pre-check partition constraints
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat!

@jogrogan jogrogan merged commit 7a74bde into main Mar 3, 2026
1 check passed
@jogrogan jogrogan deleted the jogrogan/create branch March 3, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants