This library leverages Pydantic to simplify the generation of Databricks SQL queries for managing tables. It provides a convenient way to define table schemas using Pydantic models and generates corresponding SQL statements.
pip install pydantic-databricks
from pydantic_databricks import DatabricksModel
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
class Schema(DatabricksModel):
_table_name = "test"
_schema_name = "test_schema"
col1: str
col2: int
col3: float
spark.sql(Schema.create_table())
Generated SQL:
CREATE TABLE test_schema.test (col1 STRING NOT NULL, col2 BIGINT NOT NULL, col3 DOUBLE NOT NULL) USING DELTA;
from pydantic_databricks import DatabricksModel, Grant, GrantAction
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
class Schema(DatabricksModel):
_table_name = "test"
_schema_name = "default"
_grants = {Grant(action=GrantAction.MODIFY, principal="user1"),
Grant(action=GrantAction.SELECT, principal="user2"), }
col1: str
for grant in Schema.grant_statements():
spark.sql(grant)
_catalog_name
: The catalog name for the table. Default isNone
. IfNone
then a two part namespace is used._schema_name
: The schema name for the table (required)._table_name
: The name of the table (required)._grants
: A set of Grant objects. Default isNone
._location_prefix
: The location prefix for external tables. Default isNone
. If set, then the table is created as an external table. The prefix will be appended with the full table name._table_properties
: A dictionary of table properties._table_create_mode
: The mode for table creation. Default isCreateMode.CREATE
._table_data_source
: The data source for the table. Default isDataSource.DELTA
._partition_columns
: A list of partition columns for the table. Default isNone
._options
: A dictionary of additional options for table creation. Default isNone
._comment
: A comment for the table. Default isNone
.
- Support for table and column constraints
We welcome contributions to pydantic-databricks! If you'd like to contribute, please follow these steps:
- Fork the repository.
- Create a branch for your feature or bug fix.
- Make your changes.
- Test your changes thoroughly.
- Submit a pull request.
- pydantic-databricks is licensed under the MIT License. See LICENSE for more information