Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,15 @@
- `<token>` (_required_ for PAT authentication): For Databricks personal access token (PAT) authentication, the target Databricks user's PAT value.
- `<client-id>` and `<client-secret>` (_required_ for OAuth authentication): For Databricks OAuth machine-to-machine (M2M) authentication, the Databricks managed service principal's **UUID** (or **Client ID** or **Application ID**) and OAuth **Secret** (client secret) values.
- `<catalog>` (_required_): The name of the catalog in Unity Catalog for the target volume and table in the Databricks workspace.
- `<database>`: The name of the database in Unity Catalog for the target volume and table. The default is `default` if not otherwise specified.
- `<database>`: The name of the schema (formerly known as a database) in Unity Catalog for the target table. The default is `default` if not otherwise specified.

If the target table and volume are in the same schema (formerly known as a database), then `<database>` and `<schema>` will have the same values.

- `<table-name>` (_required_): The name of the target table in Unity Catalog.
- `<schema>`: The name of the schema in Unity Catalog for the target volume and table. The default is `default` if not otherwise specified.
- `<schema>`: The name of the schema (formerly known as a database) in Unity Catalog for the target volume. The default is `default` if not otherwise specified.

If the target volume and table are in the same schema (formerly known as a database), then `<schema>` and `<database>` will have the same values.

- `<volume>` (_required_): The name of the target volume in Unity Catalog.
- `<volume-path>`: Any target folder path inside of the volume to use instead of the volume's root. If not otherwise specified, processing occurs at the volume's root.

Expand Down
15 changes: 10 additions & 5 deletions snippets/general-shared-text/databricks-delta-table-cli-api.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,11 @@ The following environment variables:
- `DATABRICKS_CLIENT_ID` - For Databricks managed service principal authenticaton, the service principal's **UUID** (or **Client ID** or **Application ID**) value, represented by `--client-id` (CLI) or `client_id` (Python).
- `DATABRICKS_CLIENT_SECRET` - For Databricks managed service principal authenticaton, the service principal's OAuth **Secret** value, represented by `--client-secret` (CLI) or `client_secret` (Python).
- `DATABRICKS_CATALOG` - The name of the catalog in Unity Catalog, represented by `--catalog` (CLI) or `catalog` (Python).
- `DATABRICKS_DATABASE` - The name of the schema (database) inside of the catalog, represented by `--database` (CLI) or `database` (Python). The default is `default` if not otherwise specified.
- `DATABRICKS_TABLE` - The name of the table inside of the schema (database), represented by `--table-name` (CLI) or `table_name` (Python). The default is `elements` if not otherwise specified.
- `DATABRICKS_DATABASE` - The name of the schema (formerly known as a database) inside of the catalog for the target table, represented by `--database` (CLI) or `database` (Python). The default is `default` if not otherwise specified.

If you are also using a volume, and the target table and volume are in the same schema (formerly known as a database), then `DATABRICKS_DATABASE` and `DATABRICKS_SCHEMA` will have the same values.

- `DATABRICKS_TABLE` - The name of the table inside of the schema (formerly known as a database), represented by `--table-name` (CLI) or `table_name` (Python). The default is `elements` if not otherwise specified.

For the SQL-based implementation, add these environment variables:

Expand All @@ -26,7 +29,9 @@ For the SQL-based implementation, add these environment variables:

For the volume-based implementation, add these environment variables:

- `DATABRICKS_SCHEMA` - The name of the schema (database) inside of the catalog, represented by `--schema` (CLI) or `schema` (Python). This name of this database (schema) must be the same as
the value of the `DATABRICKS_DATABASE` environment variable and is required for compatiblity. The default is `default` if not otherwise specified.
- `DATABRICKS_VOLUME` - The name of the volume inside of the schema (database), represented by `--volume` (CLI) or `volume` (Python).
- `DATABRICKS_SCHEMA` - The name of the schema (formerly known as a database) inside of the catalog for the target volume, represented by `--schema` (CLI) or `schema` (Python). The default is `default` if not otherwise specified.

If the target volume and table are in the same schema (formerly known as a database), then `DATABRICKS_SCHEMA` and `DATABRICKS_SCHEMA` will have the same values.

- `DATABRICKS_VOLUME` - The name of the volume inside of the schema (formerly known as a database), represented by `--volume` (CLI) or `volume` (Python).
- `DATABRICKS_VOLUME_PATH` - Optionally, a specific path inside of the volume that you want to start accessing from, starting from the volume's root, represented by `--volume-path` (CLI) or `volume_path` (Python). The default is to start accessing from the volume's root if not otherwise specified.
10 changes: 8 additions & 2 deletions snippets/general-shared-text/databricks-delta-table-platform.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,14 @@ Fill in the following fields:
- **Token** (_required_ for PAT authentication): For Databricks personal access token (PAT) authentication, the target Databricks user's PAT value.
- **UUID** and **OAuth Secret** (_required_ for OAuth authentication): For Databricks OAuth machine-to-machine (M2M) authentication, the Databricks managed service principal's **UUID** (or **Client ID** or **Application ID**) and OAuth **Secret** (client secret) values.
- **Catalog** (_required_): The name of the catalog in Unity Catalog for the target volume and table in the Databricks workspace.
- **Database**: The name of the database in Unity Catalog for the target volume and table. The default is `default` if not otherwise specified.
- **Database**: The name of the schema (formerly known as a database) in Unity Catalog for the target table. The default is `default` if not otherwise specified.

If the target table and volume are in the same schema (formerly known as a database), then **Database** and **Schema** will have the same names.

- **Table Name** (_required_): The name of the target table in Unity Catalog.
- **Schema**: The name of the schema in Unity Catalog for the target volume and table. The default is `default` if not otherwise specified.
- **Schema**: The name of the schema (formerly known as a database) in Unity Catalog for the target volume. The default is `default` if not otherwise specified.

If the target volume and table are in the same schema (formerly known as a database), then **Schema** and **Database** will have the same names.

- **Volume** (_required_): The name of the target volume in Unity Catalog.
- **Volume Path**: Any target folder path inside of the volume to use instead of the volume's root. If not otherwise specified, processing occurs at the volume's root.
112 changes: 84 additions & 28 deletions snippets/general-shared-text/databricks-delta-table.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,35 @@
- A SQL warehouse for [AWS](https://docs.databricks.com/compute/sql-warehouse/create.html),
[Azure](https://learn.microsoft.com/azure/databricks/compute/sql-warehouse/create), or
[GCP](https://docs.gcp.databricks.com/compute/sql-warehouse/create.html).

The following video shows how to create a SQL warehouse if you do not already have one available, get its **Server Hostname** and **HTTP Path** values, and set permissions for someone other than the warehouse's owner to use it:

<iframe
width="560"
height="315"
src="https://www.youtube.com/embed/N-Aw9-U3_fE"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
></iframe>

- An all-purpose cluster for [AWS](https://docs.databricks.com/compute/use-compute.html),
[Azure](https://learn.microsoft.com/azure/databricks/compute/use-compute), or
[GCP](https://docs.gcp.databricks.com/compute/use-compute.html).

The following video shows how to create an all-purpose cluster if you do not already have one available, get its **Server Hostname** and **HTTP Path** values, and set permissions for someone other than the cluster's owner to use it:

<iframe
width="560"
height="315"
src="https://www.youtube.com/embed/apgibaelVY0"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
></iframe>

- The SQL warehouse's or cluster's **Server Hostname** and **HTTP Path** values for [AWS](https://docs.databricks.com/integrations/compute-details.html),
[Azure](https://learn.microsoft.com/azure/databricks/integrations/compute-details), or
[GCP](https://docs.gcp.databricks.com/integrations/compute-details.html).
Expand All @@ -25,7 +50,7 @@
for [AWS](https://docs.databricks.com/catalogs/create-catalog.html),
[Azure](https://learn.microsoft.com/azure/databricks/catalogs/create-catalog), or
[GCP](https://docs.gcp.databricks.com/catalogs/create-catalog.html).
- A schema
- A schema (formerly known as a database)
for [AWS](https://docs.databricks.com/schemas/create-schema.html),
[Azure](https://learn.microsoft.com/azure/databricks/schemas/create-schema), or
[GCP](https://docs.gcp.databricks.com/schemas/create-schema.html)
Expand All @@ -34,7 +59,19 @@
for [AWS](https://docs.databricks.com/tables/managed.html),
[Azure](https://learn.microsoft.com/azure/databricks/tables/managed), or
[GCP](https://docs.gcp.databricks.com/tables/managed.html)
within that schema.
within that schema (formerly known as a database).

The following video shows how to create a catalog, schema (formerly known as a database), and a table in Unity Catalog if you do not already have them available, and set privileges for someone other than their owner to use them:

<iframe
width="560"
height="315"
src="https://www.youtube.com/embed/ffNnq-6bpd4"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
></iframe>

This table must contain the following column names and their data types:

Expand Down Expand Up @@ -86,22 +123,33 @@
);
```

<Info>
In Databricks, a table's _schema_ is different than a _schema_ (formerly known as a database) in a catalog-schema object relationship in Unity Catalog.
</Info>

- Within Unity Catalog, a volume
for [AWS](https://docs.databricks.com/volumes/utility-commands.html),
[Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands),
or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html)
within the same schema as the table.
- For Databricks personal access token authentication to the workspace, the
Databricks personal access token value for
[AWS](https://docs.databricks.com/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users),
[Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat#azure-databricks-personal-access-tokens-for-workspace-users), or
[GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users).
This token must be for the workspace user who
has the appropriate access permissions to the catalog, schema, table, volume, and cluster or SQL warehouse,
or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html). The volume can be in the same
schema (formerly known as a database) as the table, or the volume and table can be in separate schemas. In either case, both of these
schemas must share the same parent catalog.

The following video shows how to create a catalog, schema (formerly known as a database), and a volume in Unity Catalog if you do not already have them available, and set privileges for someone other than their owner to use them:

<iframe
width="560"
height="315"
src="https://www.youtube.com/embed/yF9DJphhQQc"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
></iframe>

- For Databricks managed service principal authentication (using Databricks OAuth M2M) to the workspace:

- A Databricks managed service principal.
This service principal must have the appropriate access permissions to the catalog, schema, table, volume, and cluster or SQL warehouse.
This service principal must have the appropriate access permissions to the catalog, schema (formerly known as a database), table, volume, and cluster or SQL warehouse.
- The service principal's **UUID** (or **Client ID** or **Application ID**) value.
- The OAuth **Secret** value for the service principal.

Expand All @@ -110,11 +158,11 @@
[GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

<Note>
For Azure Databricks, this connector only supports Databricks managed service principals.
For Azure Databricks, this connector only supports Databricks managed service principals for authentication.
Microsoft Entra ID managed service principals are not supported.
</Note>

The following video shows how to create a Databricks managed service principal:
The following video shows how to create a Databricks managed service principal if you do not already have one available:

<iframe
width="560"
Expand All @@ -126,6 +174,26 @@
allowfullscreen
></iframe>

- For Databricks personal access token authentication to the workspace, the
Databricks personal access token value for
[AWS](https://docs.databricks.com/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users),
[Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat#azure-databricks-personal-access-tokens-for-workspace-users), or
[GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users).
This token must be for the workspace user who
has the appropriate access permissions to the catalog, schema (formerly known as a database), table, volume, and cluster or SQL warehouse,

The following video shows how to create a Databricks personal access token if you do not already have one available:

<iframe
width="560"
height="315"
src="https://www.youtube.com/embed/OzEU2miAS6I"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
></iframe>

- The Databricks workspace user or Databricks managed service principal must have the following _minimum_ set of permissions and privileges to write to an
existing volume or table in Unity Catalog:

Expand All @@ -140,30 +208,18 @@
- To access a Unity Catalog volume, the following privileges:

- `USE CATALOG` on the volume's parent catalog in Unity Catalog.
- `USE SCHEMA` on the volume's parent schema in Unity Catalog.
- `USE SCHEMA` on the volume's parent schema (formerly known as a database) in Unity Catalog.
- `READ VOLUME` and `WRITE VOLUME` on the volume.

Learn how to check and set Unity Catalog privileges for
[AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
[Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
[GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

The following videos shows how to grant a Databricks managed service principal privileges to a Unity Catalog volume:

<iframe
width="560"
height="315"
src="https://www.youtube.com/embed/DykQRxgh2aQ"
title="YouTube video player"
frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen
></iframe>

- To access a Unity Catalog table, the following privileges:

- `USE CATALOG` on the table's parent catalog in Unity Catalog.
- `USE SCHEMA` on the tables's parent schema in Unity Catalog.
- `USE SCHEMA` on the tables's parent schema (formerly known as a database) in Unity Catalog.
- `MODIFY` and `SELECT` on the table.

Learn how to check and set Unity Catalog privileges for
Expand Down
Loading