Skip to content

[Hold][WIP] Delta Table source and destination connectors for Platform #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
+33 −0
Closed

[Hold][WIP] Delta Table source and destination connectors for Platform #270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions snippets/general-shared-text/delta-table-platform.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Fill in the following fields:

- **Name** (_required_): A unique name for this connector.
30 changes: 30 additions & 0 deletions snippets/general-shared-text/delta-table.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
The Delta Table prerequisites:

- The path to the Delta Table:

- For AWS, this path takes the format `s3://<bucket-name>/<path-to-table>`
- For Azure, this path takes the format `abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<path-to-table>`
- For GCP, this path takes the format `gs://<bucket-name>/<path-to-table>`

A few ways to get this path include:

- Running the Databricks SQL statement `SHOW CREATE TABLE <table-name>;`. Look for the `Location` field in the output.
This field will contain the path to the Delta Table specified by `<table-name>`.
- Running the Databricks SQL statement `SHOW CREATE TABLE <table-name>`;. This will display the `CREATE TABLE` statement,
which will include the `LOCATION` clause. This clause will contain the path to the Delta Table specified by `<table-name>`.
- Running the following PySpark code in a Databricks notebook. This will print the path to the Delta Table specified by `<table-name>`.

```python
deltaTable = DeltaTable.forName(spark, "<table-name>")
location = deltaTable.location()
print(location)
```

- The storage options to access the Delta Table:

- For AWS, this is the AWS access key ID and its corresponding AWS secret access key, which have access to the backing Amazon S3 bucket.
- For Azure, this is the [SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens),
[access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys), or
[connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account)
for the backing Azure Storage account.
- For GCP, this is the JSON content of a Google Cloud service account key that has the necessary permissions to access the specified Google Cloud bucket. The service account key must have at least the **Storage Object Viewer** role to ensure proper access permissions. [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating). [Assign a role](https://cloud.google.com/storage/docs/access-control/iam).