databricks_client

About

A REST client for the Databricks REST API.

This module is a thin layer allowing to build HTTP Requests. It does not expose API operations as distinct methods, but rather exposes generic methods allowing to build API calls.

The Databricks API sometimes returns 200 error codes and HTML content when the request is not properly authenticated. The client intercepts such occurrences (detecting non-JSON returned content) and wraps them into an exception.

This open-source project is not developed by nor affiliated with Databricks.

Installing

pip install databricks_client

Usage

import databricks_client

client = databricks_client.create("https://northeurope.azuredatabricks.net/api/2.0")
client.auth_pat_token(pat_token)
clusters_list = client.get('clusters/list')
for cluster in clusters_list["clusters"]:
    print(cluster)

Usage with Azure Active Directory

Note: Azure AD authentication for Databricks is currently in preview.

The client generates short-lived Azure AD tokens. If you need to use your client for longer than the lifetime (typically 30 minutes), rerun client.auth_azuread periodically.

Azure AD authentication with Azure CLI

Install the Azure CLI.

pip install databricks_client[azurecli]
az login

import databricks_client

client = databricks_client.create("https://northeurope.azuredatabricks.net/api/2.0")
client.auth_azuread("/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-rg/providers/Microsoft.Databricks/workspaces/my-workspace")
# or client.auth_azuread(resource_group="my-rg", workspace_name="my-workspace")
clusters_list = client.get('clusters/list')
for cluster in clusters_list["clusters"]:
    print(cluster)

This is recommended with Azure DevOps Pipelines using the Azure CLI task.

Azure AD authentication with ADAL

pip install databricks_client
pip install adal

import databricks_client
import adal

authority_host_uri = 'https://login.microsoftonline.com'
authority_uri = authority_host_uri + '/' + tenant_id
context = adal.AuthenticationContext(authority_uri)

def token_callback(resource):
    return context.acquire_token_with_client_credentials(resource, client_id, client_secret)["accessToken"]

client = databricks_client.create("https://northeurope.azuredatabricks.net/api/2.0")
client.auth_azuread("/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-rg/providers/Microsoft.Databricks/workspaces/my-workspace", token_callback)
# or client.auth_azuread(resource_group="my-rg", workspace_name="my-workspace", token_callback=token_callback)
clusters_list = client.get('clusters/list')
for cluster in clusters_list["clusters"]:
    print(cluster)

Example usages

Generating a PAT token

response = client.post(
    'token/create',
    json={"lifetime_seconds": 60, "comment": "Unit Test Token"}
)
pat_token = response['token_value']

Uploading a notebook

import base64

with open(notebook_file, "rb") as f:
    file_content = f.read()

client.post(
    'workspace/import',
    json={
        "content": base64.b64encode(file_content).decode('ascii'),
        "path": notebook_path,
        "overwrite": False,
        "language": "PYTHON",
        "format": "SOURCE"
    }
)