Your best companion for upgrading to Unity Catalog. It helps you to upgrade all Databricks workspace assets: Legacy Table ACLs, Entitlements, AWS instance profiles, Clusters, Cluster policies, Instance Pools, Databricks SQL warehouses, Delta Live Tables, Jobs, MLflow experiments, MLflow registry, SQL Dashboards & Queries, SQL Alerts, Token and Password usage permissions that are set on the workspace level, Secret scopes, Notebooks, Directories, Repos, Files.
See contributing instructions to help improve this project.
UCX will guide you, the Databricks customer, through the process of upgrading your account, groups, workspaces, jobs etc. to Unity Catalog.
- The upgrade process will first install code, libraries, and workflows into your workspace.
- After installation, you will run a series of workflows and examine the output.
UCX leverages Databricks Lakehouse platform to upgrade itself. The upgrade process includes creating jobs, notebooks, and deploying code and configuration files.
By running the installation you install the assessment job and several upgrade jobs. The assessment and upgrade jobs are outlined in the custom-generated README.py that is created by the installer.
The custom-generated README.py, config.yaml, and other assets are placed into your Databricks workspace home folder, into a sub-folder named .ucx. See interactive tutorial.
Once the custom Databricks jobs are installed, begin by triggering the assessment job. The assessment job can be found under your workflows or via the active link in the README.py. Once the assessment job is complete, you can review the results in the custom-generated Databricks dashboard (linked to by the custom README.py found in the workspace folder created for you).
You will need an account, unity catalog, and workspace administrative authority to complete the upgrade process. To run the installer, you will need to set up databricks-cli and a credential, following these instructions. Additionally, the interim metadata and config data being processed by UCX will be stored into a Hive Metastore database schema generated at install time.
For questions, troubleshooting or bug fixes, please see your Databricks account team or submit an issue to the Databricks UCX GitHub repo
- Get trained on UC [free instructor-led training 2x week] [full training schedule]
- You will need a desktop computer, running Windows, macOS, or Linux; This computer is used to install the UCX toolkit onto the Databricks workspace, the computer will also need:
- Network access to your Databricks Workspace
- Network access to the Internet to retrieve additional Python packages (e.g. PyYAML, databricks-sdk,...) and access https://github.com
- Python 3.10 or later - Windows instructions
- Databricks CLI with a workspace configuration profile for workspace - instructions
- Your Windows computer will need a shell environment (GitBash or (WSL)
- Within the Databricks Workspace you will need:
- Workspace administrator access permissions
- The ability for the installer to upload Python Wheel files to DBFS and Workspace FileSystem
- A PRO or Serverless SQL Warehouse
- The Assessment workflow will create a legacy "No Isolation Shared" and a legacy "Table ACL" jobs clusters needed to inventory Hive Metastore Table ACLS
- If your Databricks Workspace relies on an external Hive Metastore (such as AWS Glue), make sure to read the External HMS Document.
- A number of commands also require Databricks account administrator access permissions, e.g.
sync-workspace-info - [AWS] [Azure]] [GCP] Account level Identity Setup
- [AWS] [Azure] [GCP] Unity Catalog Metastore Created (per region)
We only support installations and upgrades through Databricks CLI, as UCX requires an installation script run to make sure all the necessary and correct configurations are in place.
Install UCX via Databricks CLI:
databricks labs install ucx
This will start an interactive installer with a number of configuration questions:
- Select a workspace profile that has been defined in
~/.databrickscfg - Provide the name of the inventory database where UCX will store the assessment results. This will be in the workspace
hive_metastore. Defaults toucx - Create a new or select an existing SQL warehouse to run assessment dashboards on. The existing warehouse must be Pro or Serverless.
- Configurations for workspace local groups migration:
- Provide a backup prefix. This is used to rename workspace local groups after they have been migrated. Defaults to
db-temp- - Select a workspace local groups migration strategy. UCX offers matching by name or external ID, using a prefix/suffix, or using regex. See this for more details
- Provide a specific list of workspace local groups (or all groups) to be migrated.
- Provide a backup prefix. This is used to rename workspace local groups after they have been migrated. Defaults to
- Select a Python log level, e.g.
DEBUG,INFO. Defaults toINFO - Provide the level of parallelism, which limit the number of concurrent operation as UCX scans the workspace. Defaults to 8.
- Select whether UCX should connect to the external HMS, if a cluster policy with external HMS is detected. Defaults to no.
After this, UCX will be installed locally and a number of assets will be deployed in the selected workspace. These assets are available under the installation folder, i.e. /Users/<your user>/.ucx/
Verify that UCX is installed
databricks labs installed
Name Description Version
ucx Unity Catalog Migration Toolkit (UCX) <version>
Upgrade UCX via Databricks CLI:
databricks labs upgrade ucx
The prompts will be similar to Installation
Uninstall UCX via Databricks CLI:
databricks labs uninstall ucx
Databricks CLI will confirm a few options:
- Whether you want to remove all ucx artefacts from the workspace as well. Defaults to no.
- Whether you want to delete the inventory database in
hive_metastore. Defaults to no.
After installation, a number of UCX workflows will be available in the workspace. <installation_path>/README contains further instructions and explanations of these workflows.
UCX also provides a number of command line utilities accessible via databricks labs ucx.
The assessment workflow can be triggered using the Databricks UI, or via the command line
databricks labs ucx ensure-assessment-run
After UCX assessment workflow is executed, the assessment dashboard will be populated with findings and common recommendations. This guide talks about them in more details.
Use to identify all instance profiles in the workspace, and map their access to S3 buckets.
Also captures the IAM roles which has UC arn listed, and map their access to S3 buckets
This requires awscli to be installed and configured.
databricks labs ucx principal-prefix-access --aws-profile test-profile
Use to identify all storage account used by tables, identify the relevant Azure service principals and their permissions on each storage account.
This requires azure-cli to be installed and configured.
databricks labs ucx principal-prefix-access --subscription-id test-subscription-id
Use to create a table mapping CSV file, which provides the target mapping for all hive_metastore tables identified by the assessment workflow.
This file can be reviewed offline and later will be used for table migration.
databricks labs ucx table-mapping
When installing UCX across multiple workspaces, users needs to keep UCX configurations in sync. The below commands address that.
Recommended: An account administrator executes sync-workspace-info to upload the current UCX workspace configurations to all workspaces in the account where UCX is installed.
UCX will prompt you to select an account profile that has been defined in ~/.databrickscfg.
databricks labs ucx sync-workspace-info
Not recommended: If an account admin is not available to execute sync-workspace-info, workspace admins can manually upload the current ucx workspace config to specific target workspaces.
UCX will ask to confirm the current workspace name, and the ID & name of the target workspaces
databricks labs ucx manual-workspace-info
Use to validate workspace-level & account-level groups to identify any discrepancies in membership after migration.
databricks labs ucx validate-groups-membership
Please note that all projects in the databrickslabs GitHub account are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS, and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects.
Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo. They will be reviewed as time permits, but there are no formal SLAs for support.