azure-databricks

Azure Databricks is a managed Spark offering on Azure and customers already use it for advanced analytics. It provides a collaborative Notebook based environment with CPU or GPU based compute cluster.

In this section, you will find sample notebooks on how to use Azure Machine Learning SDK with Azure Databricks. You can train a model using Spark MLlib and then deploy the model to ACI/AKS from within Azure Databricks. You can also use Automated ML capability (public preview) of Azure ML SDK with Azure Databricks.

• Customers who use Azure Databricks for advanced analytics can now use the same cluster to run experiments with or without automated machine learning.
• You can keep the data within the same cluster.
• You can leverage the local worker nodes with autoscale and auto termination capabilities.
• You can use multiple cores of your Azure Databricks cluster to perform simultenous training.
• You can further tune the model generated by automated machine learning if you chose to.
• Every run (including the best run) is available as a pipeline, which you can tune further if needed.
• The model trained using Azure Databricks can be registered in Azure ML SDK workspace and then deployed to Azure managed compute (ACI or AKS) using the Azure Machine learning SDK.

Please follow our Azure doc to install the sdk in your Azure Databricks cluster before trying any of the sample notebooks.

Single file - The following archive contains all the sample notebooks. You can the run notebooks after importing DBC in your Databricks workspace instead of downloading individually.

Notebooks 1-4 have to be run sequentially & are related to Income prediction experiment based on this dataset and demonstrate how to data prep, train and operationalize a Spark ML model with Azure ML Python SDK from within Azure Databricks.

Notebook 6 is an Automated ML sample notebook for Classification.

Learn more about how to use Azure Databricks as a development environment for Azure Machine Learning service.

Databricks as a Compute Target from Azure ML Pipelines You can use Azure Databricks as a compute target from Azure Machine Learning Pipelines. Take a look at this notebook for details: aml-pipelines-use-databricks-as-compute-target.ipynb.

Linked Azure Databricks and Azure Machine Learning Workspaces (Preview)

Customers can now link Azure Databricks and AzureML Workspaces to better enable cross-Azure ML scenarios by managing their tracking data in a single place when using the MLflow client - the Azure ML workspace.

Linking the Workspaces (Admin operation)

1. The Azure Databricks Azure portal blade now includes a new button to link an Azure ML workspace.
2. Both a new or existing Azure ML Workspace can be linked in the resulting prompt. Follow any instructions to set up the Azure ML Workspace.
3. After a successful link operation, you should see the Azure Databricks overview reflect the linked status

Configure MLflow to send data to Azure ML (All roles)

1. Add azureml-mlflow as a library to any notebook or cluster that should send data to Azure ML. You can do this via:
   1. DBUtils

      dbutils.library.installPyPI("azureml-mlflow")
      dbutils.library.restartPython()  # Removes Python state
      

   2. Cluster Libraries
2. Set the MLflow tracking URI to the following scheme:

   adbazureml://${azuremlRegion}.experiments.azureml.net/history/v1.0/subscriptions/${azuremlSubscriptionId}/resourceGroups/${azuremlResourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/${azuremlWorkspaceName}
   

   1. You can automatically configure this on your clusters for all subsequent notebook sessions using this helper script instead of manually setting the tracking URI in the notebook:
      • AzureML Tracking Cluster Init Script
3. If configured correctly, you'll now be able to see your MLflow tracking data in both Azure ML (via the REST API and all clients) and Azure Databricks (in the MLflow UI and using the MLflow client)

Known Preview Limitations

While we roll this experience out to customers for feedback, there are some known limitations we'd love comments on in addition to any other issues seen in your workflow.

1-to-1 Workspace linking

Currently, an Azure ML Workspace can only be linked to one Azure Databricks Workspace at a time.

Data synchronization

At the moment, data is only generated in the Azure Machine Learning workspace for tracking. Editing tags via the Azure Databricks MLflow UI won't be reflected in the Azure ML UI.

Java and R support

The experience currently is only available from the Python MLflow client.

For more on SDK concepts, please refer to notebooks.

Please let us know your feedback.