Skip to content

Latest commit

 

History

History
365 lines (228 loc) · 23.1 KB

how-to-secure-workspace-vnet.md

File metadata and controls

365 lines (228 loc) · 23.1 KB
Raw
title titleSuffix description services ms.service ms.subservice ms.reviewer ms.author author ms.date ms.topic ms.custom
Secure an Azure Machine Learning workspace with virtual networks
Azure Machine Learning
Use an isolated Azure Virtual Network to secure your Azure Machine Learning workspace and associated resources.
machine-learning
machine-learning
enterprise-readiness
larryfr
meerakurup
meerakurup
10/19/2023
how-to
tracking-python, security, cliv2, sdkv2, engagement-fy23, build-2023

Secure an Azure Machine Learning workspace with virtual networks

[!INCLUDE sdk/cli v2]

[!INCLUDE managed-vnet-note]

In this article, you learn how to secure an Azure Machine Learning workspace and its associated resources in an Azure Virtual Network.

This article is part of a series on securing an Azure Machine Learning workflow. See the other articles in this series:

For a tutorial on creating a secure workspace, see Tutorial: Create a secure workspace or Tutorial: Create a secure workspace using a template.

In this article you learn how to enable the following workspaces resources in a virtual network:

[!div class="checklist"]

  • Azure Machine Learning workspace
  • Azure Storage accounts
  • Azure Key Vault
  • Azure Container Registry

Prerequisites

  • Read the Network security overview article to understand common virtual network scenarios and overall virtual network architecture.

  • Read the Azure Machine Learning best practices for enterprise security article to learn about best practices.

  • An existing virtual network and subnet to use with your compute resources.

    [!WARNING] Do not use the 172.17.0.0/16 IP address range for your VNet. This is the default subnet range used by the Docker bridge network, and will result in errors if used for your VNet. Other ranges may also conflict depending on what you want to connect to the virtual network. For example, if you plan to connect your on premises network to the VNet, and your on-premises network also uses the 172.16.0.0/16 range. Ultimately, it is up to you to plan your network infrastructure.

[!INCLUDE network-rbac]

Azure Container Registry

  • Your Azure Container Registry must be Premium version. For more information on upgrading, see Changing SKUs.

  • If your Azure Container Registry uses a private endpoint, we recommend that it be in the same virtual network as the storage account and compute targets used for training or inference. However it can also be in a peered virtual network.

    If it uses a service endpoint, it must be in the same virtual network and subnet as the storage account and compute targets.

  • Your Azure Machine Learning workspace must contain an Azure Machine Learning compute cluster.

Limitations

Azure storage account

  • If you plan to use Azure Machine Learning studio and the storage account is also in the virtual network, there are extra validation requirements:

    • If the storage account uses a service endpoint, the workspace private endpoint and storage service endpoint must be in the same subnet of the virtual network.
    • If the storage account uses a private endpoint, the workspace private endpoint and storage private endpoint must be in the same virtual network. In this case, they can be in different subnets.

Azure Container Instances

When your Azure Machine Learning workspace is configured with a private endpoint, deploying to Azure Container Instances in a virtual network isn't supported. Instead, consider using a Managed online endpoint with network isolation.

Azure Container Registry

When your Azure Machine Learning workspace or any resource is configured with a private endpoint it may be required to setup a user managed compute cluster for AzureML Environment image builds. Default scenario is leveraging serverless compute and currently intended for scenarios with no network restrictions on resources associated with AzureML Workspace.

Important

The compute cluster used to build Docker images needs to be able to access the package repositories that are used to train and deploy your models. You may need to add network security rules that allow access to public repos, use private Python packages, or use custom Docker images (SDK v1) that already include the packages.

Warning

If your Azure Container Registry uses a private endpoint or service endpoint to communicate with the virtual network, you cannot use a managed identity with an Azure Machine Learning compute cluster.

Azure Monitor

Warning

Azure Monitor supports using Azure Private Link to connect to a VNet. However, you must use the open Private Link mode in Azure Monitor. For more information, see Private Link access modes: Private only vs. Open.

Required public internet access

[!INCLUDE machine-learning-required-public-internet-access]

For information on using a firewall solution, see Configure required input and output communication.

Secure the workspace with private endpoint

Azure Private Link lets you connect to your workspace using a private endpoint. The private endpoint is a set of private IP addresses within your virtual network. You can then limit access to your workspace to only occur over the private IP addresses. A private endpoint helps reduce the risk of data exfiltration.

For more information on configuring a private endpoint for your workspace, see How to configure a private endpoint.

Warning

Securing a workspace with private endpoints does not ensure end-to-end security by itself. You must follow the steps in the rest of this article, and the VNet series, to secure individual components of your solution. For example, if you use a private endpoint for the workspace, but your Azure Storage Account is not behind the VNet, traffic between the workspace and storage does not use the VNet for security.

Secure Azure storage accounts

Azure Machine Learning supports storage accounts configured to use either a private endpoint or service endpoint.

Private endpoint

  1. In the Azure portal, select the Azure Storage Account.

  2. Use the information in Use private endpoints for Azure Storage to add private endpoints for the following storage resources:

    :::image type="content" source="./media/how-to-enable-studio-virtual-network/configure-storage-private-endpoint.png" alt-text="Screenshot showing private endpoint configuration page with blob and file options":::

    [!TIP] When configuring a storage account that is not the default storage, select the Target subresource type that corresponds to the storage account you want to add.

  3. After creating the private endpoints for the storage resources, select the Firewalls and virtual networks tab under Networking for the storage account.

  4. Select Selected networks, and then under Resource instances, select Microsoft.MachineLearningServices/Workspace as the Resource type. Select your workspace using Instance name. For more information, see Trusted access based on system-assigned managed identity.

    [!TIP] Alternatively, you can select Allow Azure services on the trusted services list to access this storage account to more broadly allow access from trusted services. For more information, see Configure Azure Storage firewalls and virtual networks.

    :::image type="content" source="./media/how-to-enable-virtual-network/storage-firewalls-and-virtual-networks-no-vnet.png" alt-text="The networking area on the Azure Storage page in the Azure portal when using private endpoint":::

  5. Select Save to save the configuration.

Tip

When using a private endpoint, you can also disable anonymous access. For more information, see disallow anonymous access.

Service endpoint

  1. In the Azure portal, select the Azure Storage Account.

  2. From the Security + networking section on the left of the page, select Networking and then select the Firewalls and virtual networks tab.

  3. Select Selected networks. Under Virtual networks, select the Add existing virtual network link and select the virtual network that your workspace uses.

    [!IMPORTANT] The storage account must be in the same virtual network and subnet as the compute instances or clusters used for training or inference.

  4. Under Resource instances, select Microsoft.MachineLearningServices/Workspace as the Resource type and select your workspace using Instance name. For more information, see Trusted access based on system-assigned managed identity.

    [!TIP] Alternatively, you can select Allow Azure services on the trusted services list to access this storage account to more broadly allow access from trusted services. For more information, see Configure Azure Storage firewalls and virtual networks.

    :::image type="content" source="./media/how-to-enable-virtual-network/storage-firewalls-and-virtual-networks.png" alt-text="The networking area on the Azure Storage page in the Azure portal":::

  5. Select Save to save the configuration.

Tip

When using a service endpoint, you can also disable anonymous access. For more information, see disallow anonymous access.

Secure Azure Key Vault

Azure Machine Learning uses an associated Key Vault instance to store the following credentials:

  • The associated storage account connection string
  • Passwords to Azure Container Repository instances
  • Connection strings to data stores

Azure key vault can be configured to use either a private endpoint or service endpoint. To use Azure Machine Learning experimentation capabilities with Azure Key Vault behind a virtual network, use the following steps:

Tip

We recommend that the key vault be in the same VNet as the workspace, however it can be in a peered VNet.

Private endpoint

For information on using a private endpoint with Azure Key Vault, see Integrate Key Vault with Azure Private Link.

Service endpoint

  1. Go to the Key Vault that's associated with the workspace.

  2. On the Key Vault page, in the left pane, select Networking.

  3. On the Firewalls and virtual networks tab, do the following actions:

    1. Under Allow access from, select Allow public access from specific virtual networks and IP addresses.
    2. Under Virtual networks, select Add a virtual network, Add existing virtual networks, and add the virtual network/subnet where your experimentation compute resides.
    3. Verify that Allow trusted Microsoft services to bypass this firewall is checked, and then select Apply.

    :::image type="content" source="./media/how-to-enable-virtual-network/key-vault-firewalls-and-virtual-networks-page.png" alt-text="The Firewalls and virtual networks section in the Key Vault pane":::

For more information, see Configure Azure Key Vault network settings.

Enable Azure Container Registry (ACR)

Tip

If you did not use an existing Azure Container Registry when creating the workspace, one may not exist. By default, the workspace will not create an ACR instance until it needs one. To force the creation of one, train or deploy a model using your workspace before using the steps in this section.

Azure Container Registry can be configured to use a private endpoint. Use the following steps to configure your workspace to use ACR when it is in the virtual network:

  1. Find the name of the Azure Container Registry for your workspace, using one of the following methods:

    Azure CLI

    [!INCLUDE cli v2]

    If you've installed the Machine Learning extension v2 for Azure CLI, you can use the az ml workspace show command to show the workspace information. The v1 extension doesn't return this information.

    az ml workspace show -n yourworkspacename -g resourcegroupname --query 'container_registry'

    This command returns a value similar to "/subscriptions/{GUID}/resourceGroups/{resourcegroupname}/providers/Microsoft.ContainerRegistry/registries/{ACRname}". The last part of the string is the name of the Azure Container Registry for the workspace.

    Python SDK

    [!INCLUDE sdk v2]

    The following code snippet demonstrates how to get the container registry information using the Azure Machine Learning SDK:

     # import required libraries
 from azure.ai.ml import MLClient
 from azure.identity import DefaultAzureCredential

 subscription_id = "<your subscription ID>"
 resource_group = "<your resource group name>"
 workspace = "<your workspace name>"

 ml_client = MLClient(
     DefaultAzureCredential(), subscription_id, resource_group, workspace
 )
 
 # Get workspace info
 ws=ml_client.workspaces.get(name=workspace)
 print(ws.container_registry)

    This code returns a value similar to "/subscriptions/{GUID}/resourceGroups/{resourcegroupname}/providers/Microsoft.ContainerRegistry/registries/{ACRname}". The last part of the string is the name of the Azure Container Registry for the workspace.

    Portal

    From the overview section of your workspace, the Registry value links to the Azure Container Registry.

    :::image type="content" source="./media/how-to-enable-virtual-network/azure-machine-learning-container-registry.png" alt-text="Azure Container Registry for the workspace" border="true":::

  2. Limit access to your virtual network using the steps in Connect privately to an Azure Container Registry. When adding the virtual network, select the virtual network and subnet for your Azure Machine Learning resources.

  3. Configure the ACR for the workspace to Allow access by trusted services.

  4. Create an Azure Machine Learning compute cluster. This cluster is used to build Docker images when ACR is behind a virtual network. For more information, see Create a compute cluster.

  5. Use one of the following methods to configure the workspace to build Docker images using the compute cluster.

    [!IMPORTANT] The following limitations apply When using a compute cluster for image builds:

    • Only a CPU SKU is supported.
    • If you use a compute cluster configured for no public IP address, you must provide some way for the cluster to access the public internet. Internet access is required when accessing images stored on the Microsoft Container Registry, packages installed on Pypi, Conda, etc. You need to configure User Defined Routing (UDR) to reach to a public IP to access the internet. For example, you can use the public IP of your firewall, or you can use Virtual Network NAT with a public IP. For more information, see How to securely train in a VNet.

    Azure CLI

    You can use the az ml workspace update command to set a build compute. The command is the same for both the v1 and v2 Azure CLI extensions for machine learning. In the following command, replace myworkspace with your workspace name, myresourcegroup with the resource group that contains the workspace, and mycomputecluster with the compute cluster name:

    az ml workspace update --name myworkspace --resource-group myresourcegroup --image-build-compute mycomputecluster

    You can switch back to serverless compute by executing the same command and referencing the compute as an empty space: --image-build-compute ' '.

    Python SDK

    The following code snippet demonstrates how to update the workspace to set a build compute using the Azure Machine Learning SDK. Replace mycomputecluster with the name of the cluster to use:

    [!INCLUDE sdk v2]

    # import required libraries
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

subscription_id = "<your subscription ID>"
resource_group = "<your resource group name>"
workspace = "<your workspace name>"

ml_client = MLClient(
    DefaultAzureCredential(), subscription_id, resource_group, workspace
)

# Get workspace info
ws=ml_client.workspaces.get(name=workspace)
# Update to use cpu-cluster for image builds
ws.image_build_compute="cpu-cluster"
ml_client.workspaces.begin_update(ws)

# To switch back to serverless compute:
# ws.image_build_compute = ''
# ml_client.workspaces.begin_update(ws)

    For more information, see the begin_update method reference.

    Portal

    Currently there isn't a way to set the image build compute from the Azure portal.

Tip

When ACR is behind a VNet, you can also disable public access to it.

Secure Azure Monitor and Application Insights

To enable network isolation for Azure Monitor and the Application Insights instance for the workspace, use the following steps:

  1. Open your Application Insights resource in the Azure portal. The Overview tab may or may not have a Workspace property. If it doesn't have the property, perform step 2. If it does, then you can proceed directly to step 3.

    [!TIP] New workspaces create a workspace-based Application Insights resource by default. If your workspace was recently created, then you would not need to perform step 2.

  2. Upgrade the Application Insights instance for your workspace. For steps on how to upgrade, see Migrate to workspace-based Application Insights resources.

  3. Create an Azure Monitor Private Link Scope and add the Application Insights instance from step 1 to the scope. For more information, see Configure your Azure Monitor private link.

Securely connect to your workspace

[!INCLUDE machine-learning-connect-secure-workspace]

Workspace diagnostics

[!INCLUDE machine-learning-workspace-diagnostics]

Public access to workspace

Important

While this is a supported configuration for Azure Machine Learning, Microsoft doesn't recommend it. You should verify this configuration with your security team before using it in production.

In some cases, you may need to allow access to the workspace from the public network (without connecting through the virtual network using the methods detailed the Securely connect to your workspace section). Access over the public internet is secured using TLS.

To enable public network access to the workspace, use the following steps:

  1. Enable public access to the workspace after configuring the workspace's private endpoint.
  2. Configure the Azure Storage firewall to allow communication with the IP address of clients that connect over the public internet. You may need to change the allowed IP address if the clients don't have a static IP. For example, if one of your Data Scientists is working from home and can't establish a VPN connection to the virtual network.

Next steps

This article is part of a series on securing an Azure Machine Learning workflow. See the other articles in this series: