Skip to content

Files

Latest commit

 

History

History
344 lines (247 loc) · 26.1 KB

how-to-network-security-overview.md

File metadata and controls

344 lines (247 loc) · 26.1 KB
title titleSuffix description services ms.service ms.subservice ms.reviewer ms.author author ms.date ms.topic ms.custom monikerRange
Secure workspace resources using virtual networks (VNets)
Azure Machine Learning
Secure Azure Machine Learning workspace resources and compute environments using an isolated Azure Virtual Network (VNet).
machine-learning
machine-learning
enterprise-readiness
None
larryfr
Blackmist
10/19/2023
how-to
references_regions, security, build-2023
azureml-api-2 || azureml-api-1

Secure Azure Machine Learning workspace resources using virtual networks (VNets)

:::moniker range="azureml-api-2" [!INCLUDE dev v2] :::moniker-end :::moniker range="azureml-api-1" [!INCLUDE dev v1] :::moniker-end

[!INCLUDE managed-vnet-note]

Secure Azure Machine Learning workspace resources and compute environments using Azure Virtual Networks (VNets). This article uses an example scenario to show you how to configure a complete virtual network.

This article is part of a series on securing an Azure Machine Learning workflow. See the other articles in this series:

This article is part of a series on securing an Azure Machine Learning workflow. See the other articles in this series:

:::moniker range="azureml-api-2"

For a tutorial on creating a secure workspace, see Tutorial: Create a secure workspace or Tutorial: Create a secure workspace using a template.

Prerequisites

This article assumes that you have familiarity with the following articles:

Example scenario

In this section, you learn how a common network scenario is set up to secure Azure Machine Learning communication with private IP addresses.

The following table compares how services access different parts of an Azure Machine Learning network with and without a VNet:

| Scenario | Workspace | Associated resources | Training compute environment | Inferencing compute environment | |-|-|-|-|-|-| |No virtual network| Public IP | Public IP | Public IP | Public IP | |Public workspace, all other resources in a virtual network | Public IP | Public IP (service endpoint)
- or -
Private IP (private endpoint) | Public IP | Private IP | |Secure resources in a virtual network| Private IP (private endpoint) | Public IP (service endpoint)
- or -
Private IP (private endpoint) | Private IP | Private IP |

  • Workspace - Create a private endpoint for your workspace. The private endpoint connects the workspace to the vnet through several private IP addresses.
    • Public access - You can optionally enable public access for a secured workspace.
  • Associated resource - Use service endpoints or private endpoints to connect to workspace resources like Azure storage, Azure Key Vault. For Azure Container Services, use a private endpoint.
    • Service endpoints provide the identity of your virtual network to the Azure service. Once you enable service endpoints in your virtual network, you can add a virtual network rule to secure the Azure service resources to your virtual network. Service endpoints use public IP addresses.
    • Private endpoints are network interfaces that securely connect you to a service powered by Azure Private Link. Private endpoint uses a private IP address from your VNet, effectively bringing the service into your VNet.
  • Training compute access - Access training compute targets like Azure Machine Learning Compute Instance and Azure Machine Learning Compute Clusters with public or private IP addresses.
  • Inference compute access - Access Azure Kubernetes Services (AKS) compute clusters with private IP addresses.

The next sections show you how to secure the network scenario described previously. To secure your network, you must:

  1. Secure the workspace and associated resources.
  2. Secure the training environment.
  3. Secure the inferencing environment.
  4. Optionally: enable studio functionality.
  5. Configure firewall settings.
  6. Configure DNS name resolution.

Public workspace and secured resources

Important

While this is a supported configuration for Azure Machine Learning, Microsoft doesn't recommend it. The data in the Azure Storage Account behind the virtual network can be exposed on the public workspace. You should verify this configuration with your security team before using it in production.

If you want to access the workspace over the public internet while keeping all the associated resources secured in a virtual network, use the following steps:

  1. Create an Azure Virtual Network. This network secures the resources used by the workspace.

  2. Use one of the following options to create a publicly accessible workspace:

    :::moniker range="azureml-api-2"

    OR

  3. Add the following services to the virtual network by using either a service endpoint or a private endpoint. Also allow trusted Microsoft services to access these services:

    :::moniker range="azureml-api-2"

    Service Endpoint information Allow trusted information
    Azure Key Vault Service endpoint
    Private endpoint    		 Allow trusted Microsoft services to bypass this firewall
    Azure Storage Account Service and private endpoint
    Private endpoint    		 Grant access to trusted Azure services
    Azure Container Registry Private endpoint Allow trusted services
    :::moniker-end
    :::moniker range="azureml-api-1"
    Service Endpoint information Allow trusted information
    ----- ----- -----
    Azure Key Vault Service endpoint
    Private endpoint    		 Allow trusted Microsoft services to bypass this firewall
    Azure Storage Account Service and private endpoint
    Private endpoint    		 Grant access to trusted Azure services
    Azure Container Registry Private endpoint Allow trusted services
    :::moniker-end

  4. In properties for the Azure Storage Account(s) for your workspace, add your client IP address to the allowed list in firewall settings. For more information, see Configure firewalls and virtual networks.

Secure the workspace and associated resources

Use the following steps to secure your workspace and associated resources. These steps allow your services to communicate in the virtual network.

:::moniker range="azureml-api-2"

  1. Create an Azure Virtual Networks. This network secures the workspace and other resources. Then create a Private Link-enabled workspace to enable communication between your VNet and workspace.

  2. Add the following services to the virtual network by using either a service endpoint or a private endpoint. Also allow trusted Microsoft services to access these services:

    Service Endpoint information Allow trusted information
    Azure Key Vault Service endpoint
    Private endpoint    		 Allow trusted Microsoft services to bypass this firewall
    Azure Storage Account Service and private endpoint
    Private endpoint    		 Grant access from Azure resource instances
    or
    Grant access to trusted Azure services
    Azure Container Registry Private endpoint Allow trusted services

:::moniker-end :::moniker range="azureml-api-1"

  1. Create an Azure Virtual Networks. This virtual network secures the workspace and other resources. Then create a Private Link-enabled workspace to enable communication between your VNet and workspace.

  2. Add the following services to the virtual network by using either a service endpoint or a private endpoint. Also allow trusted Microsoft services to access these services:

    Service Endpoint information Allow trusted information
    Azure Key Vault Service endpoint
    Private endpoint    		 Allow trusted Microsoft services to bypass this firewall
    Azure Storage Account Service and private endpoint
    Private endpoint    		 Grant access from Azure resource instances
    or
    Grant access to trusted Azure services
    Azure Container Registry Private endpoint Allow trusted services

:::moniker-end

:::image type="content" source="./media/how-to-network-security-overview/secure-workspace-resources.svg" alt-text="Diagram showing how the workspace and associated resources communicate inside a VNet.":::

:::moniker range="azureml-api-2" For detailed instructions on how to complete these steps, see Secure an Azure Machine Learning workspace. :::moniker-end :::moniker range="azureml-api-1" For detailed instructions on how to complete these steps, see Secure an Azure Machine Learning workspace. :::moniker-end

Limitations

Securing your workspace and associated resources within a virtual network have the following limitations:

  • The workspace and default storage account must be in the same VNet. However, subnets within the same VNet are allowed. For example, the workspace in one subnet and storage in another.

    We recommend that the Azure Key Vault and Azure Container Registry for the workspace are also in the same VNet. However both of these resources can also be in a peered VNet.

Secure the training environment

In this section, you learn how to secure the training environment in Azure Machine Learning. You also learn how Azure Machine Learning completes a training job to understand how the network configurations work together.

To secure the training environment, use the following steps:

:::moniker range="azureml-api-2"

  1. Create an Azure Machine Learning compute instance and computer cluster in the virtual network to run the training job.

  2. If your compute cluster or compute instance uses a public IP address, you must Allow inbound communication so that management services can submit jobs to your compute resources.

    [!TIP] Compute cluster and compute instance can be created with or without a public IP address. If created with a public IP address, you get a load balancer with a public IP to accept the inbound access from Azure batch service and Azure Machine Learning service. You need to configure User Defined Routing (UDR) if you use a firewall. If created without a public IP, you get a private link service to accept the inbound access from Azure batch service and Azure Machine Learning service without a public IP. :::moniker-end :::moniker range="azureml-api-1"

  3. Create an Azure Machine Learning compute instance and computer cluster in the virtual network to run the training job.

  4. If your compute cluster or compute instance uses a public IP address, you must Allow inbound communication so that management services can submit jobs to your compute resources.

    [!TIP] Compute cluster and compute instance can be created with or without a public IP address. If created with a public IP address, you get a load balancer with a public IP to accept the inbound access from Azure batch service and Azure Machine Learning service. You need to configure User Defined Routing (UDR) if you use a firewall. If created without a public IP, you get a private link service to accept the inbound access from Azure batch service and Azure Machine Learning service without a public IP. :::moniker-end

:::image type="content" source="./media/how-to-network-security-overview/secure-training-environment.svg" alt-text="Diagram showing how to secure managed compute clusters and instances.":::

:::moniker range="azureml-api-2" For detailed instructions on how to complete these steps, see Secure a training environment. :::moniker-end :::moniker range="azureml-api-1" For detailed instructions on how to complete these steps, see Secure a training environment. :::moniker-end

Example training job submission

In this section, you learn how Azure Machine Learning securely communicates between services to submit a training job. This example shows you how all your configurations work together to secure communication.

  1. The client uploads training scripts and training data to storage accounts that are secured with a service or private endpoint.

  2. The client submits a training job to the Azure Machine Learning workspace through the private endpoint.

  3. Azure Batch service receives the job from the workspace. It then submits the training job to the compute environment through the public load balancer for the compute resource.

  4. The compute resource receives the job and begins training. The compute resource uses information stored in key vault to access storage accounts to download training files and upload output.

:::image type="content" source="./media/how-to-network-security-overview/secure-training-job-submission.svg" alt-text="Diagram showing the secure training job submission workflow.":::

Limitations

  • Azure Compute Instance and Azure Compute Clusters must be in the same VNet, region, and subscription as the workspace and its associated resources.

Secure the inferencing environment

:::moniker range="azureml-api-2" You can enable network isolation for managed online endpoints to secure the following network traffic:

  • Inbound scoring requests.
  • Outbound communication with the workspace, Azure Container Registry, and Azure Blob Storage.

For more information, see Enable network isolation for managed online endpoints. :::moniker-end :::moniker range="azureml-api-1" In this section, you learn the options available for securing an inferencing environment when using the Azure CLI extension for ML v1 or the Azure Machine Learning Python SDK v1. When doing a v1 deployment, we recommend that you use Azure Kubernetes Services (AKS) clusters for high-scale, production deployments.

You have two options for AKS clusters in a virtual network:

  • Deploy or attach a default AKS cluster to your VNet.
  • Attach a private AKS cluster to your VNet.

Default AKS clusters have a control plane with public IP addresses. You can add a default AKS cluster to your VNet during the deployment or attach a cluster after it's created.

Private AKS clusters have a control plane, which can only be accessed through private IPs. Private AKS clusters must be attached after the cluster is created.

For detailed instructions on how to add default and private clusters, see Secure an inferencing environment.

Regardless default AKS cluster or private AKS cluster used, if your AKS cluster is behind of VNET, your workspace and its associate resources (storage, key vault, and ACR) must have private endpoints or service endpoints in the same VNET as the AKS cluster.

The following network diagram shows a secured Azure Machine Learning workspace with a private AKS cluster attached to the virtual network.

:::image type="content" source="./media/how-to-network-security-overview/secure-inferencing-environment.svg" alt-text="Diagram showing an attached private AKS cluster."::: :::moniker-end

Optional: Enable public access

You can secure the workspace behind a VNet using a private endpoint and still allow access over the public internet. The initial configuration is the same as securing the workspace and associated resources.

After securing the workspace with a private endpoint, use the following steps to enable clients to develop remotely using either the SDK or Azure Machine Learning studio:

:::moniker range="azureml-api-2"

  1. Enable public access to the workspace.
  2. Configure the Azure Storage firewall to allow communication with the IP address of clients that connect over the public internet. :::moniker-end :::moniker range="azureml-api-1"
  3. Enable public access to the workspace.
  4. Configure the Azure Storage firewall to allow communication with the IP address of clients that connect over the public internet. :::moniker-end

Optional: enable studio functionality

If your storage is in a VNet, you must use extra configuration steps to enable full functionality in studio. By default, the following features are disabled:

  • Preview data in the studio.
  • Visualize data in the designer.
  • Deploy a model in the designer.
  • Submit an AutoML experiment.
  • Start a labeling project.

To enable full studio functionality, see Use Azure Machine Learning studio in a virtual network.

Limitations

ML-assisted data labeling doesn't support a default storage account behind a virtual network. Instead, use a storage account other than the default for ML assisted data labeling.

Tip

As long as it is not the default storage account, the account used by data labeling can be secured behind the virtual network.

Configure firewall settings

Configure your firewall to control traffic between your Azure Machine Learning workspace resources and the public internet. While we recommend Azure Firewall, you can use other firewall products.

For more information on firewall settings, see Use workspace behind a Firewall.

Custom DNS

If you need to use a custom DNS solution for your virtual network, you must add host records for your workspace.

For more information on the required domain names and IP addresses, see how to use a workspace with a custom DNS server.

Microsoft Sentinel

Microsoft Sentinel is a security solution that can integrate with Azure Machine Learning. For example, using Jupyter notebooks provided through Azure Machine Learning. For more information, see Use Jupyter notebooks to hunt for security threats.

Public access

Microsoft Sentinel can automatically create a workspace for you if you're OK with a public endpoint. In this configuration, the security operations center (SOC) analysts and system administrators connect to notebooks in your workspace through Sentinel.

For information on this process, see Create an Azure Machine Learning workspace from Microsoft Sentinel

:::image type="content" source="./media/how-to-network-security-overview/common-public-endpoint-deployment.svg" alt-text="Diagram showing Microsoft Sentinel public connection.":::

Private endpoint

If you want to secure your workspace and associated resources in a VNet, you must create the Azure Machine Learning workspace first. You must also create a virtual machine 'jump box' in the same VNet as your workspace, and enable Azure Bastion connectivity to it. Similar to the public configuration, SOC analysts and administrators can connect using Microsoft Sentinel, but some operations must be performed using Azure Bastion to connect to the VM.

For more information on this configuration, see Create an Azure Machine Learning workspace from Microsoft Sentinel

:::image type="content" source="./media/how-to-network-security-overview/private-endpoint-deploy-bastion.svg" alt-text="Daigram showing Microsoft Sentinel connection through a VNet.":::

Next steps

This article is part of a series on securing an Azure Machine Learning workflow. See the other articles in this series:

:::moniker range="azureml-api-2"