Feature/aml managed vnet #43

yogitasrivastava · 2024-04-16T01:22:35Z

Add automation for managed-vnet AML workspace (allow internet outbound) to Azure recipes

...

Does this introduce a breaking change?

[ ] Yes
[x ] No

Pull Request Type

What kind of change does this Pull Request introduce?

[ ] Bugfix
[x ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

Get the code

git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
npm install

Test the code

Please follow the test instructions in the Readme.md for test cases.

What to Check

Verify that the following are valid

...

Other Information

Please follow the Readme.md to install the code for automation.

…rkspace.jpg

…rkspace

managed vnet creation template

promisinganuj

More to come!

promisinganuj · 2024-05-05T22:52:34Z

src/az-aml-managed-vnet/README.md

+## Scenario
+
+<!-- Describe the usage scenario for this template.  Describe the challenges this recipes aims to address. -->
+This scenario aims to address the challenge of correctly configuring an Azure machine learning workspace within a Microsoft managed VNet including ensuring appropriate connectivity with common services such as Azure Storage Account, Azure Key Vault, Azure Container Registry.


Suggested change

This scenario aims to address the challenge of correctly configuring an Azure machine learning workspace within a Microsoft managed VNet including ensuring appropriate connectivity with common services such as Azure Storage Account, Azure Key Vault, Azure Container Registry.

This scenario aims to address the challenge of correctly configuring an Azure Machine Learning (AML) workspace within a managed virtual network (managed VNet). This includes ensuring appropriate connectivity with common Azure services such as Azure Storage Account, Azure Key Vault, and Azure Container Registry.

promisinganuj · 2024-05-05T23:06:57Z

src/az-aml-managed-vnet/README.md

+### Problem Summary
+
+<!--Briefly describe the problme that this recipe intends to resolve or make easier. -->
+Azure machine learning workspace is composed of a number of different components: Machine Learning Studio, workspace storage account, key vault, machine learning Data Pipelines, container registry and other external data sources like Azure SQL Server, ADLS Gen2. Despite being under a single machine learning umbrella service, each of these sub-components require a slightly different VNet configuration treatment to properly isolate network traffic. For example, generally you need at least four Private Endpoints configured for a single workspace each with connecting to a different sub-component. Another example, while managed workspace are generally a single tenant service with compute resources spun up within a designated Managed VNet, data scientist vm's, azure dev ops pipelines could be multi-tenanted and therefore require provisioning a Private Endpoint within the bridge vnet in order to connect to the workspace successfully. 


Suggested change

Azure machine learning workspace is composed of a number of different components: Machine Learning Studio, workspace storage account, key vault, machine learning Data Pipelines, container registry and other external data sources like Azure SQL Server, ADLS Gen2. Despite being under a single machine learning umbrella service, each of these sub-components require a slightly different VNet configuration treatment to properly isolate network traffic. For example, generally you need at least four Private Endpoints configured for a single workspace each with connecting to a different sub-component. Another example, while managed workspace are generally a single tenant service with compute resources spun up within a designated Managed VNet, data scientist vm's, azure dev ops pipelines could be multi-tenanted and therefore require provisioning a Private Endpoint within the bridge vnet in order to connect to the workspace successfully.

Azure machine learning workspace is composed of a number of different components: Machine Learning Studio, workspace storage account, key vault, machine learning Data Pipelines, container registry and other external data sources like Azure SQL Server, Azure Data Lake Storage Gen2 (ADLS Gen2). Despite being under a single machine learning umbrella service, each of these sub-components require a slightly different VNet configuration treatment to properly isolate network traffic. For instance, configuring at least four private endpoints is necessary for a single workspace, each connecting to a distinct sub-component. Another example pertains to managed workspaces, which are typically single-tenant services with compute resources deployed within a designated managed VNet. However, data scientist VMs and Azure DevOps pipelines could be multi-tenanted, and thus require the provisioning of a private endpoint within the bridge VNet in order to connect to the workspace successfully.

promisinganuj · 2024-05-05T23:20:13Z

src/az-aml-managed-vnet/README.md

+<!--Briefly describe the problme that this recipe intends to resolve or make easier. -->
+Azure machine learning workspace is composed of a number of different components: Machine Learning Studio, workspace storage account, key vault, machine learning Data Pipelines, container registry and other external data sources like Azure SQL Server, ADLS Gen2. Despite being under a single machine learning umbrella service, each of these sub-components require a slightly different VNet configuration treatment to properly isolate network traffic. For example, generally you need at least four Private Endpoints configured for a single workspace each with connecting to a different sub-component. Another example, while managed workspace are generally a single tenant service with compute resources spun up within a designated Managed VNet, data scientist vm's, azure dev ops pipelines could be multi-tenanted and therefore require provisioning a Private Endpoint within the bridge vnet in order to connect to the workspace successfully. 
+
+In addition to this, customers will also need to ensure that traffic between the Azure machine learning workspace studio can still privately flow between the workspace components and additional Azure services such as storage external to the managed VNet. This is done through the use of Private Endpoints. Another important that one has to keep in mind is the secure integration of Azure Machine Learning with Azure DevOps pipelines and Github actions that are enabled through the bridge virtual network (VNet to access resources in the architecure diagram).


Suggested change

In addition to this, customers will also need to ensure that traffic between the Azure machine learning workspace studio can still privately flow between the workspace components and additional Azure services such as storage external to the managed VNet. This is done through the use of Private Endpoints. Another important that one has to keep in mind is the secure integration of Azure Machine Learning with Azure DevOps pipelines and Github actions that are enabled through the bridge virtual network (VNet to access resources in the architecure diagram).

In addition to this, customers will also need to ensure that traffic within the Azure machine learning workspace studio can still privately flow between the workspace components and additional Azure services such as external storage accounts. This is done through the use of private endpoints. Another crucial aspect to consider is the secure integration of Azure Machine Learning with Azure DevOps pipelines and GitHub actions, which is facilitated through a bridge virtual network ("VNet to access AML workspace and resources" in the architecture diagram).

promisinganuj · 2024-05-05T23:23:26Z

src/az-aml-managed-vnet/README.md

+
+In addition to this, customers will also need to ensure that traffic between the Azure machine learning workspace studio can still privately flow between the workspace components and additional Azure services such as storage external to the managed VNet. This is done through the use of Private Endpoints. Another important that one has to keep in mind is the secure integration of Azure Machine Learning with Azure DevOps pipelines and Github actions that are enabled through the bridge virtual network (VNet to access resources in the architecure diagram).
+
+This recipe aims to provide developers a starting point with an IaC example of an Azure machine learning managed-vnet workspace with all sub-components correctly configured to ensure traffic stays private, while still being able to connect to common additional services such as Azure Storage Account, ADLS Gen2 and Azure Key Vault.


Suggested change

This recipe aims to provide developers a starting point with an IaC example of an Azure machine learning managed-vnet workspace with all sub-components correctly configured to ensure traffic stays private, while still being able to connect to common additional services such as Azure Storage Account, ADLS Gen2 and Azure Key Vault.

This recipe aims to provide developers with a starting point, offering an Infrastructure as Code (IaC) example of an Azure Machine Learning managed VNet workspace with the required secured networking configured as described above.

promisinganuj · 2024-05-05T23:26:35Z

src/az-aml-managed-vnet/README.md

+### Architecture
+
+<!-- Include a high-level architecture diagram of the components used in this recipe. -->
+![architecture](./media/AML_ManagedVNet_Secure_Architecture.png)


Suggest removing the Azure services if they are not deployed as part of the recipe.

Also, suggest using a different font "helvetica" or "tahoma" for the text.

Suggest providing a brief description (few paragraphs) of what's happening in this architecture diagram.

promisinganuj · 2024-05-06T03:19:58Z

src/az-aml-managed-vnet/README.md

+In addition to adding a private endpoint for the AML workspace, a private endpoint should be created for the AML dependant resources such as Azure Blob Storage and Key Vault inside your chosen VNet to enable connectivity to these resources.
+
+### Testing Solution
+To test Azure Machine Learning (AML) end-to-end, you can follow these steps using the **AzureML in a day notebook** from the **Samples** folder within the Notebooks section of Azure Machine Learning:


Where is the notebook? If it's just one notebook, I suggest copying it in the repo and add reference to it. That way, user doesn't have to go to another repo.

Having said that, please feel free to add link to the "AML rep" for reference.

promisinganuj · 2024-05-06T03:22:16Z

src/az-aml-managed-vnet/README.md

+
+In addition to adding a private endpoint for the AML workspace, a private endpoint should be created for the AML dependant resources such as Azure Blob Storage and Key Vault inside your chosen VNet to enable connectivity to these resources.
+
+### Testing Solution


In general, please following the markdown linting rules. For example, leaving spaces before and after the headers. "markdownlint" VS Code extension can help with this.

promisinganuj · 2024-05-06T03:22:57Z

src/az-aml-managed-vnet/README.md

+### Testing Solution
+To test Azure Machine Learning (AML) end-to-end, you can follow these steps using the **AzureML in a day notebook** from the **Samples** folder within the Notebooks section of Azure Machine Learning:
+- Create a Compute Instance:
+  - On the left navigation, select Compute and then Compute Instance. Create a new compute instance and supply a name. Keep all the defaults, expect under networking select No Public IP. Select Review & Create and wait for the deployment to be completed.


Can we add a GIF image to show this?

promisinganuj · 2024-05-06T03:27:19Z

src/az-aml-managed-vnet/deploy/Terraform/workspace.tf

+  name                     = "sa${random_string.sa_prefix.result}${var.environment}"
+  location                 = azurerm_resource_group.default.location
+  resource_group_name      = azurerm_resource_group.default.name
+  account_tier             = "Standard"


Is there any particular reason for not deploying "ADLS Gen2" account? (is_hns_enabled=true).

promisinganuj · 2024-05-06T04:19:52Z

src/az-aml-managed-vnet/README.md

+  - Sign in to the Azure Machine Learning studio.
+  - Navigate to Notebooks and select the Samples tab.
+  - Look for the AzureML in a day notebook.
+  - Open the notebook and clikc on 'Clone' to create a copy in your workspace file share.


I am getting the following error:

I have already created "blob" PE for the storage account and AKV in the customer VNet.

promisinganuj · 2024-05-06T04:29:45Z

Hi @yogitasrivastava / @oloomi / @RenSilvaAU ,

Overall, this is looking really good. Well done!

The recipes in this repo also deploys the "customer VNet" and that's missing from this sample. Suggest you add the following Azure resources:

A customer VNet.
Private Endpoint from storage account to this VNet.
Private Endpoint from key vault to this VNet.
Private Endpoint from AML workspace to this VNet.
Grant get/set access policy on AKV to the deployer.
You would also need to create private DNS Zones for storage, key vault, and AML workspace.

promisinganuj · 2024-05-06T04:35:06Z

Hi @yogitasrivastava / @oloomi / @RenSilvaAU ,

Overall, this is looking really good. Well done!

The recipes in this repo also deploys the "customer VNet" and that's missing from this sample. Suggest you add the following Azure resources:

A customer VNet.

Private Endpoint from storage account to this VNet.

Private Endpoint from key vault to this VNet.

Private Endpoint from AML workspace to this VNet.

Grant get/set access policy on AKV to the deployer.

You would also need to create private DNS Zones for storage, key vault, and AML workspace.

Refer https://github.com/Azure-Samples/virtual-network-integration-recipes/pull/27/files?short_path=64a62a4#diff-64a62a4611883e924d833faeb925f4339bb201c9a3b2df284c1446b9db30f94b for additional guidance. Happy to chat about the details.

yogitasrivastava and others added 30 commits January 9, 2024 16:31

Update recipe-template.md

138f355

Update recipe-template.md

63062ce

Create architecture_managedvnetworkspace.jpg

cde911d

Delete src/az-managed-vnet-workspace/media/architecture_managedvnetwo…

fce3fff

…rkspace.jpg

Create architecture_managedvnetworkspace

0734859

Delete src/az-managed-vnet-workspace/media/architecture_managedvnetwo…

b14f769

…rkspace

Create test

d0e865d

Create test

b99f2de

Delete test

765dc66

Delete src/az-managed-vnet-workspace directory

bb1087a

Create az-managed-vnet-workspace

848a497

Delete src/az-managed-vnet-workspace

924d34d

Create test

e2a2845

Delete src/az-managed-vnet-workspace/media directory

2fbbe33

created aml recipes

13d890e

Update README.md

21d2e41

Add files via upload

5136224

Delete src/az-aml-managed-vnet/recipe-template.md

40ef920

Update README.md

143fd1d

Add files via upload

25ed102

Delete src/az-aml-managed-vnet/deploy/bicep directory

fc0ed4c

Add files via upload

3315b9b

Delete src/az-aml-managed-vnet/media/solution-architecture.png

e36b713

template

338dcf8

managed vnet creation template

Delete src/az-aml-managed-vnet/bicep/test.bicep.txt

d281902

Add files via upload

1a1dc53

Add files via upload

ed199f5

Update README.md

afab950

Delete src/az-aml-managed-vnet/bicep/template.bicep

6d4b464

Update README.md

d790a3c

yogitasrivastava and others added 23 commits February 23, 2024 14:57

Update README.md

37829da

Update README.md and add testing solution

4755cc2

Update README.md

d96457f

Update README.md

4c62514

Update README.md

a94e475

Update README.md

767775c

Update README.md

9dcb88a

Update README.md

3ef8f22

Update README.md

e2fb9d6

Add files via upload

a93b0fd

Update README.md

d29be92

Update README.md

fea7172

Delete src/az-aml-managed-vnet/media/gitbash_pwd.png

640a553

Add files via upload

fe577e1

Update README.md

4646576

Update README.md

a4e6400

Update README.md

b32885a

Add files via upload

5dbb84a

Update README.md

c17f87d

Update README.md

7569010

Update README.md

c9fdff9

Add files via upload

d019f6c

Update README.md

ced173d

yogitasrivastava requested a review from promisinganuj April 16, 2024 01:22

yogitasrivastava assigned yogitasrivastava, RenSilvaAU and oloomi Apr 16, 2024

promisinganuj requested changes May 6, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/aml managed vnet #43

Feature/aml managed vnet #43

yogitasrivastava commented Apr 16, 2024

promisinganuj left a comment

promisinganuj May 5, 2024

promisinganuj May 5, 2024

promisinganuj May 5, 2024

promisinganuj May 5, 2024

promisinganuj May 5, 2024

promisinganuj May 5, 2024

promisinganuj May 6, 2024

promisinganuj May 6, 2024

promisinganuj May 6, 2024

promisinganuj May 6, 2024

promisinganuj May 6, 2024

promisinganuj commented May 6, 2024 •

edited

promisinganuj commented May 6, 2024

	This scenario aims to address the challenge of correctly configuring an Azure machine learning workspace within a Microsoft managed VNet including ensuring appropriate connectivity with common services such as Azure Storage Account, Azure Key Vault, Azure Container Registry.
	This scenario aims to address the challenge of correctly configuring an Azure Machine Learning (AML) workspace within a managed virtual network (managed VNet). This includes ensuring appropriate connectivity with common Azure services such as Azure Storage Account, Azure Key Vault, and Azure Container Registry.


		In addition to this, customers will also need to ensure that traffic between the Azure machine learning workspace studio can still privately flow between the workspace components and additional Azure services such as storage external to the managed VNet. This is done through the use of Private Endpoints. Another important that one has to keep in mind is the secure integration of Azure Machine Learning with Azure DevOps pipelines and Github actions that are enabled through the bridge virtual network (VNet to access resources in the architecure diagram).

		This recipe aims to provide developers a starting point with an IaC example of an Azure machine learning managed-vnet workspace with all sub-components correctly configured to ensure traffic stays private, while still being able to connect to common additional services such as Azure Storage Account, ADLS Gen2 and Azure Key Vault.

	This recipe aims to provide developers a starting point with an IaC example of an Azure machine learning managed-vnet workspace with all sub-components correctly configured to ensure traffic stays private, while still being able to connect to common additional services such as Azure Storage Account, ADLS Gen2 and Azure Key Vault.
	This recipe aims to provide developers with a starting point, offering an Infrastructure as Code (IaC) example of an Azure Machine Learning managed VNet workspace with the required secured networking configured as described above.


		In addition to adding a private endpoint for the AML workspace, a private endpoint should be created for the AML dependant resources such as Azure Blob Storage and Key Vault inside your chosen VNet to enable connectivity to these resources.

		### Testing Solution

Feature/aml managed vnet #43

Are you sure you want to change the base?

Feature/aml managed vnet #43

Conversation

yogitasrivastava commented Apr 16, 2024

Add automation for managed-vnet AML workspace (allow internet outbound) to Azure recipes

Does this introduce a breaking change?

Pull Request Type

How to Test

What to Check

Other Information

promisinganuj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

promisinganuj commented May 6, 2024 • edited

promisinganuj commented May 6, 2024

promisinganuj commented May 6, 2024 •

edited