Feature/initial code

README.md

@@ -1,3 +1,94 @@
+### Author: Praneet Singh Solanki
+
+# DevOps For AI
+
+[DevOps for AI template](https://azuredevopsdemogenerator.azurewebsites.net/?name=azure%20machine%20learning) will help you to understand how to build the Continuous Integration and Continuous Delivery pipeline for a ML/AI project. We will be using the Azure DevOps Project for build and release pipelines along with Azure ML services for ML/AI model management and operationalization. 
+
+This template contains code and pipeline definition for a machine learning project demonstrating how to automate the end to end ML/AI project. The build pipelines include DevOps tasks for data sanity test, unit test, model training on different compute targets, model version management, model evaluation/model selection, model deployment as realtime web service, staged deployment to QA/prod, integration testing and functional testing.
+
+## Prerequisite
+- Active Azure subscription
+- Minimum contributor access to Azure subscription
+
+## Getting Started:
+
+### Import the DevOps for AI solution template from Azure DevOps Demo Generator: [Click here](https://azuredevopsdemogenerator.azurewebsites.net/?name=azure%20machine%20learning)
+
+Skip above step if already done.
+
+Once the template is imported for personal Azure DevOps account using DevOps demo generator, you need to follow below steps to get the pipeline running:
+
+### Update Pipeline Config:
+
+#### Build Pipeline
+1. Go to the **Pipelines -> Builds** on the newly created project and click **Edit** on top right
+![EditPipeline1](/docs/images/EditPipeline1.png)
+2. Click on **Create or Get Workspace** task, select the Azure subscription where you want to deploy and run the solution, and click **Authorize**
+![EditPipeline2](/docs/images/EditPipeline2.png)
+3. Click all other tasks below it and select the same subscription (no need to authorize again)
+4. Once the tasks are updated with subscription, click on **Save & queue** and select **Save**
+![EditPipeline3](/docs/images/EditPipeline3.png)
+
+#### Release Pipeline
+1. Go to the **Pipelines -> Releases** and click **Edit** on top  
+![EditPipeline4](/docs/images/EditPipeline4.png)
+2. Click on **1 job, 4 tasks** to open the tasks in **QA stage**
+![EditPipeline5](/docs/images/EditPipeline5.png)
+3. Update the subscription details in two tasks 
+![EditPipeline6](/docs/images/EditPipeline6.png)
+4. Click on **Tasks** on the top to switch to the Prod stage, update the subscription details for the two tasks in prod
+![EditPipeline7](/docs/images/EditPipeline7.png)
+5. Once you fix all the missing subscription, the **Save** is no longer grayed, click on save to save the changes in release pepeline
+![EditPipeline8](/docs/images/EditPipeline8.png)
+
+### Update Repo config:
+1. Go to the **Repos** on the newly created Azure DevOps project
+2. Open the config file [/aml_config/config.json](/aml_config/config.json) and edit it
+3. Put your Azure subscription ID in place of <>
+4. Change resource group and AML workspace name if you want
+5. Put the location where you want to deploy your Azure ML service workspace
+6. Save the changes and commit these changes to master branch 
+7. The commit will trigger the build pipeline to run deploying AML end to end solution
+8. Go to **Pipelines -> Builds** to see the pipeline run
+
+## Steps Performed in the Build Pipeline:
+
+1. Prepare the python environment
+2. Get or Create the workspace
+3. Submit Training job on the remote DSVM / Local Python Env
+4. Register model to workspace
+5. Create Docker Image for Scoring Webservice
+6. Copy and Publish the Artifacts to Release Pipeline
+
+## Steps Performed in the Release Pipeline
+In Release pipeline we deploy the image created from the build pipeline to Azure Container Instance and Azure Kubernetes Services
+
+### Deploy on ACI - QA Stage
+1. Prepare the python environment
+2. Create ACI and Deploy webservice image created in Build Pipeline
+3. Test the scoring image
+
+### Deploy on AKS - PreProd/Prod Stage
+1. Prepare the python environment
+2. Deploy on AKS
+    - Create AKS and create a new webservice on AKS with the scoring docker image
+
+        OR
+
+    - Get the existing AKS and update the webservice with new image created in Build Pipeline
+3. Test the scoring image
+
+### Repo Details
+
+You can find the details of the code ans scripts in the repository [here](/docs/code_description.md)
+
+### References
+
+- [Azure Machine Learning(Azure ML) Service Workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/overview-what-is-azure-ml)
+
+- [Azure ML Samples](https://docs.microsoft.com/en-us/azure/machine-learning/service/samples-notebooks)
+- [Azure ML Python SDK Quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-create-workspace-with-python)
+- [Azure DevOps](https://docs.microsoft.com/en-us/azure/devops/?view=vsts)
 
 # Contributing
 

aml_config/conda_dependencies.yml

@@ -0,0 +1,50 @@
+# Conda environment specification. The dependencies defined in this file will
+
+# be automatically provisioned for managed runs. These include runs against
+
+# the localdocker, remotedocker, and cluster compute targets.
+
+
+# Note that this file is NOT used to automatically manage dependencies for the
+
+# local compute target. To provision these dependencies locally, run:
+
+# conda env update --file conda_dependencies.yml
+
+
+# Details about the Conda environment file format:
+
+# https://conda.io/docs/using/envs.html#create-environment-file-by-hand
+
+
+# For managing Spark packages and configuration, see spark_dependencies.yml.
+
+
+# Version of this configuration file's structure and semantics in AzureML.
+
+# This directive is stored in a comment to preserve the Conda file structure.
+
+# [AzureMlVersion] = 2
+
+
+name: project_environment
+dependencies:
+  # The python interpreter version.
+
+  # Currently Azure ML Workbench only supports 3.5.2 and later.
+
+- python=3.6.2
+  # Required by azureml-defaults, installed separately through Conda to
+
+  # get a prebuilt version and not require build tools for the install.
+
+- psutil=5.3
+
+- pip:
+    # Required packages for AzureML execution, history, and data preparation.
+  - azureml-sdk[notebooks]
+  - pynacl==1.2.1
+  - scipy==1.0.0
+  - scikit-learn==0.19.1
+  - pandas==0.23.1
+  - numpy==1.14.5

aml_config/config.json

@@ -0,0 +1,6 @@
+{
+    "subscription_id": "<>",
+    "resource_group": "DevOps_AzureML_Demo",
+    "workspace_name": "AzureML_Demo_ws",
+    "location": "southcentralus"
+}

aml_config/security_config.json

@@ -0,0 +1,9 @@
+{
+    "sp_user" : "<>",
+    "sp_password" : "<>",
+    "sp_tenant_id" : "<>",
+    "remote_vm_name" : "<>",
+    "remote_vm_username" : "<>",
+    "remote_vm_password" : "<>",
+    "remote_vm_ip" : "<>"
+}

aml_service/00-WorkSpace.py

@@ -0,0 +1,59 @@
+"""
+Copyright (C) Microsoft Corporation. All rights reserved.​
+ ​
+Microsoft Corporation (“Microsoft”) grants you a nonexclusive, perpetual,
+royalty-free right to use, copy, and modify the software code provided by us
+("Software Code"). You may not sublicense the Software Code or any use of it
+(except to your affiliates and to vendors to perform work on your behalf)
+through distribution, network access, service agreement, lease, rental, or
+otherwise. This license does not purport to express any claim of ownership over
+data you may have shared with Microsoft in the creation of the Software Code.
+Unless applicable law gives you more rights, Microsoft reserves all other
+rights not expressly granted herein, whether by implication, estoppel or
+otherwise. ​
+ ​
+THE SOFTWARE CODE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS
+OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+MICROSOFT OR ITS LICENSORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THE SOFTWARE CODE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+"""
+from azureml.core import Workspace
+import os, json
+import azureml.core
+
+print("SDK Version:", azureml.core.VERSION)
+# print('current dir is ' +os.curdir)
+with open("aml_config/config.json") as f:
+    config = json.load(f)
+
+workspace_name = config["workspace_name"]
+resource_group = config["resource_group"]
+subscription_id = config["subscription_id"]
+location = config["location"]
+# location = 'southcentralus'
+try:
+    ws = Workspace.get(
+        name=workspace_name,
+        subscription_id=subscription_id,
+        resource_group=resource_group,
+    )
+
+except:
+    # this call might take a minute or two.
+    print("Creating new workspace")
+    ws = Workspace.create(
+        name=workspace_name,
+        subscription_id=subscription_id,
+        resource_group=resource_group,
+        # create_resource_group=True,
+        location=location,
+    )
+
+# print Workspace details
+print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep="\n")

aml_service/01-Experiment.py

@@ -0,0 +1,41 @@
+"""
+Copyright (C) Microsoft Corporation. All rights reserved.​
+ ​
+Microsoft Corporation (“Microsoft”) grants you a nonexclusive, perpetual,
+royalty-free right to use, copy, and modify the software code provided by us
+("Software Code"). You may not sublicense the Software Code or any use of it
+(except to your affiliates and to vendors to perform work on your behalf)
+through distribution, network access, service agreement, lease, rental, or
+otherwise. This license does not purport to express any claim of ownership over
+data you may have shared with Microsoft in the creation of the Software Code.
+Unless applicable law gives you more rights, Microsoft reserves all other
+rights not expressly granted herein, whether by implication, estoppel or
+otherwise. ​
+ ​
+THE SOFTWARE CODE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS
+OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+MICROSOFT OR ITS LICENSORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THE SOFTWARE CODE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+"""
+import os
+from azureml.core import Experiment
+from azureml.core import Workspace
+
+
+def getExperiment():
+    ws = Workspace.from_config()
+    script_folder = "."
+    experiment_name = "devops-ai-demo"
+    exp = Experiment(workspace=ws, name=experiment_name)
+    print(exp.name, exp.workspace.name, sep="\n")
+    return exp
+
+
+if __name__ == "__main__":
+    exp = getExperiment()

aml_service/02-AttachTrainingVM.py

@@ -0,0 +1,76 @@
+"""
+Copyright (C) Microsoft Corporation. All rights reserved.​
+ ​
+Microsoft Corporation (“Microsoft”) grants you a nonexclusive, perpetual,
+royalty-free right to use, copy, and modify the software code provided by us
+("Software Code"). You may not sublicense the Software Code or any use of it
+(except to your affiliates and to vendors to perform work on your behalf)
+through distribution, network access, service agreement, lease, rental, or
+otherwise. This license does not purport to express any claim of ownership over
+data you may have shared with Microsoft in the creation of the Software Code.
+Unless applicable law gives you more rights, Microsoft reserves all other
+rights not expressly granted herein, whether by implication, estoppel or
+otherwise. ​
+ ​
+THE SOFTWARE CODE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS
+OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+MICROSOFT OR ITS LICENSORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THE SOFTWARE CODE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+"""
+
+from azureml.core import Workspace
+from azureml.core import Run
+from azureml.core import Experiment
+from azureml.core.conda_dependencies import CondaDependencies
+from azureml.core.runconfig import RunConfiguration
+import os, json
+from azureml.core.compute import RemoteCompute
+from azureml.core.compute import DsvmCompute
+from azureml.core.compute_target import ComputeTargetException
+
+
+# Get workspace
+ws = Workspace.from_config()
+
+# Read the New VM Config
+with open("aml_config/security_config.json") as f:
+    config = json.load(f)
+
+remote_vm_name = config["remote_vm_name"]
+remote_vm_username = config["remote_vm_username"]
+remote_vm_password = config["remote_vm_password"]
+remote_vm_ip = config["remote_vm_ip"]
+
+try:
+    dsvm_compute = RemoteCompute.attach(
+        ws,
+        name=remote_vm_name,
+        username=remote_vm_username,
+        address=remote_vm_ip,
+        ssh_port=22,
+        password=remote_vm_password,
+    )
+    dsvm_compute.wait_for_completion(show_output=True)
+
+except Exception as e:
+    print("Caught = {}".format(e.message))
+    print("Compute config already attached.")
+
+
+## Create VM if not available
+# compute_target_name = remote_vm_name
+
+# try:
+#     dsvm_compute = DsvmCompute(workspace=ws, name=compute_target_name)
+#     print('found existing:', dsvm_compute.name)
+# except ComputeTargetException:
+#     print('creating new.')
+#     dsvm_config = DsvmCompute.provisioning_configuration(vm_size="Standard_D2_v2")
+#     dsvm_compute = DsvmCompute.create(ws, name=compute_target_name, provisioning_configuration=dsvm_config)
+#     dsvm_compute.wait_for_completion(show_output=True)