Enhancing the deployment experience

docs/README.md

@@ -28,15 +28,34 @@ Feathr automatically computes your feature values and joins them to your trainin
 - **Native cloud integration** with simplified and scalable architecture, which is illustrated in the next section.
 - **Feature sharing and reuse made easy:** Feathr has built-in feature registry so that features can be easily shared across different teams and boost team productivity.
 
-## Running Feathr on Azure with 3 Simple Steps
+## Running Feathr on Azure with few Simple Steps
 
-Feathr has native cloud integration. To use Feathr on Azure, you only need three steps:
+1. To enable authentication on the Feathr UI (which gets created as part of the deployment script) we need to create an Azure Active Directory (AAD) application. Currently it is not possible to create one through ARM template but you can easily create one by running the following CLI commands in the [Cloud Shell](https://shell.azure.com/bash)
 
-1. Get the `Principal ID` of your account by running `az ad signed-in-user show --query id -o tsv` in the link below (Select "Bash" if asked), and write down that value (something like `b65ef2e0-42b8-44a7-9b55-abbccddeefff`). Think this ID as something representing you when accessing Azure, and it will be used to grant permissions in the next step in the UI.
+```bash
+# This is the prefix you want to name your resources with, make a note of it, you will need it during deployment.
+prefix="YOUR_RESOURCE_PREFIX" 
+
+# Please don't change this name, a corresponding webapp with same name gets created in subsequent steps.
+sitename="${prefix}webapp" 
+
+# This will create the Azure AD application, note that we need to create an AAD app of platform type Single Page Application(SPA). By default passing the redirect-uris with create command creates an app of type web. 
+az ad app create --display-name $sitename --sign-in-audience AzureADMyOrg --web-home-page-url "https://$sitename.azurewebsites.net" --enable-id-token-issuance true
+
+#Fetch the ClientId, TenantId and ObjectId for the created app
+aad_clientId=$(az ad app list --display-name $sitename --query [].appId -o tsv)
+aad_tenantId=$(az account tenant list --query [].tenantId -o tsv)
+aad_objectId=$(az ad app list --display-name $sitename --query [].id -o tsv)
+
+# Updating the SPA app created above, currently there is no CLI support to add redirectUris to a SPA, so we have to patch manually via az rest
+az rest --method PATCH --uri "https://graph.microsoft.com/v1.0/applications/$aad_objectId" --headers "Content-Type=application/json" --body "{spa:{redirectUris:['https://$sitename.azurewebsites.net/.auth/login/aad/callback']}}"
 
-[Launch Cloud Shell](https://shell.azure.com/bash)
+# Make a note of the ClientId and TenantId, you will need it during deployment.
+echo "AAD_CLIENT_ID: $aad_clientId"
+echo "AZURE_TENANT_ID: $aad_tenantId"
+``` 
 
-2. Click the button below to deploy a minimal set of Feathr resources for demo purpose. You will need to fill in the `Principal ID` and `Resource Prefix`. You will need "Owner" permission of the selected subscription.
+2. Click the button below to deploy a minimal set of Feathr resources. This is not for production use as we choose a minimal set of resources, but treat it as a template that you can modify for further use. Note that you should have "Owner" access in your subscription to perform some of the actions.
 
 [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Flinkedin%2Ffeathr%2Fmain%2Fdocs%2Fhow-to-guides%2Fazure_resource_provision.json)
 

docs/how-to-guides/azure-deployment.md

@@ -11,14 +11,9 @@ Due to the complexity of the possible cloud environment, it is almost impossible
 
 ## Method 1: Provision Azure Resources with Current User's Identity:
 
-Feathr has native cloud integration and getting started with Feathr is very straightforward. You only need three steps:
+Feathr has native cloud integration and getting started with Feathr is very straightforward. Here are the instructions:
 
-1. Get the principal ID of your account by running `az ad signed-in-user show --query id -o tsv` in the link below (Select "Bash" if you are asked to choose one), and write down that value (will be something like `b65ef2e0-42b8-44a7-9b55-abbccddeefff`)
-
-[Launch Cloud Shell](https://shell.azure.com/bash)
-
-
-2. To enable authentication on the Feathr UI (which gets created as part of the deployment script) we need to create an Azure Active Directory (AAD) application. Currently it is not possible to create one through ARM template but you can easily create one by running the following CLI commands in the [Cloud Shell](https://shell.azure.com/bash)
+1. To enable authentication on the Feathr UI (which gets created as part of the deployment script) we need to create an Azure Active Directory (AAD) application. Currently it is not possible to create one through ARM template but you can easily create one by running the following CLI commands in the [Cloud Shell](https://shell.azure.com/bash)
 
 ```bash
 # This is the prefix you want to name your resources with, make a note of it, you will need it during deployment.
@@ -43,7 +38,7 @@ echo "AAD_CLIENT_ID: $aad_clientId"
 echo "AZURE_TENANT_ID: $aad_tenantId"
 ``` 
 
-3. Click the button below to deploy a minimal set of Feathr resources. This is not for production use as we choose a minimal set of resources, but treat it as a template that you can modify for further use. Note that you should have "Owner" access in your subscription to perform some of the actions.
+2. Click the button below to deploy a minimal set of Feathr resources. This is not for production use as we choose a minimal set of resources, but treat it as a template that you can modify for further use. Note that you should have "Owner" access in your subscription to perform some of the actions.
 
 [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Flinkedin%2Ffeathr%2Fmain%2Fdocs%2Fhow-to-guides%2Fazure_resource_provision.json)
 

docs/how-to-guides/azure_resource_provision.json

@@ -10,23 +10,6 @@
         "description": "Resource prefix for all the resource provisioned. This should be an alphanumeric string."
       }
     },
-    "principalId": {
-      "type": "String",
-      "metadata": {
-        "description": "Specifies the principal ID assigned to the role. You can find it by logging into 'https://shell.azure.com/bash' and run 'az ad signed-in-user show --query id -o tsv'"
-      }
-    },
-    "allowAllConnections": {
-      "defaultValue": "true",
-      "allowedValues": [
-        "true",
-        "false"
-      ],
-      "type": "String",
-      "metadata": {
-        "description": "Specifies whether to allow client IPs to connect to Synapse"
-      }
-    },
     "provisionPurview": {
       "defaultValue": "true",
       "allowedValues": [
@@ -35,7 +18,7 @@
       ],
       "type": "String",
       "metadata": {
-        "description": "Whether or not put purview in the provision script"
+        "description": "Whether or not to provision purview as part of deployment script"
       }
     },
     "provisionEventHub": {
@@ -46,7 +29,7 @@
       ],
       "type": "String",
       "metadata": {
-        "description": "Whether or not to deploy eventhub provision script"
+        "description": "Whether or not to provision EventHub as part of deployment script"
       }
     },
     "sqlAdminUsername": {
@@ -61,24 +44,15 @@
         "description": "Specifies the password for admin"
       }
     },
-    "storageAccountKey": {
-      "type": "string",
-      "metadata": {
-        "description": "Specifies the key of the storage account where the BACPAC file is stored."
-      }
-    },
-    "bacpacFileUrl": {
-      "type": "string",
-      "defaultValue": "https://azurefeathrstorage.blob.core.windows.net/public/feathr-registry-schema.bacpac",
-      "metadata": {
-        "description": "This is the pre-created BACPAC file that contains required schemas by the registry server."
-      }
-    },
-    "preBuiltFeathrUIDockerImage": {
-      "defaultValue": "feathrfeaturestore/sql-registry",
+    "registryBackend": {
+      "defaultValue": "Azure-SQL",
+      "allowedValues": [
+        "Azure-SQL",
+        "Azure-Purview"
+      ],
       "type": "String",
       "metadata": {
-        "description": "Pre-Built Docker Image of Feathr UI (from dockerhub) with support for specific registry backends. Current supported registry backends are Azure Purview and Azure SQL"
+        "description": "Backend for registry metadata storage. Current supported registry backends are Azure SQL and Azure Purview"
       }
     },
     "azureADClientId": {
@@ -112,16 +86,22 @@
     "dlsFsName": "[toLower(concat(parameters('resourcePrefix'),'fs'))]",
     "dlsAccount": "[resourceId('Microsoft.Storage/storageAccounts', variables('dlsName'))]",
     "purviewName": "[concat(parameters('resourcePrefix'),'purview' )]",
+    "identityName": "[concat(parameters('resourcePrefix'),'identity' )]",
     "roleDefinitionIdForBlobContributor": "ba92f5b4-2d11-453d-a403-e96b0029c9fe",
     "roleDefinitionIdForKeyVaultSecretsUser": "4633458b-17de-408a-b874-0445c86b69e6",
-    "roleAssignmentNameForBlobContributor": "[guid(parameters('principalId'), variables('roleDefinitionIdForBlobContributor'), resourceGroup().id)]",
-    "roleAssignmentNameForKeyVaultSecretsUser": "[guid(parameters('principalId'), variables('roleDefinitionIdForKeyVaultSecretsUser'), resourceGroup().id)]",
+    "roleAssignmentNameForBlobContributor": "[guid(variables('roleDefinitionIdForBlobContributor'), resourceGroup().id)]",
+    "roleAssignmentNameForKeyVaultSecretsUser": "[guid(variables('roleDefinitionIdForKeyVaultSecretsUser'), resourceGroup().id)]",
     "webAppName": "[concat(parameters('resourcePrefix'),'webapp' )]",
     "webAppPlanName": "[concat(parameters('resourcePrefix'),'appplan' )]",
     "webAppPlanSku": "P1v2",
     "webAppAPIVersion": "2021-03-01",
     "sqlServerName": "[concat(parameters('resourcePrefix'),'dbserver' )]",
-    "sqlDatabaseName": "[concat(parameters('resourcePrefix'),'db' )]"
+    "sqlDatabaseName": "[concat(parameters('resourcePrefix'),'db' )]",
+    "sourceBacpacBlobUrl": "https://azurefeathrstorage.blob.core.windows.net/public/feathr-registry-schema.bacpac",
+    "bacpacBlobName": "feathr-registry-schema.bacpac",
+    "destinationBacpacBlobUrl": "[concat('https://',variables('dlsName'),'.blob.core.windows.net/',variables('dlsFsName'),'/',variables('bacpacBlobName'))]",
+    "deploymentScriptName": "CopyBacpacFile",
+    "preBuiltdockerImage": "feathrfeaturestore/sql-registry"
   },
   "functions": [],
   "resources": [
@@ -317,8 +297,7 @@
           "properties": {
             "startIpAddress": "0.0.0.0",
             "endIpAddress": "255.255.255.255"
-          },
-          "condition": "[equals(parameters('allowAllConnections'),'true')]"
+          }
         },
         {
           "type": "firewallrules",
@@ -380,7 +359,7 @@
         "httpsOnly": "true",
         "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('webAppPlanName'))]",
         "siteConfig": {
-          "linuxFxVersion": "[concat('DOCKER|', parameters('preBuiltFeathrUIDockerImage'))]",
+          "linuxFxVersion": "[concat('DOCKER|', variables('preBuiltdockerImage'))]",
           "alwaysOn": true,
           "ftpsState": "Disabled",
           "appSettings": [
@@ -432,30 +411,43 @@
         "nodeSize": "Medium"
       }
     },
+    {
+        "type": "Microsoft.ManagedIdentity/userAssignedIdentities",
+        "apiVersion": "2018-11-30",
+        "name": "[variables('identityName')]",
+        "location": "[variables('location')]"
+    },
     {
       "type": "Microsoft.Authorization/roleAssignments",
       "apiVersion": "2020-10-01-preview",
       "name": "[variables('roleAssignmentNameForBlobContributor')]",
       "dependsOn": [
-        "[variables('dlsAccount')]"
+        "[variables('dlsAccount')]",
+        "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('identityName'))]"
       ],
       "properties": {
         "roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', variables('roleDefinitionIdForBlobContributor'))]",
-        "principalId": "[parameters('principalId')]",
-        "scope": "[resourceGroup().id]"
+        "principalId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('identityName')), '2018-11-30').principalId]",
+        "scope": "[resourceGroup().id]",
+        "principalType": "ServicePrincipal"
+
       }
     },
     {
       "type": "Microsoft.Authorization/roleAssignments",
       "apiVersion": "2020-10-01-preview",
       "name": "[variables('roleAssignmentNameForKeyVaultSecretsUser')]",
       "dependsOn": [
-        "[variables('keyVault')]"
+        "[variables('keyVault')]",
+        "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('identityName'))]"
+
       ],
       "properties": {
         "roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', variables('roleDefinitionIdForKeyVaultSecretsUser'))]",
-        "principalId": "[parameters('principalId')]",
-        "scope": "[resourceGroup().id]"
+        "principalId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('identityName')), '2018-11-30').principalId]",
+        "scope": "[resourceGroup().id]",
+        "principalType": "ServicePrincipal"
+
       }
     },
     {
@@ -484,6 +476,31 @@
         }
       ]
     },
+    {
+    "type": "Microsoft.Resources/deploymentScripts",
+    "apiVersion": "2020-10-01",
+    "name": "[variables('deploymentScriptName')]",
+    "location": "[variables('location')]",
+    "kind": "AzureCLI",
+    "identity": {
+        "type": "UserAssigned",
+        "userAssignedIdentities": {
+            "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('identityName'))]": {}
+        }
+    },
+    "dependsOn": [
+        "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('identityName'))]",
+        "[resourceId('Microsoft.Storage/storageAccounts/blobServices/containers', variables('dlsName'), 'default', variables('dlsFsName'))]"
+    ],
+    "properties": {
+        "AzCliVersion": "2.2.0",
+        "timeout": "PT30M",
+        "arguments": "[concat(listKeys(variables('dlsAccount'),'2019-04-01').keys[0].value, ' ',variables('dlsName'), ' ', variables('bacpacBlobName'), ' ', variables('dlsFsName'),' ' ,variables('sourceBacpacBlobUrl'))]",
+        "scriptContent": "az storage blob copy start --account-key $1 --account-name $2 --destination-blob $3 --destination-container $4 --source-uri $5",
+        "cleanupPreference": "OnSuccess",
+        "retentionInterval": "P1D"
+    }
+},
     {
       "type": "Microsoft.Sql/servers/databases",
       "apiVersion": "2021-11-01-preview",
@@ -498,12 +515,14 @@
           "apiVersion": "2021-11-01-preview",
           "name": "Import",
           "dependsOn": [
-            "[resourceId('Microsoft.Sql/servers/databases', variables('sqlServerName'), variables('sqlDatabaseName'))]"
+            "[resourceId('Microsoft.Sql/servers/databases', variables('sqlServerName'), variables('sqlDatabaseName'))]",
+            "[resourceId('Microsoft.Resources/deploymentScripts', variables('deploymentScriptName'))]"
+
           ],
           "properties": {
             "storageKeyType": "StorageAccessKey",
-            "storageKey": "[parameters('storageAccountKey')]",
-            "storageUri": "[parameters('bacpacFileUrl')]",
+            "storageKey": "[listKeys(variables('dlsAccount'),'2019-04-01').keys[0].value]",
+            "storageUri": "[variables('destinationBacpacBlobUrl')]",
             "administratorLogin": "[parameters('sqlAdminUsername')]",
             "administratorLoginPassword": "[parameters('sqlAdminPassword')]",
             "operationMode": "Import"