Merge work from Global Hackathon 2023 (#25)

* Changes 0825-0333pm

* added KV Secrets loops.

* Vishal's Changes 0831-0257

* Prepare .gitignore for customer param files

* Fix vnet and web app max lengths

* Create sample bicepparam file

* Create deployment PS script

* Support for App Service Plan and App Insights

* Naming cleanup:
Create resource group names that match namingConvention
Add parameter descriptions
Remove storage FQDN hardcoding
Update module names
Add TODO comments

* Add TODO comments

* Fixes DBHostName Web App setting is incorrect kalalvishal/azure-redcap-paas #13

* Remove unnecessary default param value

* Fixes #20

* Add TODO comment

* Add comments

* Fixes Network address parameter #24

* Add `vnetAddressSpace` param to sample param file

* fixed keyvault role assignment deployment issues

* feedback update

* Add TODO comments

* Fixes Key Vault reference in App Service #15

* Specify MySQL credentials as parameters
Create Generate-Password PS module to create a strong password

* Fixes #19

* Add support for UAMI and deployment scripts

* Set sql_generate_invisible_primary_key OFF using deployment script

* Updated WebApp and created a new module for monitoring.

* parameterize redcapZipUri, redcapCommunityUserName&Password (#30)

* parameterize redcapZipUri, redcapCommunityUserName&Password

* store redcap credentials in kv and reference from web app settings

* feedback update

* Updates performed as per the comments.

* Updates performed as per the comments.

* Update law.bicep

* Conflict fixed.

* Use JSON file for deploy to support inline param
Update sample param file

* Update param descriptions

* Create structured and unique deployment names

* Update sample param file with ref to param val

* github workflow added.

* fixed the changes required for issue #37

* fixed the changes required for issue #36

* update Bicep-build.yml based on the comments.

* Add clarifying comments to sample param file

* Change webApp to app to align with recommendations

* Reference MySQL username from KV secret

* Fix Bicep linting

* Deploy.sh support & fixes (#47)

* Author: Seokwon Yang <seyan@microsoft.com>
Date:   Fri Sep 15 07:44:52 2023 -0700

    deployment enhancement & fixes

* feedback upate

* Perform root folder cleanup; fixes #39
Rename azDeploySecureSub to main

* Update GH action from Vishal

---------

Co-authored-by: Sven Aelterman <17446043+SvenAelterman@users.noreply.github.com>

* Remove location list in deploy.ps1, main.bicep

* Update README
Add information about deploy.ps1
Remove or comment out outdated text

* Exclude `/` from password characters
Fixes #57

* Add additional storage-related app settings
Fixes #58

* Cleanup

* Fixes #55 and #56

* Addresses #63 but needs more work to ensure reliability of Key Vault refs

* changed based on the last test.

* added manual.md and configuration.md

* Update env var names

* Remove @secret attribute from KV reference params

---------

Co-authored-by: Vishal Kalal <vishal.kalal@outlook.com>
Co-authored-by: kalalvishal <vishal.rajasthan@gmail.com>
Co-authored-by: Sven Aelterman <17446043+SvenAelterman@users.noreply.github.com>
Co-authored-by: Seokwon Yang <seyan@microsoft.com>
Co-authored-by: sjyang18 <41694933+sjyang18@users.noreply.github.com>

.deployment

@@ -1,3 +1,3 @@
 [config]
-command = bash deploy.sh
-SCM_COMMAND_IDLE_TIMEOUT=600
+command = bash scripts/bash/deploy.sh
+SCM_COMMAND_IDLE_TIMEOUT=1200

.github/workflows/bicep-build.yml

@@ -0,0 +1,76 @@
+## deploy azDeploySecureSub.bicep
+
+name: Azure REDCap Deployment
+
+on:
+    workflow_dispatch:
+
+    push:
+        branches:
+            - main
+
+permissions:
+    id-token: write
+    contents: read
+
+env:
+    azCliVersion: 2.30.0
+    environment: 'env-redcap'
+    region: 'eastus'
+
+jobs:
+    # Validate the Bicep templates
+    validateDeployment:
+        runs-on: ubuntu-latest
+
+        steps:
+          - uses: actions/checkout@main
+            name: Checkout
+
+          - uses: azure/login@v1
+            name: Azure Login
+            with:
+              creds: ${{ secrets.AZURE_CREDENTIALS }}
+
+          # Deploy Bicep file
+          - name: validateTemplates
+            uses: azure/arm-deploy@v1
+            with:
+              scope: 'subscription'
+              template: ./main.bicep
+              deploymentMode: 'Validate'
+              region: ${{ env.region }}
+
+          - name: planDeployment
+            uses: azure/arm-deploy@v1
+            with:
+              scope: 'subscription'
+              template: ./main.bicep
+              additionalArguments: "--what-if"
+              region: ${{ env.region }}
+
+    # Deploy the resources
+    deployResources:
+      if: ( github.ref == 'refs/heads/main' )
+      runs-on: ubuntu-latest
+      environment: 'nonProduction' ## Replce with your environment name
+      needs: [
+        validateDeployment
+      ]
+
+      steps:
+          - uses: actions/checkout@main
+            name: Checkout
+
+          - uses: azure/login@v1
+            name: Azure Login
+            with:
+              creds: ${{ secrets.AZURE_CREDENTIALS }}
+
+          # Deploy Bicep file
+          - name: deploy
+            uses: azure/arm-deploy@v1
+            with:
+              scope: 'subscription'
+              template: ./main.bicep
+              region: ${{ env.region }}

.gitignore

@@ -330,3 +330,10 @@ ASALocalRun/
 .mfractor/
 *.sln
 *.deployproj
+
+/*.json
+
+# Exclude Bicep parameter files
+*.bicepparam
+# Except for the sample file
+!/*-sample.bicepparam

.vscode/settings.json

@@ -1,5 +1,5 @@
 {
   "dotnetAcquisitionExtension.existingDotnetPath": [
     "/usr/local/dotnet/current/dotnet"
-  ]
+  ],
 }

Files/settings.ini

@@ -4,7 +4,7 @@ upload_max_filesize = 32M
 post_max_size = 32M
 
 ; Mail settings
-SMTP = 'replace_smtp_server_name'
-smpt_port = replace_smtp_port
-sendmail_from = 'replace_sendmail_from'
-sendmail_path='replace_sendmail_path'
+SMTP = ''
+smtp_port = 
+sendmail_from = ''
+sendmail_path = ''

README.md

@@ -1,11 +1,24 @@
-# ARM Template for REDCap automated deployment in Azure
+# REDCap Deployment on Azure
 
-## Quick Start
+### Overview
+This repository provides you with the necessary resources and guidance to deploy the REDCap application on Microsoft’s Azure cloud platform. This allows you to leverage the power of cloud computing for your research data management needs.
 
-| Description                 | Link                                                                                                                                                                                                                                                        | Azure US Gov Link                                                                                                                                                                                                                                                       |
-| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - |
-| Deploy with your SMTP Relay | [![Deploy To Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2Fazure-redcap-paas%2Fmain%2Fazuredeploy.json)               | [![Deploy To Azure US Gov](https://aka.ms/deploytoazuregovbutton)](https://portal.azure.us/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2Fazure-redcap-paas%2Fmain%2Fazuredeploy.json) |
-| Deploy using SendGrid       | [![Deploy To Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2Fazure-redcap-paas%2Fmain%2Fazuredeploy_with_SendGrid.json) | [![Deploy To Azure US Gov](https://aka.ms/deploytoazuregovbutton)](https://portal.azure.us/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2Fazure-redcap-paas%2Fmain%2Fazuredeploy_with_SendGrid.json) |
+This template automates the deployment of the REDCap solution into Azure using managed PaaS resources. The template assumes you are deploying a version of REDCap that supports direct connection to Azure Blob Storage. If you deploy an older version, deployment will succeed but you will need to manually provision NFS storage in Azure, and delete the new storage account. For NFS, consider:
+
+
+
+### Deployment Options
+  - ### Manual deployment
+
+    - For manual deployment process, please navigate [***here***](manual.md)
+
+  - ### CI/CD Deployment with GitHub
+
+    - Information pending
+
+  - ### CI/CD Deployment with Azure DevOps
+
+    - Information pending
 
 ### Details
 
@@ -15,51 +28,47 @@ This template automates the deployment of the REDCap solution into Azure using m
 - <https://azuremarketplace.microsoft.com/marketplace/apps/softnas.buurst_nas>
 - <https://learn.microsoft.com/samples/azure/azure-quickstart-templates/nfs-ha-cluster-ubuntu/>
 
-To deploy REDCap source to Azure App Service, you must supply your REDCap Community site credentials which the deployment automation will use to pull your copy of the REDCap source directly from the community site.
+To deploy the REDCap source to Azure App Service, you must supply your REDCap Community site credentials. The deployment automation will use them to pull the REDCap source directly from the community site.
 
-> NOTE: These values will be stored within the Azure App Service as configuration settings. Once your deployment has succeeded, you should navigate to your Azure App Service resource and delete or empty out the values so that they aren't stored here.
+> NOTE: These values will be stored within the Azure App Service as configuration settings. Once your deployment has succeeded, you should navigate to your Azure App Service resource and delete or clear the values so that they aren't stored here.
 
 ![Azure App Service](/images/app-settings.png)
 
 <https://projectredcap.org/wp-content/resources/REDCapTechnicalOverview.pdf>
 
-- ARM template deploys the following:
+- The template deploys the following:
   - Azure Web App
   - Azure DB for MySQL (1)
   - Azure Storage Account
-  - (optional) SendGrid 3rd Party Email service (2)
+  - Key Vault
+  - Private DNS zones
+  - Virtual Network
+  - Application Insights
+  <!-- - (optional) SendGrid 3rd Party Email service (2) -->
 
 (1) Review <https://learn.microsoft.com/azure/mysql/flexible-server/concepts-service-tiers-storage> for details on available features, regions, and pricing models for Azure DB for MySQL.
 
-(2) SendGrid is a paid service with a free tier offering 25k messages per month, with additional paid tiers offering more volume, whitelisting, custom domains, etc. There is a limit of two instances per subscription using the free tier. For more information see <https://docs.microsoft.com/en-us/azure/store-sendgrid-php-how-to-send-email#create-a-sendgrid-account>. The service will be accessed initially using the password you enter in the deployment template. You can click "Manage" on the SendGrid service after deployment to administrate the service in their portal, including options to create an API key that can be used for access instead of the password.
+<!--(2) SendGrid is a paid service with a free tier offering 25k messages per month, with additional paid tiers offering more volume, whitelisting, custom domains, etc. There is a limit of two instances per subscription using the free tier. For more information see <https://docs.microsoft.com/en-us/azure/store-sendgrid-php-how-to-send-email#create-a-sendgrid-account>. The service will be accessed initially using the password you enter in the deployment template. You can click "Manage" on the SendGrid service after deployment to administrate the service in their portal, including options to create an API key that can be used for access instead of the password.
 
 If after deployment, you would instead like to use a different SMTP relay, edit the values "smtp_fqdn_name", "smtp_port", "smtp_user_name", and "smtp_password" to point to your preferred endpoint. You can then delete the SendGrid service from this resource group.
 
-If you use Exchange Online (part of the Microsoft 365 Suite), you can follow these steps to set it up and use it as an SMTP relay for this service: <https://learn.microsoft.com/Exchange/mail-flow-best-practices/how-to-set-up-a-multifunction-device-or-application-to-send-email-using-microsoft-365-or-office-365>
+If you use Exchange Online (part of the Microsoft 365 Suite), you can follow these steps to set it up and use it as an SMTP relay for this service: <https://learn.microsoft.com/Exchange/mail-flow-best-practices/how-to-set-up-a-multifunction-device-or-application-to-send-email-using-microsoft-365-or-office-365> -->
 
 ### Setup
 
 This template will automatically deploy the resources necessary to run REDCap in Azure using PaaS (Platform-as-a-Service) features.
 
-**IMPORTANT**: _The "Site Name" you choose will be re-used as part of the storage, website, and MySql database name. Make sure you don't use characters that will be rejected by MySql._
+**IMPORTANT**: _The "Workload Name" you choose will be re-used as part of the storage, website, and MySQL database name. Make sure you don't use characters that will be rejected by MySQL._
 
 After the template is deployed, deployment automation will download the REDCap ZIP file you specify, and install it in your web app. It will then automatically update the database connection information in the app.
 
 > NOTE: The database will not be initialized; therefore, REDCap will not be usable until then. See the [Post-Setup](#post-setup) section below on how to initialize the database.
 
 With the download and unzipping of REDCap application, the entire operation will take between 12-16 minutes.
 
-If you need to connect to the MySQL database using the MySQL client, you will need to open the firewall to your managed MySQL instance and allow connections from the location where you will run the client. Here are the instructions:
-<https://docs.microsoft.com/en-us/azure/mysql/quickstart-create-mysql-server-database-using-azure-portal#configure-a-server-level-firewall-rule>
-
-(Add your current IP address by clicking "+ Add My IP")
-
-Once you've opened the firewall, you will need your database name. The credentials are those you supplied in this template. The name is available from the portal where you updated the firewall rules:
-
-![alt text][mysql]
+If you need to connect to the MySQL database using the MySQL client, you will need to deploy a Virtual Machine with Bastion or AVD to the virtual network to run the client.
 
-Please also review:
-<https://learn.microsoft.com/azure/mysql/flexible-server/how-to-connect-tls-ssl>
+The database user name defaults to `sqladmin` and the password is a random string of 25 characters. The password is stored in Key Vault.
 
 ### Post-Setup
 
@@ -79,13 +88,13 @@ bash install.sh
 
 It will take a few minutes to execute the SQL.
 
-Once you regain access to the console, you can navigate to the root of your app service and confirm everything shows green on the REDCap Configuration Check page - with the exception of CronJob status which you may have to manually invoke. If anything displays on that page in red or yellow, it is recommended that you perform a "Restart" of the Azure "App Service". This needs to be done due to the fact that some necessary server environment settings get changed after the initial deployment, but restarting the App Service will load the service with the intended settings. Everything should be fine after that initial restart though.
+Once you regain access to the console, you can navigate to the root of your app service and confirm everything shows green on the REDCap Configuration Check page - with the exception of CronJob status which you may have to manually invoke. If anything displays on that page in red or yellow, it is recommended that you perform a "Restart" of the Azure "App Service". This needs to be done due to the fact that some necessary server environment settings get changed after the initial deployment, but restarting the App Service will load the service with the intended settings.
 
-### Note about REDCap "Easy Upgade"
+## Note about REDCap "Easy Upgade"
 
 The "Easy Upgrade" feature in REDCap 8.11.0 and later is currently _not_ supported when deploying a REDCap instance on Azure. Support for "Easy Upgrade" on Azure is expected to come at a later time in a future REDCap release.
 
-### Resources
+## Resources
 
 - App Services overview
   <https://learn.microsoft.com/azure/app-service/overview>