From 81b237db48dc66bff029008a0a1fa7ae58d7a1f8 Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 12 Jan 2017 11:08:34 -0600 Subject: [PATCH 01/16] Updating Azure Automation SQL DB Export instructions to work correctly, testing view in github --- .../azure-automation-automated-export/README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index d06ce6938c..8e6ece1a87 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -12,11 +12,13 @@ Provides the scripts and lists the steps to set up automatically exporting your 1. Create and uploade the certificates that you will use to authenticate your connection to azure. - Run powershell as admin. - - Run the New-SelfSignedCertificate command: New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName + - Run the New-SelfSignedCertificate command: $cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName <certificateName> + - Export the certificate as a .cer file + - Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath <PathAndFileName>.cer - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: - - $CertPassword = ConvertTo-SecureString -String -Force -AsPlainText - - Export-PfxCertificate -Cert cert:\localmachine\my\ -FilePath .pfx -Password $CertPassword - - Upload the .cer file to your subscription [here][https://manage.windowsazure.com/] + - $CertPassword = ConvertTo-SecureString -String <YourPassword> -Force -AsPlainText + - Export-PfxCertificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath <PathAndFileName>.pfx -Password $CertPassword + - Upload the .cer file to your subscription [in the old portal](https://manage.windowsazure.com/) - Upload the .pfx file to the certificates under Assets in the automation account that you want to use on Azure. You will use the password you gave in the previous step to authenticate it. 2. Create new a new credentials asset to authenticate your server with. - Under assets, click on Credentials, and then click on Add a credential. @@ -25,7 +27,7 @@ Provides the scripts and lists the steps to set up automatically exporting your - Under assets, click on variables and then Add a variable. - Give the value of the storage key and you can make it encrypted so that only Azure Automation can read the variable and it won't show the key in plaintext if someone looks at the variable. 4. Set Up Log Analytics (OMS) and Alerts - - If you don't have Log Analytics set up on your Azure account, follow [these][https://azure.microsoft.com/en-us/documentation/articles/automation-manage-send-joblogs-log-analytics/] instructions for setting it up. + - If you don't have Log Analytics set up on your Azure account, follow [these](https://azure.microsoft.com/en-us/documentation/articles/automation-manage-send-joblogs-log-analytics/) instructions for setting it up. 5. Set Up Log Analytics Alerts - To send yourself an email if an error occurs or one of the jobs fails, you need to set up alerts. - Select your log analytics account that you want to use in the azure portal and click on the OMS Portal box under Management. @@ -50,4 +52,4 @@ Provides the scripts and lists the steps to set up automatically exporting your 2. In AutoExportBlobRetention, here are the values that need to be modified: - -Name for Get-AzureAutomationVariable: This is the AutomationAccount you created the StorageKey variable under (probably the same one you are running the RunBook under) and -Name is the name of the variable. - $storageContainer: This is the name of the storage container where you will be monitoring the exported blobs. - - $retentionInDays: This is how many days you want to keep the exported blobs stored for before deleting. \ No newline at end of file + - $retentionInDays: This is how many days you want to keep the exported blobs stored for before deleting. From 6b68afd5155b0743541800afff9b9feb6cc9395c Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 12 Jan 2017 11:11:52 -0600 Subject: [PATCH 02/16] Test inline code --- samples/manage/azure-automation-automated-export/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index 8e6ece1a87..47d0b43fde 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -12,9 +12,9 @@ Provides the scripts and lists the steps to set up automatically exporting your 1. Create and uploade the certificates that you will use to authenticate your connection to azure. - Run powershell as admin. - - Run the New-SelfSignedCertificate command: $cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName <certificateName> + - Run the New-SelfSignedCertificate command: `$cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName ` - Export the certificate as a .cer file - - Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath <PathAndFileName>.cer + - `Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .cer` - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: - $CertPassword = ConvertTo-SecureString -String <YourPassword> -Force -AsPlainText - Export-PfxCertificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath <PathAndFileName>.pfx -Password $CertPassword From bec643a5732b4dd58c9f770038e138339a51fe57 Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 12 Jan 2017 11:13:47 -0600 Subject: [PATCH 03/16] Test inline code blocks --- samples/manage/azure-automation-automated-export/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index 47d0b43fde..95fa5645f4 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -12,7 +12,10 @@ Provides the scripts and lists the steps to set up automatically exporting your 1. Create and uploade the certificates that you will use to authenticate your connection to azure. - Run powershell as admin. - - Run the New-SelfSignedCertificate command: `$cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName ` + - Run the New-SelfSignedCertificate command: + ```powershell +$cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName +``` - Export the certificate as a .cer file - `Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .cer` - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: From 0a2cdd604089f84f3d3ae0f921f6a1ff02986d6a Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 12 Jan 2017 11:15:56 -0600 Subject: [PATCH 04/16] Convert to code blocks for syntax highlight --- .../azure-automation-automated-export/README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index 95fa5645f4..54fced0bfc 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -13,14 +13,18 @@ Provides the scripts and lists the steps to set up automatically exporting your 1. Create and uploade the certificates that you will use to authenticate your connection to azure. - Run powershell as admin. - Run the New-SelfSignedCertificate command: - ```powershell +```powershell $cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName ``` - Export the certificate as a .cer file - - `Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .cer` +```powershell +Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .cer +``` - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: - - $CertPassword = ConvertTo-SecureString -String <YourPassword> -Force -AsPlainText - - Export-PfxCertificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath <PathAndFileName>.pfx -Password $CertPassword +```powershell +$CertPassword = ConvertTo-SecureString -String -Force -AsPlainText +Export-PfxCertificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .pfx -Password $CertPassword +``` - Upload the .cer file to your subscription [in the old portal](https://manage.windowsazure.com/) - Upload the .pfx file to the certificates under Assets in the automation account that you want to use on Azure. You will use the password you gave in the previous step to authenticate it. 2. Create new a new credentials asset to authenticate your server with. From e37685dc06701b3203dc8aa33f8db4b90bdda9c5 Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 12 Jan 2017 11:17:36 -0600 Subject: [PATCH 05/16] Convert to code blocks for syntax highlight --- samples/manage/azure-automation-automated-export/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index 54fced0bfc..e4d2f0383e 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -16,15 +16,18 @@ Provides the scripts and lists the steps to set up automatically exporting your ```powershell $cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName ``` + - Export the certificate as a .cer file ```powershell Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .cer ``` + - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: ```powershell $CertPassword = ConvertTo-SecureString -String -Force -AsPlainText Export-PfxCertificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .pfx -Password $CertPassword ``` + - Upload the .cer file to your subscription [in the old portal](https://manage.windowsazure.com/) - Upload the .pfx file to the certificates under Assets in the automation account that you want to use on Azure. You will use the password you gave in the previous step to authenticate it. 2. Create new a new credentials asset to authenticate your server with. From b1499c58041b7a9e6cca43fe1018a013ebf97972 Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 12 Jan 2017 11:31:33 -0600 Subject: [PATCH 06/16] Convert to code blocks for syntax highlight --- samples/manage/azure-automation-automated-export/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index e4d2f0383e..d455bb83d5 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -13,16 +13,19 @@ Provides the scripts and lists the steps to set up automatically exporting your 1. Create and uploade the certificates that you will use to authenticate your connection to azure. - Run powershell as admin. - Run the New-SelfSignedCertificate command: + ```powershell $cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName ``` - Export the certificate as a .cer file + ```powershell Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .cer ``` - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: + ```powershell $CertPassword = ConvertTo-SecureString -String -Force -AsPlainText Export-PfxCertificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .pfx -Password $CertPassword From df1a424c1713f0b2adc615c5678355f855ec5cca Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 12 Jan 2017 11:40:06 -0600 Subject: [PATCH 07/16] Add tabs to fix list issue with code blocks --- .../azure-automation-automated-export/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index d455bb83d5..7a71850813 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -14,22 +14,22 @@ Provides the scripts and lists the steps to set up automatically exporting your - Run powershell as admin. - Run the New-SelfSignedCertificate command: -```powershell + ```powershell $cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName -``` + ``` - Export the certificate as a .cer file -```powershell + ```powershell Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .cer -``` + ``` - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: -```powershell + ```powershell $CertPassword = ConvertTo-SecureString -String -Force -AsPlainText Export-PfxCertificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .pfx -Password $CertPassword -``` + ``` - Upload the .cer file to your subscription [in the old portal](https://manage.windowsazure.com/) - Upload the .pfx file to the certificates under Assets in the automation account that you want to use on Azure. You will use the password you gave in the previous step to authenticate it. From 6b86abe1911de5469bbf3344992d1e1b78c1514f Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 16 Mar 2017 14:15:22 -0500 Subject: [PATCH 08/16] Test code blocks --- .../manage/azure-automation-automated-export/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index 7a71850813..08dcdfd0b2 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -14,15 +14,15 @@ Provides the scripts and lists the steps to set up automatically exporting your - Run powershell as admin. - Run the New-SelfSignedCertificate command: - ```powershell +```powershell $cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName - ``` +``` - Export the certificate as a .cer file - ```powershell +```powershell Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .cer - ``` +``` - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: From dafe8b85b5c97475027554952dcfcb995d713962 Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 16 Mar 2017 14:18:49 -0500 Subject: [PATCH 09/16] Test inline --- .../manage/azure-automation-automated-export/README.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index 08dcdfd0b2..09dc26528d 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -14,15 +14,12 @@ Provides the scripts and lists the steps to set up automatically exporting your - Run powershell as admin. - Run the New-SelfSignedCertificate command: -```powershell -$cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName -``` + + `$cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName ` - Export the certificate as a .cer file -```powershell -Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .cer -``` + `Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .cer` - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: From a16cf0f3f854fb375c14a5ed5705554709ec08cf Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 16 Mar 2017 14:21:06 -0500 Subject: [PATCH 10/16] Edits for proper authentication to azure automation --- samples/manage/azure-automation-automated-export/README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index 09dc26528d..8df38c2dab 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -23,10 +23,8 @@ Provides the scripts and lists the steps to set up automatically exporting your - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: - ```powershell -$CertPassword = ConvertTo-SecureString -String -Force -AsPlainText -Export-PfxCertificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .pfx -Password $CertPassword - ``` + `$CertPassword = ConvertTo-SecureString -String -Force -AsPlainText` + `Export-PfxCertificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .pfx -Password $CertPassword` - Upload the .cer file to your subscription [in the old portal](https://manage.windowsazure.com/) - Upload the .pfx file to the certificates under Assets in the automation account that you want to use on Azure. You will use the password you gave in the previous step to authenticate it. From 4fb964bec69ad3534f658a36b2e12a7f2c9e26ab Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 16 Mar 2017 14:22:01 -0500 Subject: [PATCH 11/16] Edits for proper authentication to azure automation --- samples/manage/azure-automation-automated-export/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index 8df38c2dab..5ad0febb25 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -24,6 +24,7 @@ Provides the scripts and lists the steps to set up automatically exporting your - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: `$CertPassword = ConvertTo-SecureString -String -Force -AsPlainText` + `Export-PfxCertificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .pfx -Password $CertPassword` - Upload the .cer file to your subscription [in the old portal](https://manage.windowsazure.com/) From b6878603669ca584e036b9fddadae94329f635bd Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 16 Mar 2017 15:39:15 -0500 Subject: [PATCH 12/16] Moving to new authentication method, more in depth documentation --- .../AutoExport.ps1 | 33 ++++++++++---- .../README.md | 45 ++++++------------- 2 files changed, 37 insertions(+), 41 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/AutoExport.ps1 b/samples/manage/azure-automation-automated-export/AutoExport.ps1 index 37cb872d33..e95961587b 100644 --- a/samples/manage/azure-automation-automated-export/AutoExport.ps1 +++ b/samples/manage/azure-automation-automated-export/AutoExport.ps1 @@ -31,11 +31,12 @@ $retryLimit = 5; # The number of minutes you want to wait for an operation to finish before you fail. $waitInMinutes = 30; +# Connection Asset Name for Authenticating (Keep as AzureClassicRunAsConnection if you created the default RunAs accounts) +$connectionAssetName = "AzureClassicRunAsConnection"; + $storageKeyVariableName = "STORAGEKEYVARIABLENAME"; $storageAccountName = "STORAGEACCOUNTNAME"; -$automationCertificateName = "CERTIFICATENAME"; -$subId = "00000000-0000-0000-0000-000000000000"; -$subName = "SUBSCRIPTIONNAME"; +$storageContainerName = "STORAGECONTAINERNAME"; function LogMessage($message) { @@ -147,7 +148,7 @@ function StartExport($dbObj) # Get the storage context. $stgctx = New-AzureStorageContext -StorageAccountName $global:storageAccountName -StorageAccountKey $storageKey; # Start the export. If there is an error, stop the export and set the state to ToDrop. - $dbObj.Export = Start-AzureSqlDatabaseExport -SqlConnectionContext $ctx -StorageContext $stgctx -StorageContainerName autoexportcontainer -DatabaseName $dbObj.DatabaseCopyName -BlobName $blobName; + $dbObj.Export = Start-AzureSqlDatabaseExport -SqlConnectionContext $ctx -StorageContext $stgctx -StorageContainerName $global:storageContainerName -DatabaseName $dbObj.DatabaseCopyName -BlobName $blobName; # $? is true if the last command succeeded and false if the last command failed. If it is false, go to the ToDrop state. if (-not $? -and $global:retryLimit -ile $dbObj.RetryCount) { @@ -266,11 +267,25 @@ function ExportProcess } } -# Get the certificate to authenticate the subscription -$cert = Get-AutomationCertificate -Name $global:automationCertificateName; -# Set the subscription to use -Set-AzureSubscription -SubscriptionName $global:subName -Certificate $cert -SubscriptionId $global:subID; -Select-AzureSubscription -Current $global:subName; +# Authenticate to Azure with certificate +Write-Verbose "Get connection asset: $connectionAssetName" -Verbose; +$Conn = Get-AutomationConnection -Name $connectionAssetName; +if ($Conn -eq $null) +{ + throw "Could not retrieve connection asset: $connectionAssetName. Assure that this asset exists in the Automation account."; +} + +$CertificateAssetName = $Conn.CertificateAssetName; +Write-Verbose "Getting the certificate: $CertificateAssetName" -Verbose; +$AzureCert = Get-AutomationCertificate -Name $CertificateAssetName; +if ($AzureCert -eq $null) +{ + throw "Could not retrieve certificate asset: $CertificateAssetName. Assure that this asset exists in the Automation account."; +} + +Write-Verbose "Authenticating to Azure with certificate." -Verbose; +Set-AzureSubscription -SubscriptionName $Conn.SubscriptionName -SubscriptionId $Conn.SubscriptionID -Certificate $AzureCert; +Select-AzureSubscription -SubscriptionId $Conn.SubscriptionID; $currentIndex = 0; for($currentRun = 0; $currentRun -lt ([math]::Ceiling($databaseServerPairs.Length/$batchingLimit)); $currentRun++) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index 5ad0febb25..1ac4b8aaf8 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -8,35 +8,18 @@ author: trgrie-msft Provides the scripts and lists the steps to set up automatically exporting your databases to Azure Storage with Azure Automation. -## Azure Automation Set Up - -1. Create and uploade the certificates that you will use to authenticate your connection to azure. - - Run powershell as admin. - - Run the New-SelfSignedCertificate command: - - - `$cert = New-SelfSignedCertificate -CertStoreLocation cert:\localmachine\my -DnsName ` - - - Export the certificate as a .cer file - - `Export-Certificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .cer` - - - Create a corresponding pfx certificate by taking the thumbprint of the newly created certificate and running these commands: - - `$CertPassword = ConvertTo-SecureString -String -Force -AsPlainText` - - `Export-PfxCertificate -Cert "cert:\localmachine\my\$($cert.Thumbprint)" -FilePath .pfx -Password $CertPassword` - - - Upload the .cer file to your subscription [in the old portal](https://manage.windowsazure.com/) - - Upload the .pfx file to the certificates under Assets in the automation account that you want to use on Azure. You will use the password you gave in the previous step to authenticate it. -2. Create new a new credentials asset to authenticate your server with. - - Under assets, click on Credentials, and then click on Add a credential. - - Name the credential and give the username and password that you will be logging into the server with. -3. Create a new variable asset to pass the storage key of the Azure storage account you will be using. - - Under assets, click on variables and then Add a variable. - - Give the value of the storage key and you can make it encrypted so that only Azure Automation can read the variable and it won't show the key in plaintext if someone looks at the variable. -4. Set Up Log Analytics (OMS) and Alerts - - If you don't have Log Analytics set up on your Azure account, follow [these](https://azure.microsoft.com/en-us/documentation/articles/automation-manage-send-joblogs-log-analytics/) instructions for setting it up. +## Prerequisite Set Up + +1. Create and set up your Azure Automation Account + - Create an Azure Automation Account by [following the instructions here](https://docs.microsoft.com/en-us/azure/automation/automation-sec-configure-azure-runas-account). +2. Add Azure Automation Credential assets for your SQL Azure servers + - Create your Automation Credential for each of your SQL Azure servers you intend to export by [following the instructions here](https://docs.microsoft.com/en-us/azure/automation/automation-credentials#creating-a-new-credential-asset). +3. Create the Azure Storage Account to hold your bacpac files + - Create the Storage Account by [following the instructions here](https://docs.microsoft.com/en-us/azure/storage/storage-create-storage-account#create-a-storage-account). + - Copy your Storage Account access keys by [following the instructions here](https://docs.microsoft.com/en-us/azure/storage/storage-create-storage-account#view-and-copy-storage-access-keys). + - Create an Azure Automation string Variable asset for your Storage Account access key by [following the instructions here](https://docs.microsoft.com/en-us/azure/automation/automation-variables#creating-an-automation-variable). +4. Set Up Log Analytics (OMS) and Alerts (optional for alerting) + - If you don't have Log Analytics set up on your Azure account, [follow these](https://docs.microsoft.com/en-us/azure/automation/automation-manage-send-joblogs-log-analytics) instructions for setting it up. 5. Set Up Log Analytics Alerts - To send yourself an email if an error occurs or one of the jobs fails, you need to set up alerts. - Select your log analytics account that you want to use in the azure portal and click on the OMS Portal box under Management. @@ -55,9 +38,7 @@ Provides the scripts and lists the steps to set up automatically exporting your - $waitTimeInMinutes: This tells the script how long it can wait for an operation to complete before it fails. - $storageKeyVariableName: This is the AutomationAccount you created the StorageKey variable under (probably the same one you are running the RunBook under) and -Name is the name of the variable. - $storageAccountName: This is the name of the storage account you are exporting to. - - $automationCertificateName for Get-AutomationCertificate: This is the name of the certificate you setup to authenticate with Azure. - - $subId: The ID of the subscription you are using. This will be used to tell Azure Automation which subscription to use. - - $subName: The name of the subscription you are using. This will be used to tell Azure Automation which subscription to use. + - $connectionAssetName: Connection Asset Name for Authenticating (Keep as AzureClassicRunAsConnection if you created the default RunAs accounts) 2. In AutoExportBlobRetention, here are the values that need to be modified: - -Name for Get-AzureAutomationVariable: This is the AutomationAccount you created the StorageKey variable under (probably the same one you are running the RunBook under) and -Name is the name of the variable. - $storageContainer: This is the name of the storage container where you will be monitoring the exported blobs. From 47205a456ffe9c975ce93326a0f24fdff2e1f410 Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 16 Mar 2017 15:43:53 -0500 Subject: [PATCH 13/16] aligning AutoExportBlobRetention documentation and script to AutoExport script --- .../AutoExportBlobRetention.ps1 | 23 ++++++++++++------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/AutoExportBlobRetention.ps1 b/samples/manage/azure-automation-automated-export/AutoExportBlobRetention.ps1 index 7840a092a4..89d257468e 100644 --- a/samples/manage/azure-automation-automated-export/AutoExportBlobRetention.ps1 +++ b/samples/manage/azure-automation-automated-export/AutoExportBlobRetention.ps1 @@ -1,14 +1,21 @@ -# The storage key for the storage account you are using. -$storageKey = Get-AutomationVariable -Name "STORAGEKEYVARIABLENAME"; -# The name of the storage container you are using. -$storageContainer = "STORAGECONTAINERNAME"; -# Set up the storage context for the storage account. -$context = New-AzureStorageContext -StorageAccountName "STORAGEACCOUNTNAME" -StorageAccountKey $storageKey -# Get all of the blobs in the storage account. -$blobs = Get-AzureStorageBlob -Container $storageContainer -Context $context +#Azure Automation String Variable name for your Storage Account Key +$storageKeyVariableName = "STORAGEKEYVARIABLENAME"; +#Name of your Storage Account +$storageAccountName = "STORAGEACCOUNTNAME"; +#Name of your Storage Container +$storageContainerName = "STORAGECONTAINERNAME"; # Set the number of days that you want the blob to be stored for. $retentionInDays = 30 + +# Get the storage key +$storageKey = Get-AutomationVariable -Name $storageKeyVariableName; +# Set up the storage context for the storage account. +$context = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageKey +# Get all of the blobs in the storage account. +$blobs = Get-AzureStorageBlob -Container $storageContainerName -Context $context + + foreach($blob in $blobs) { # Get the current time to compare to the time that the blob was created. From f6a3ce33099457a0e4accd32e4f849de926f1601 Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 16 Mar 2017 16:37:29 -0500 Subject: [PATCH 14/16] Finished out documentation with how to add the script to Azure Automation --- .../AutoExport.ps1 | 14 +++++++---- .../README.md | 23 ++++++++++++++++--- 2 files changed, 29 insertions(+), 8 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/AutoExport.ps1 b/samples/manage/azure-automation-automated-export/AutoExport.ps1 index e95961587b..08ea71e0d1 100644 --- a/samples/manage/azure-automation-automated-export/AutoExport.ps1 +++ b/samples/manage/azure-automation-automated-export/AutoExport.ps1 @@ -18,11 +18,16 @@ Add-Type -TypeDefinition @" $databaseServerPairs = @([pscustomobject]@{serverName="SAMPLESERVER1";databaseName="SAMPLEDATABASE1"}, [pscustomobject]@{serverName="SAMPLESERVER1";databaseName="SAMPLEDATABASE2"}, - [pscustomobject]@{serverName="SAMPLESERVER2";databaseName="SAMPLEDATABASE3"}); + [pscustomobject]@{serverName="SAMPLESERVER2";databaseName="SAMPLEDATABASE3"} + ); -$serverCred = Get-AutomationPSCredential -Name 'NAMEOFSERVERCREDENTIAL1'; +# The Credentials for the database servers +$serverCred1 = Get-AutomationPSCredential -Name 'NAMEOFSERVERCREDENTIAL1'; $serverCred2 = Get-AutomationPSCredential -Name 'NAMEOFSERVERCREDENTIAL2'; -$serverCredentialsDictionary = @{'SAMPLESERVER1'=$serverCred;'SAMPLESERVER2'=$serverCred2} +$serverCredentialsDictionary = @{ + 'SAMPLESERVER1'=$serverCred1; + 'SAMPLESERVER2'=$serverCred2; + } # The number of databases you want to have running at the same time. $batchingLimit = 10; @@ -34,6 +39,7 @@ $waitInMinutes = 30; # Connection Asset Name for Authenticating (Keep as AzureClassicRunAsConnection if you created the default RunAs accounts) $connectionAssetName = "AzureClassicRunAsConnection"; + $storageKeyVariableName = "STORAGEKEYVARIABLENAME"; $storageAccountName = "STORAGEACCOUNTNAME"; $storageContainerName = "STORAGECONTAINERNAME"; @@ -134,8 +140,6 @@ function CheckCopy($dbObj) # This function starts the export. If there is an error, we set the state to ToDrop. Otherwise, we set the state to Exporting. function StartExport($dbObj) { - # Setup the server connection that the storage account is on. - $serverManageUrl = "https://autoexportserver.database.windows.net"; # Get the current time to use as a unique identifier for the blob name. $currentTime = Get-Date -format "_yyyy-MM-dd_HH:mm.ss"; $blobName = $dbObj.DatabaseName + "_ExportBlob" + $currentTime; diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index 1ac4b8aaf8..f56cb832e5 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -30,16 +30,33 @@ Provides the scripts and lists the steps to set up automatically exporting your ## Script Set Up +Save the AutoExport.ps1 and AutoExportBlobRetention.ps1 files locally to make these edits + 1. In the AutoExport.ps1 script, here are the values that need to be modified: - $databaseServerPairs: This is where you put in the names of the databases you want to export along with the name of the server they are on. + Add them in the format: `[pscustomobject]@{serverName="SAMPLESERVER1";databaseName="SAMPLEDATABASE1"}` make sure to comma separate the items - $serverCredentialsDictionary: If you are backing up from multiple servers, you can setup all of the credentials here and look them up by the server’s name later. + Add a $serverCred variable in the format `$serverCred1 = Get-AutomationPSCredential -Name 'NAMEOFSERVERCREDENTIAL1';` for each Azure Automation Credential you created. Increment the variable name (eg. $serverCred2 $serverCred3) for each one. + Add the $serverCreds to the dictionary in the format `'SAMPLESERVERNAME1'=$serverCred1;` - $batchingLimit: This tells the script how many databases can be worked on at the same time (basically, the maximum number of database copies that there will be at once). - $retryLimit: This tells the script how many times it can retry an operation. - $waitTimeInMinutes: This tells the script how long it can wait for an operation to complete before it fails. - - $storageKeyVariableName: This is the AutomationAccount you created the StorageKey variable under (probably the same one you are running the RunBook under) and -Name is the name of the variable. + - $storageKeyVariableName: This is the Azure Automation string Variable name you created to store your Storage Key. - $storageAccountName: This is the name of the storage account you are exporting to. - $connectionAssetName: Connection Asset Name for Authenticating (Keep as AzureClassicRunAsConnection if you created the default RunAs accounts) 2. In AutoExportBlobRetention, here are the values that need to be modified: - - -Name for Get-AzureAutomationVariable: This is the AutomationAccount you created the StorageKey variable under (probably the same one you are running the RunBook under) and -Name is the name of the variable. - - $storageContainer: This is the name of the storage container where you will be monitoring the exported blobs. + - $storageKeyVariableName: This is the Azure Automation string Variable name you created to store your Storage Key. + - $storageAccountName: This is the name of your Storage Account you exported your bacpacs to. + - $storageContainerName: This is the name of the storage container where you will be monitoring the exported blobs. - $retentionInDays: This is how many days you want to keep the exported blobs stored for before deleting. + +## Adding the Script to Azure Automation + +1. Import the scripts as Azure Automation Runbooks + - Create runbooks from the scripts you editted above by [following the instructions here](https://docs.microsoft.com/en-us/azure/automation/automation-creating-importing-runbook#to-import-a-runbook-from-a-file-with-the-azure-portal) for both scripts. + - [Make sure to publish the runbook.](https://docs.microsoft.com/en-us/azure/automation/automation-creating-importing-runbook#to-publish-a-runbook-using-the-azure-portal) +2. Add a schedule for your Automated Export runbook + - Create a recurring schedule by [following the instructions here](https://docs.microsoft.com/en-us/azure/automation/automation-schedules#to-create-a-new-schedule-in-the-azure-portal). + - Link the schedule(s) you created to the runbooks by [following the instructions here](https://docs.microsoft.com/en-us/azure/automation/automation-schedules#to-link-a-schedule-to-a-runbook-with-the-azure-portal). + +You should now be all set up for Automated Exports into blob storage of your selected SQL Azure databases. \ No newline at end of file From 83fe1ad49cabdd9aff0ff894b5c721fa86af75bb Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Thu, 16 Mar 2017 16:40:07 -0500 Subject: [PATCH 15/16] Bold variable names --- .../README.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index f56cb832e5..9e53c2b76c 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -33,22 +33,22 @@ Provides the scripts and lists the steps to set up automatically exporting your Save the AutoExport.ps1 and AutoExportBlobRetention.ps1 files locally to make these edits 1. In the AutoExport.ps1 script, here are the values that need to be modified: - - $databaseServerPairs: This is where you put in the names of the databases you want to export along with the name of the server they are on. + - **$databaseServerPairs:** This is where you put in the names of the databases you want to export along with the name of the server they are on. Add them in the format: `[pscustomobject]@{serverName="SAMPLESERVER1";databaseName="SAMPLEDATABASE1"}` make sure to comma separate the items - - $serverCredentialsDictionary: If you are backing up from multiple servers, you can setup all of the credentials here and look them up by the server’s name later. + - **$serverCredentialsDictionary:** If you are backing up from multiple servers, you can setup all of the credentials here and look them up by the server’s name later. Add a $serverCred variable in the format `$serverCred1 = Get-AutomationPSCredential -Name 'NAMEOFSERVERCREDENTIAL1';` for each Azure Automation Credential you created. Increment the variable name (eg. $serverCred2 $serverCred3) for each one. Add the $serverCreds to the dictionary in the format `'SAMPLESERVERNAME1'=$serverCred1;` - - $batchingLimit: This tells the script how many databases can be worked on at the same time (basically, the maximum number of database copies that there will be at once). - - $retryLimit: This tells the script how many times it can retry an operation. - - $waitTimeInMinutes: This tells the script how long it can wait for an operation to complete before it fails. - - $storageKeyVariableName: This is the Azure Automation string Variable name you created to store your Storage Key. - - $storageAccountName: This is the name of the storage account you are exporting to. - - $connectionAssetName: Connection Asset Name for Authenticating (Keep as AzureClassicRunAsConnection if you created the default RunAs accounts) + - **$batchingLimit:** This tells the script how many databases can be worked on at the same time (basically, the maximum number of database copies that there will be at once). + - **$retryLimit:** This tells the script how many times it can retry an operation. + - **$waitTimeInMinutes:** This tells the script how long it can wait for an operation to complete before it fails. + - **$storageKeyVariableName:** This is the Azure Automation string Variable name you created to store your Storage Key. + - **$storageAccountName:** This is the name of the storage account you are exporting to. + - **$connectionAssetName:** Connection Asset Name for Authenticating (Keep as AzureClassicRunAsConnection if you created the default RunAs accounts) 2. In AutoExportBlobRetention, here are the values that need to be modified: - - $storageKeyVariableName: This is the Azure Automation string Variable name you created to store your Storage Key. - - $storageAccountName: This is the name of your Storage Account you exported your bacpacs to. - - $storageContainerName: This is the name of the storage container where you will be monitoring the exported blobs. - - $retentionInDays: This is how many days you want to keep the exported blobs stored for before deleting. + - **$storageKeyVariableName:** This is the Azure Automation string Variable name you created to store your Storage Key. + - **$storageAccountName:** This is the name of your Storage Account you exported your bacpacs to. + - **$storageContainerName:** This is the name of the storage container where you will be monitoring the exported blobs. + - **retentionInDays:** This is how many days you want to keep the exported blobs stored for before deleting. ## Adding the Script to Azure Automation From 5935502b36c5513d131df8409c35b67ea50086d2 Mon Sep 17 00:00:00 2001 From: Cameron Battagler Date: Fri, 17 Mar 2017 15:35:16 -0500 Subject: [PATCH 16/16] Added instructions for adding the Blob Storage container, added better information messages, added edits suggested in pull request #199 --- .../AutoExport.ps1 | 33 +++++++++---------- .../README.md | 5 +++ 2 files changed, 21 insertions(+), 17 deletions(-) diff --git a/samples/manage/azure-automation-automated-export/AutoExport.ps1 b/samples/manage/azure-automation-automated-export/AutoExport.ps1 index 08ea71e0d1..14b0cea2a2 100644 --- a/samples/manage/azure-automation-automated-export/AutoExport.ps1 +++ b/samples/manage/azure-automation-automated-export/AutoExport.ps1 @@ -39,7 +39,6 @@ $waitInMinutes = 30; # Connection Asset Name for Authenticating (Keep as AzureClassicRunAsConnection if you created the default RunAs accounts) $connectionAssetName = "AzureClassicRunAsConnection"; - $storageKeyVariableName = "STORAGEKEYVARIABLENAME"; $storageAccountName = "STORAGEACCOUNTNAME"; $storageContainerName = "STORAGECONTAINERNAME"; @@ -144,9 +143,9 @@ function StartExport($dbObj) $currentTime = Get-Date -format "_yyyy-MM-dd_HH:mm.ss"; $blobName = $dbObj.DatabaseName + "_ExportBlob" + $currentTime; # Use the stored credential to create a server credential to use to login to the server. - $servercredential = $global:serverCredentialsDictionary[$dbObj.ServerName]; + $serverCredential = $global:serverCredentialsDictionary[$dbObj.ServerName]; # Set up a SQL connection context to use when exporting. - $ctx = New-AzureSqlDatabaseServerContext -ServerName $dbObj.ServerName -Credential $servercredential; + $ctx = New-AzureSqlDatabaseServerContext -ServerName $dbObj.ServerName -Credential $serverCredential; # Get the storage key to setup the storage context. $storageKey = Get-AutomationVariable -Name $global:storageKeyVariableName; # Get the storage context. @@ -171,7 +170,7 @@ function StartExport($dbObj) } # Set the state to Exporting. $dbObj.DatabaseState = ([DatabaseState]::Exporting); - LogMessage ("Exporting " + $dbObj.DatabaseCopyName); + LogMessage ("Exporting " + $dbObj.DatabaseCopyName + " with RequestID: " + $dbObj.Export.RequestGuid); $dbObj.OperationStartTime = Get-Date; } @@ -192,7 +191,7 @@ function CheckExport($dbObj) { # If the status is "Failed" and we have more retries left, try to export the database copy again. LogMessage ("The last export failed on database " + $dbObj.DatabaseName + ", going back to ToExport state to try again"); - LogMessage $check + LogMessage $check.ErrorMessage $dbObj.DatabaseState = ([DatabaseState]::ToExport); $dbObj.RetryCount++; return; @@ -229,7 +228,7 @@ function ExportProcess $dbsToCopy = $global:dbs | Where-Object DatabaseState -eq ([DatabaseState]::ToCopy); for($i = 0; $i -lt $dbsToCopy.Count; $i++) { - LogMessage $dbsToCopy[$i]; + LogMessage "Database Name: $($dbsToCopy[$i].DatabaseName) State: $($dbsToCopy[$i].DatabaseState) Retry Count: $($dbsToCopy[$i].RetryCount)"; StartCopy($dbsToCopy[$i]); } @@ -244,7 +243,7 @@ function ExportProcess $dbsToExport = $global:dbs | Where-Object DatabaseState -eq ([DatabaseState]::ToExport); for($i = 0; $i -lt $dbsToExport.Count; $i++) { - LogMessage $dbsToExport[$i]; + LogMessage "Database Name: $($dbsToExport[$i].DatabaseName) State: $($dbsToExport[$i].DatabaseState) Retry Count: $($dbsToExport[$i].RetryCount)"; StartExport($dbsToExport[$i]); } @@ -259,7 +258,7 @@ function ExportProcess $dbsToDrop = $global:dbs | Where-Object DatabaseState -eq ([DatabaseState]::ToDrop); for($i = 0; $i -lt $dbsToDrop.Count; $i++) { - LogMessage $dbsToDrop[$i]; + LogMessage "Database Name: $($dbsToDrop[$i].DatabaseName) State: $($dbsToDrop[$i].DatabaseState) Retry Count: $($dbsToDrop[$i].RetryCount)"; StartDrop($dbsToDrop[$i]); } @@ -273,23 +272,23 @@ function ExportProcess # Authenticate to Azure with certificate Write-Verbose "Get connection asset: $connectionAssetName" -Verbose; -$Conn = Get-AutomationConnection -Name $connectionAssetName; -if ($Conn -eq $null) +$automationConnection = Get-AutomationConnection -Name $connectionAssetName; +if ($automationConnection -eq $null) { throw "Could not retrieve connection asset: $connectionAssetName. Assure that this asset exists in the Automation account."; } -$CertificateAssetName = $Conn.CertificateAssetName; -Write-Verbose "Getting the certificate: $CertificateAssetName" -Verbose; -$AzureCert = Get-AutomationCertificate -Name $CertificateAssetName; -if ($AzureCert -eq $null) +$certificateAssetName = $automationConnection.CertificateAssetName; +Write-Verbose "Getting the certificate: $certificateAssetName" -Verbose; +$automationCertificate = Get-AutomationCertificate -Name $certificateAssetName; +if ($automationCertificate -eq $null) { - throw "Could not retrieve certificate asset: $CertificateAssetName. Assure that this asset exists in the Automation account."; + throw "Could not retrieve certificate asset: $certificateAssetName. Assure that this asset exists in the Automation account."; } Write-Verbose "Authenticating to Azure with certificate." -Verbose; -Set-AzureSubscription -SubscriptionName $Conn.SubscriptionName -SubscriptionId $Conn.SubscriptionID -Certificate $AzureCert; -Select-AzureSubscription -SubscriptionId $Conn.SubscriptionID; +Set-AzureSubscription -SubscriptionName $automationConnection.SubscriptionName -SubscriptionId $automationConnection.SubscriptionID -Certificate $automationCertificate; +Select-AzureSubscription -SubscriptionId $automationConnection.SubscriptionID; $currentIndex = 0; for($currentRun = 0; $currentRun -lt ([math]::Ceiling($databaseServerPairs.Length/$batchingLimit)); $currentRun++) diff --git a/samples/manage/azure-automation-automated-export/README.md b/samples/manage/azure-automation-automated-export/README.md index 9e53c2b76c..ac3fcce754 100644 --- a/samples/manage/azure-automation-automated-export/README.md +++ b/samples/manage/azure-automation-automated-export/README.md @@ -16,6 +16,11 @@ Provides the scripts and lists the steps to set up automatically exporting your - Create your Automation Credential for each of your SQL Azure servers you intend to export by [following the instructions here](https://docs.microsoft.com/en-us/azure/automation/automation-credentials#creating-a-new-credential-asset). 3. Create the Azure Storage Account to hold your bacpac files - Create the Storage Account by [following the instructions here](https://docs.microsoft.com/en-us/azure/storage/storage-create-storage-account#create-a-storage-account). + - Create the Blob Storage Container + - Go to your Storage Account + - Click the Blobs tile + - Click the Add Container button + - Name the container, keep the access type as Private, and click the Create button - Copy your Storage Account access keys by [following the instructions here](https://docs.microsoft.com/en-us/azure/storage/storage-create-storage-account#view-and-copy-storage-access-keys). - Create an Azure Automation string Variable asset for your Storage Account access key by [following the instructions here](https://docs.microsoft.com/en-us/azure/automation/automation-variables#creating-an-automation-variable). 4. Set Up Log Analytics (OMS) and Alerts (optional for alerting)