Bug: Investigate root cause of incremental build failure

[erikschlegel]

Description

As a developer, I'd like to troubleshoot the underlying cause to a re-occurring failure with incremental Azure DevOPS builds. Here's an example of the build failure

Acceptance Criteria

Reference: [Done-Done Checklist] (https://github.com/Microsoft/code-with-engineering-playbook/blob/master/Engineering/BestPractices/DoneDone.md)

• Should provide a description of why azure resources are included in the Terraform execution plan for builds where the Terraform configuration files are unmodified.
• Should include a list of changes necessary to resolve this issue once and for all.

[erikschlegel]

I was able to complete the research around this issue. There are two classes of azure resource changes being included in the terraform execution plan for every incremental release.

1. Keyvault access policies - KV access policies are being deleted and re-created following the initial TF apply. This was happening due to a bug within the keyvault-policy module.

   Resolution
   @nmiodice fixed this issue with last week's release of Do not KV delete access policies on incremental deployments #332.

2. Service Principal - Both the service principal password and app service config settings are being re-created on each incremental deployment due to the password being re-generated on each deployment.

   Resolution
   The service principal password provider should be ignored from state changes via the ignore_changes lifecycle block, as Terraform shouldn't manage service principal password's in Azure after the initial creation. This fix will be covered in Ignore Key Vault Secrets after initial create #330