Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azurerm_frontdoor - Fix for Frontdoor resource elements being returned out of order #11456

Merged
merged 35 commits into from
May 6, 2021

Conversation

WodansSon
Copy link
Collaborator

@WodansSon WodansSon commented Apr 24, 2021

BREAKING CHANGE:
This PR remove the Custom HTTPS Configuration section from the main Front Door resource and delegates it completely to the dedicated azurerm_frontdoor_custom_https_configuration resource. TBH it never should have been released with support in both locations as it was causing contention in the API because both of the resources were calling the same API multiple times, that is fixed as well in this PR.

azurerm_frontdoor:
The values that have been removed are:

  • custom_https_provisioning_enabled
  • custom_https_configuration

azurerm_frontdoor_custom_https_configuration:
The values that have been removed are:

  • resource_group_name

UNIQUE CUSTOM HTTPS CONFIGURATION ID:
Exposed a unique ID string for the azurerm_frontdoor_custom_https_configuration resource with the following format:

/subscriptions/12345678-1234-9876-4563-123456789012/resourceGroups/resGroup1/providers/Microsoft.Network/frontDoors/frontdoor1/customHttpsConfiguration/endpoint1

BEHAVIOR CHANGE:
With the release of the v2.58.0 provider, if you run the apply command against an existing Front Door resource the changes will not be applied. This will only happen once with preexisting Front Door instances and will not affect newly provisioned Front Door resources. This change in behavior in Terraform is due to an issue where the underlying service teams API is now returning the response JSON out of order from the way it was sent to the resource provider by Terraform causing unexpected discrepancies in the plan after the resource has been provisioned. This will only happen one time, to avoid unwanted changes from being provisioned, once the explicit_resource_order mapping structure has been persisted to the state file the resource will resume functioning normally.

(fixes #9153)
(fixes #8039)
(fixes #10661)
(fixes #9075)
(fixes #11287)
(fixes #7613)
(fixes #7208)
(fixes #6351)

@ghost ghost added the size/S label Apr 24, 2021
@katbyte
Copy link
Collaborator

katbyte commented Apr 24, 2021

related to #10828 and #10659

@WodansSon
Copy link
Collaborator Author

WodansSon commented Apr 24, 2021

related to #9153 and #11348

@ghost ghost added size/XL and removed size/S labels Apr 24, 2021
@WodansSon WodansSon changed the title [WIP] Fix for Frontdoor out of order [WIP] azurerm_frontdoor - Fix for Frontdoor resource elements being returned out of order Apr 24, 2021
@WodansSon
Copy link
Collaborator Author

My current hypothesis is that the Frontend Endpoint with the current implementation was being modified simultaneously by both the azurerm_frontdoor resource, and the newly introduced as of v2.20.0 release of the provider, azurerm_frontdoor_custom_https_configuration resource. Since both of these resources were attempting to modify the same resource at the same time this ended up with the error msg:

Error: updating Custom HTTPS configuration for Frontend Endpoint "acctest-FD-default-FE" (Front Door "acctest-FD" / Resource Group "XXXXXX-frontdoor-order"): unable to enable/update Custom Domain HTTPS for Frontend Endpoint "acctest-FD-default-FE" (Resource Group "XXXXXX-frontdoor-order"): enabling Custom Domain HTTPS for Frontend Endpoint: frontdoor.FrontendEndpointsClient#EnableHTTPS: Failure sending request: StatusCode=0 -- Original Error: Code="BadRequest" Message="That action isn’t allowed in this profile."

So to fix this I did a few things, with this PR I now block the modification of the custom HTTPS configuration from the main azurerm_frontdoor resource, you can only change these settings via the azurerm_frontdoor_custom_https_configuration resource. I also removed all of the code from the azurerm_frontdoor resource that could modify the custom HTTPS settings to remove the contention from the resource. I also fixed a bug where the azurerm_frontdoor_custom_https_configuration ID was the same as the azurerm_frontdoor frontendEndpoint ID and now generate a unique ID for the azurerm_frontdoor_custom_https_configuration ID.

I believe with the changes I have introduced in this PR, even though they maybe painful, this will solve most, if not all, of the issues that so many of you have been hitting with the azurerm_frontdoor resource. Sorry about that, but a lot of this is out of our control.

If you have the bandwidth and can build a private provider from this PR and test it against your current resources (please do not test this private against your prod deployments, this is just an experiment. If you can one off the issue in a test environment that would be ideal) that would be very helpful, as I cannot test all of the scenarios that exist in the wild, but the ones I have tested locally this PR did fix. So if you are up to help to get this fix tested it would be much appreciated to get as much data as possible. Cheers. 🚀

@WodansSon WodansSon added bug service/frontdoor upstream/microsoft Indicates that there's an upstream issue blocking this issue/PR labels Apr 24, 2021
@WodansSon WodansSon changed the title [WIP] azurerm_frontdoor - Fix for Frontdoor resource elements being returned out of order azurerm_frontdoor - Fix for Frontdoor resource elements being returned out of order Apr 24, 2021
@JayDoubleu
Copy link

JayDoubleu commented Apr 24, 2021

While on this patch it returns below even if custom_https_configuration is not present in the configuration as it seems to be automatically appending it :

Error: setting the deprecated 'custom_https_configuration' block is no longer supported within the 'azurerm_frontdoor' resource, 
please use the 'azurerm_frontdoor_custom_https_configuration' resource to set these values.

Plan always appends below :

      + frontend_endpoint {
          + custom_https_provisioning_enabled       = false
          + host_name                               = "<redacted>.azurefd.net"
          + id                                      = (known after apply)
          + name                                    = "<redacted>"
          + session_affinity_enabled                = false
          + session_affinity_ttl_seconds            = 0
          + web_application_firewall_policy_link_id = "<redacted>

          + custom_https_configuration {
              + certificate_source    = "FrontDoor"
              + minimum_tls_version   = (known after apply)
              + provisioning_state    = (known after apply)
              + provisioning_substate = (known after apply)
            }
        }

After applying below in azurerm/internal/services/frontdoor/frontdoor_resource.go it seems to allow me to create the resource however no luck around ordering as after first creation running terraform plan tries to move routing and backend pools around.

-		if customHttpsConfigurationNew != nil {
+		if len(customHttpsConfigurationNew) > 0 {
  • no custom_https_configuration block present in code
  • no custom_https_provisioning_enabled set anywhere
  • azurerm_frontdoor_custom_https_configuration not used anywhere either.

@WodansSon
Copy link
Collaborator Author

WodansSon commented Apr 25, 2021

@JayDoubleu, dagnabbit... Every configuration I ran last night worked perfectly for me locally. Would you mind posting your configuration file(with vars file if you have one) you used so I may add it as a test case?

The always seeing the custom_https_configuration is an artifact of the fields being marked as Optional and Computed however what you are seeing is different from what I am seeing on the initial creation of the resource:

      + frontend_endpoint {
          + custom_https_provisioning_enabled = (known after apply)
          + host_name                         = "acctest-FD.azurefd.net"
          + id                                = (known after apply)
          + name                              = "acctest-FD-default-FE"
          + session_affinity_enabled          = false
          + session_affinity_ttl_seconds      = 0

          + custom_https_configuration {
              + azure_key_vault_certificate_secret_name    = (known after apply)
              + azure_key_vault_certificate_secret_version = (known after apply)
              + azure_key_vault_certificate_vault_id       = (known after apply)
              + certificate_source                         = (known after apply)
              + minimum_tls_version                        = (known after apply)
              + provisioning_state                         = (known after apply)
              + provisioning_substate                      = (known after apply)
            }
        }

I am currently investigating why we would be seeing this delta between our two experiences...

@WodansSon WodansSon changed the title azurerm_frontdoor - Fix for Frontdoor resource elements being returned out of order [WIP]azurerm_frontdoor - Fix for Frontdoor resource elements being returned out of order Apr 25, 2021
@favoretti
Copy link
Collaborator

@timja Would you mind giving this one a shot on your config please?

@timja
Copy link
Contributor

timja commented Apr 25, 2021

Will do tomorrow

@JayDoubleu
Copy link

JayDoubleu commented Apr 25, 2021

@WodansSon

Terraform version

Terraform v0.14.4

Containerfile used to create provider

Click to expand!

Containerfile

# docker build -t terraform-provider-azurerm-11456 -f Containerfile #--build-arg PR="11456"
# docker run -v $(pwd):/tmp/pwd terraform-provider-azurerm-11456 cp /go/bin/terraform-provider-azurerm /tmp/pwd
FROM docker.io/library/golang:1.16
ARG REPOSITORY="https://github.com/terraform-providers/terraform-provider-azurerm.git"
ARG BRANCH="master"
#ARG BRANCH="v2.53.0"
ARG PR="11456"
ARG GOOS="linux"

RUN mkdir -p $GOPATH/src/github.com/terraform-providers
RUN printf "\nRepository: ${REPOSITORY}\nBranch: ${BRANCH}\nPR: ${PR}\n\n"
RUN git clone ${REPOSITORY} $GOPATH/src/github.com/terraform-providers/terraform-provider-azurerm

WORKDIR $GOPATH/src/github.com/terraform-providers/terraform-provider-azurerm
RUN git checkout ${BRANCH}
RUN mkdir -p bin
RUN git log --name-status HEAD^..HEAD
RUN curl -sO https://patch-diff.githubusercontent.com/raw/terraform-providers/terraform-provider-azurerm/pull/${PR}.patch
RUN git config --global user.email "you@example.com" && git config --global user.name "Your Name"
RUN git am ${PR}.patch
RUN git log --name-status HEAD^..HEAD
RUN make build
RUN sha256sum $GOPATH/bin/terraform-provider-azurerm

Build log

STEP 1: FROM docker.io/library/golang:1.16
STEP 2: ARG REPOSITORY="https://github.com/terraform-providers/terraform-provider-azurerm.git"
--> a5dc090f84b
STEP 3: ARG BRANCH="master"
--> a72d78a4c4e
STEP 4: ARG PR="11456"
--> f93cabe1078
STEP 5: ARG GOOS="linux"
--> c1ca016402e
STEP 6: RUN mkdir -p $GOPATH/src/github.com/terraform-providers
--> 3bb78674208
STEP 7: RUN printf "\nRepository: ${REPOSITORY}\nBranch: ${BRANCH}\nPR: ${PR}\n\n"

Repository: https://github.com/terraform-providers/terraform-provider-azurerm.git
Branch: master
PR: 11456

--> ad0c390be5f
STEP 8: RUN git clone ${REPOSITORY} $GOPATH/src/github.com/terraform-providers/terraform-provider-azurerm
Cloning into '/go/src/github.com/terraform-providers/terraform-provider-azurerm'...
Checking out files: 100% (11062/11062), done.
--> 3cff408ee4c
STEP 9: WORKDIR $GOPATH/src/github.com/terraform-providers/terraform-provider-azurerm
--> 527c865b900
STEP 10: RUN git checkout ${BRANCH}
Already on 'master'
Your branch is up to date with 'origin/master'.
--> 77f6324f76d
STEP 11: RUN mkdir -p bin
--> 834fc2d2bcd
STEP 12: RUN git log --name-status HEAD^..HEAD
commit 3e1a54615fac153689b3d092feed7bc1a0193be9
Author: magodo <wztdyl@sina.com>
Date:   Sun Apr 25 12:08:20 2021 +0800

    GNUMakefile: use `go install` to install tools since Go v1.16 (#11459)

M	GNUmakefile
--> 997f938447d
STEP 13: RUN curl -sO https://patch-diff.githubusercontent.com/raw/terraform-providers/terraform-provider-azurerm/pull/${PR}.patch
--> 8bda1367fbf
STEP 14: RUN git config --global user.email "you@example.com" && git config --global user.name "Your Name"
--> 463a860d8f2
STEP 15: RUN git am ${PR}.patch
Applying: Fix for Frontdoor out of order
Applying: Remove all FE mods from AFD add faux ID to HTTPS
Applying: Block custom HTTPS values in main AFD resource
Applying: Fix comment spelling and spacing I think?
Applying: Fix comment spacing? :/
Applying: Update custom HTTP check from nil to len
--> 31d10cd4baf
STEP 16: RUN git log --name-status HEAD^..HEAD
commit f889e75459d8cf5c3abc20177a857a4b626f26dc
Author: Jeffrey Cline <20408400+WodansSon@users.noreply.github.com>
Date:   Sun Apr 25 02:49:46 2021 -0700

    Update custom HTTP check from nil to len

M	azurerm/internal/services/frontdoor/frontdoor_resource.go
--> ece89d33e0a
STEP 17: RUN make build
==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
==> Checking that acceptance test packages are used...
go generate ./azurerm/internal/services/...
go generate ./azurerm/internal/provider/
go install
--> 98195559461
STEP 18: RUN sha256sum $GOPATH/bin/terraform-provider-azurerm
b9de36c423b4278126b7e878aca71faa0101a3527b86d1be132a688e4b33b807  /go/bin/terraform-provider-azurerm
STEP 19: COMMIT terraform-provider-azurerm-11456
--> 38b30463af6
38b30463af6560fc0615b2b34d7ed920373885180a9a714c8c10de10505e7fe3

Tf code used to reproduce the sorting issue

Click to expand!
# az feature register --namespace Microsoft.Network --name BypassCnameCheckForCustomDomainDeletion --subscription "<redacted>" # whoever created this feature deserves a beer
terraform {
required_version = "~> 0.14.4"
required_providers { azurerm = "=2.56.0" }
}
provider "azurerm" {
subscription_id = "<redacted>"
tenant_id       = "<redacted>"
features {}
}
locals {
environment      = "dev001" #
default_location = "uksouth"
prefix           = "acctest-FD-001-${local.environment}" # replace 001 to make unique
storage_prefix   = lower(replace(local.prefix, "-", ""))
tags             = {}
fd_settings = {
  default_frontend         = local.prefix
  custom_frontend          = "${local.prefix}-custom"
  custom_frontend_domain   = "jaydoubleu.co.uk"
  default_hp_settings_name = "exampleHealthProbeSetting1"
  default_lb_settings_name = "exampleLoadBalancingSettings1"
}
}

###############
resource "azurerm_resource_group" "default" {
name     = "rg-${lower(local.prefix)}"
location = local.default_location
tags     = local.tags
}
### storage accounts
resource "azurerm_storage_account" "site01" {
name                     = "${local.storage_prefix}site01"
resource_group_name      = azurerm_resource_group.default.name
location                 = azurerm_resource_group.default.location
account_tier             = "Standard"
account_replication_type = "LRS"
tags                     = local.tags
static_website {
  error_404_document = "error.html"
  index_document     = "index.html"
}
}
resource "azurerm_storage_account" "site02" {
name                     = "${local.storage_prefix}site02"
resource_group_name      = azurerm_resource_group.default.name
location                 = azurerm_resource_group.default.location
account_tier             = "Standard"
account_replication_type = "LRS"
tags                     = local.tags
static_website {
  error_404_document = "error.html"
  index_document     = "index.html"
}
}
resource "azurerm_storage_account" "site03" {
name                     = "${local.storage_prefix}site03"
resource_group_name      = azurerm_resource_group.default.name
location                 = azurerm_resource_group.default.location
account_tier             = "Standard"
account_replication_type = "LRS"
tags                     = local.tags
static_website {
  error_404_document = "error.html"
  index_document     = "index.html"
}
}

resource "azurerm_frontdoor" "default" {
name                                         = local.prefix
resource_group_name                          = azurerm_resource_group.default.name
enforce_backend_pools_certificate_name_check = true
tags                                         = local.tags
frontend_endpoint {
  name      = local.fd_settings.default_frontend
  host_name = "${lower(local.fd_settings.default_frontend)}.azurefd.net"
}
frontend_endpoint {
  name      = local.fd_settings.custom_frontend
  host_name = "${lower(local.fd_settings.custom_frontend)}.${local.fd_settings.custom_frontend_domain}"
}
backend_pool_health_probe {
  name                = local.fd_settings.default_hp_settings_name
  interval_in_seconds = "30"
  protocol            = "Https"
}
backend_pool_load_balancing {
  name = local.fd_settings.default_lb_settings_name
}
### site01
backend_pool {
  name = azurerm_storage_account.site01.name
  backend {
    host_header = azurerm_storage_account.site01.primary_web_host
    address     = azurerm_storage_account.site01.primary_web_host
    http_port   = 80
    https_port  = 443
  }
  load_balancing_name = local.fd_settings.default_lb_settings_name
  health_probe_name   = local.fd_settings.default_hp_settings_name
}
routing_rule {
  name               = azurerm_storage_account.site01.name
  enabled            = true
  accepted_protocols = ["Https"]
  patterns_to_match  = ["/*"]
  frontend_endpoints = [local.fd_settings.default_frontend, local.fd_settings.custom_frontend]
  forwarding_configuration {
    forwarding_protocol    = "HttpsOnly"
    backend_pool_name      = azurerm_storage_account.site01.name
    custom_forwarding_path = "/"
  }
}
routing_rule {
  name               = "${azurerm_storage_account.site01.name}RedirectHTTPtoHTTPS"
  enabled            = true
  accepted_protocols = ["Http"]
  frontend_endpoints = [local.fd_settings.default_frontend, local.fd_settings.custom_frontend]
  patterns_to_match  = ["/*"]
  redirect_configuration {
    redirect_protocol = "HttpsOnly"
    redirect_type     = "Found"
    custom_path       = "/"
  }
}

### site02
backend_pool {
  name = azurerm_storage_account.site02.name
  backend {
    host_header = azurerm_storage_account.site02.primary_web_host
    address     = azurerm_storage_account.site02.primary_web_host
    http_port   = 80
    https_port  = 443
  }
  load_balancing_name = local.fd_settings.default_lb_settings_name
  health_probe_name   = local.fd_settings.default_hp_settings_name
}
routing_rule {
  name               = azurerm_storage_account.site02.name
  enabled            = true
  accepted_protocols = ["Https"]
  patterns_to_match  = ["/site02", "/site02/*"]
  frontend_endpoints = [local.fd_settings.default_frontend, local.fd_settings.custom_frontend]
  forwarding_configuration {
    forwarding_protocol    = "HttpsOnly"
    backend_pool_name      = azurerm_storage_account.site02.name
    custom_forwarding_path = "/"
  }
}
routing_rule {
  name               = "${azurerm_storage_account.site02.name}RedirectHTTPtoHTTPS"
  enabled            = true
  accepted_protocols = ["Http"]
  frontend_endpoints = [local.fd_settings.default_frontend, local.fd_settings.custom_frontend]
  patterns_to_match  = ["/site02", "/site02/*"]
  redirect_configuration {
    redirect_protocol = "HttpsOnly"
    redirect_type     = "Found"
    custom_path       = "/site02/"
  }
}
### site03
backend_pool {
  name = azurerm_storage_account.site03.name
  backend {
    host_header = azurerm_storage_account.site03.primary_web_host
    address     = azurerm_storage_account.site03.primary_web_host
    http_port   = 80
    https_port  = 443
  }
  load_balancing_name = local.fd_settings.default_lb_settings_name
  health_probe_name   = local.fd_settings.default_hp_settings_name
}
routing_rule {
  name               = azurerm_storage_account.site03.name
  enabled            = true
  accepted_protocols = ["Https"]
  patterns_to_match  = ["/site03", "/site03/*"]
  frontend_endpoints = [local.fd_settings.default_frontend, local.fd_settings.custom_frontend]
  forwarding_configuration {
    forwarding_protocol    = "HttpsOnly"
    backend_pool_name      = azurerm_storage_account.site03.name
    custom_forwarding_path = "/"
  }
}
routing_rule {
  name               = "${azurerm_storage_account.site03.name}RedirectHTTPtoHTTPS"
  enabled            = true
  accepted_protocols = ["Http"]
  frontend_endpoints = [local.fd_settings.default_frontend, local.fd_settings.custom_frontend]
  patterns_to_match  = ["/site03", "/site03/*"]
  redirect_configuration {
    redirect_protocol = "HttpsOnly"
    redirect_type     = "Found"
    custom_path       = "/site03/"
  }
}
}

First terraform apply

Click to expand!
Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
- hashicorp/azurerm in /home/jaydoubleu/devel/providers/devel/terraform-provider-azurerm_bins/11456

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.

azurerm_resource_group.default: Refreshing state... [id=/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001]
azurerm_storage_account.site01: Refreshing state... [id=/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Storage/storageAccounts/acctestfd001dev001site01]
azurerm_storage_account.site02: Refreshing state... [id=/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Storage/storageAccounts/acctestfd001dev001site02]
azurerm_storage_account.site03: Refreshing state... [id=/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Storage/storageAccounts/acctestfd001dev001site03]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# azurerm_frontdoor.default will be created
+ resource "azurerm_frontdoor" "default" {
    + backend_pool_health_probes                   = (known after apply)
    + backend_pool_load_balancing_settings         = (known after apply)
    + backend_pools                                = (known after apply)
    + backend_pools_send_receive_timeout_seconds   = 60
    + cname                                        = (known after apply)
    + enforce_backend_pools_certificate_name_check = true
    + frontend_endpoints                           = (known after apply)
    + header_frontdoor_id                          = (known after apply)
    + id                                           = (known after apply)
    + load_balancer_enabled                        = true
    + location                                     = (known after apply)
    + name                                         = "acctest-FD-001-dev001"
    + resource_group_name                          = "rg-acctest-fd-001-dev001"
    + routing_rules                                = (known after apply)

    + backend_pool {
        + health_probe_name   = "exampleHealthProbeSetting1"
        + id                  = (known after apply)
        + load_balancing_name = "exampleLoadBalancingSettings1"
        + name                = "acctestfd001dev001site01"

        + backend {
            + address     = "acctestfd001dev001site01.z33.web.core.windows.net"
            + enabled     = true
            + host_header = "acctestfd001dev001site01.z33.web.core.windows.net"
            + http_port   = 80
            + https_port  = 443
            + priority    = 1
            + weight      = 50
          }
      }
    + backend_pool {
        + health_probe_name   = "exampleHealthProbeSetting1"
        + id                  = (known after apply)
        + load_balancing_name = "exampleLoadBalancingSettings1"
        + name                = "acctestfd001dev001site02"

        + backend {
            + address     = "acctestfd001dev001site02.z33.web.core.windows.net"
            + enabled     = true
            + host_header = "acctestfd001dev001site02.z33.web.core.windows.net"
            + http_port   = 80
            + https_port  = 443
            + priority    = 1
            + weight      = 50
          }
      }
    + backend_pool {
        + health_probe_name   = "exampleHealthProbeSetting1"
        + id                  = (known after apply)
        + load_balancing_name = "exampleLoadBalancingSettings1"
        + name                = "acctestfd001dev001site03"

        + backend {
            + address     = "acctestfd001dev001site03.z33.web.core.windows.net"
            + enabled     = true
            + host_header = "acctestfd001dev001site03.z33.web.core.windows.net"
            + http_port   = 80
            + https_port  = 443
            + priority    = 1
            + weight      = 50
          }
      }

    + backend_pool_health_probe {
        + enabled             = true
        + id                  = (known after apply)
        + interval_in_seconds = 30
        + name                = "exampleHealthProbeSetting1"
        + path                = "/"
        + probe_method        = "GET"
        + protocol            = "Https"
      }

    + backend_pool_load_balancing {
        + additional_latency_milliseconds = 0
        + id                              = (known after apply)
        + name                            = "exampleLoadBalancingSettings1"
        + sample_size                     = 4
        + successful_samples_required     = 2
      }

    + frontend_endpoint {
        + custom_https_provisioning_enabled = (known after apply)
        + host_name                         = "acctest-fd-001-dev001.azurefd.net"
        + id                                = (known after apply)
        + name                              = "acctest-FD-001-dev001"
        + session_affinity_enabled          = false
        + session_affinity_ttl_seconds      = 0

        + custom_https_configuration {
            + azure_key_vault_certificate_secret_name    = (known after apply)
            + azure_key_vault_certificate_secret_version = (known after apply)
            + azure_key_vault_certificate_vault_id       = (known after apply)
            + certificate_source                         = (known after apply)
            + minimum_tls_version                        = (known after apply)
            + provisioning_state                         = (known after apply)
            + provisioning_substate                      = (known after apply)
          }
      }
    + frontend_endpoint {
        + custom_https_provisioning_enabled = (known after apply)
        + host_name                         = "acctest-fd-001-dev001-custom.jaydoubleu.co.uk"
        + id                                = (known after apply)
        + name                              = "acctest-FD-001-dev001-custom"
        + session_affinity_enabled          = false
        + session_affinity_ttl_seconds      = 0

        + custom_https_configuration {
            + azure_key_vault_certificate_secret_name    = (known after apply)
            + azure_key_vault_certificate_secret_version = (known after apply)
            + azure_key_vault_certificate_vault_id       = (known after apply)
            + certificate_source                         = (known after apply)
            + minimum_tls_version                        = (known after apply)
            + provisioning_state                         = (known after apply)
            + provisioning_substate                      = (known after apply)
          }
      }

    + routing_rule {
        + accepted_protocols = [
            + "Https",
          ]
        + enabled            = true
        + frontend_endpoints = [
            + "acctest-FD-001-dev001",
            + "acctest-FD-001-dev001-custom",
          ]
        + id                 = (known after apply)
        + name               = "acctestfd001dev001site01"
        + patterns_to_match  = [
            + "/*",
          ]

        + forwarding_configuration {
            + backend_pool_name                     = "acctestfd001dev001site01"
            + cache_enabled                         = false
            + cache_query_parameter_strip_directive = "StripAll"
            + cache_use_dynamic_compression         = false
            + custom_forwarding_path                = "/"
            + forwarding_protocol                   = "HttpsOnly"
          }
      }
    + routing_rule {
        + accepted_protocols = [
            + "Http",
          ]
        + enabled            = true
        + frontend_endpoints = [
            + "acctest-FD-001-dev001",
            + "acctest-FD-001-dev001-custom",
          ]
        + id                 = (known after apply)
        + name               = "acctestfd001dev001site01RedirectHTTPtoHTTPS"
        + patterns_to_match  = [
            + "/*",
          ]

        + redirect_configuration {
            + custom_path       = "/"
            + redirect_protocol = "HttpsOnly"
            + redirect_type     = "Found"
          }
      }
    + routing_rule {
        + accepted_protocols = [
            + "Https",
          ]
        + enabled            = true
        + frontend_endpoints = [
            + "acctest-FD-001-dev001",
            + "acctest-FD-001-dev001-custom",
          ]
        + id                 = (known after apply)
        + name               = "acctestfd001dev001site02"
        + patterns_to_match  = [
            + "/site02",
            + "/site02/*",
          ]

        + forwarding_configuration {
            + backend_pool_name                     = "acctestfd001dev001site02"
            + cache_enabled                         = false
            + cache_query_parameter_strip_directive = "StripAll"
            + cache_use_dynamic_compression         = false
            + custom_forwarding_path                = "/"
            + forwarding_protocol                   = "HttpsOnly"
          }
      }
    + routing_rule {
        + accepted_protocols = [
            + "Http",
          ]
        + enabled            = true
        + frontend_endpoints = [
            + "acctest-FD-001-dev001",
            + "acctest-FD-001-dev001-custom",
          ]
        + id                 = (known after apply)
        + name               = "acctestfd001dev001site02RedirectHTTPtoHTTPS"
        + patterns_to_match  = [
            + "/site02",
            + "/site02/*",
          ]

        + redirect_configuration {
            + custom_path       = "/site02/"
            + redirect_protocol = "HttpsOnly"
            + redirect_type     = "Found"
          }
      }
    + routing_rule {
        + accepted_protocols = [
            + "Https",
          ]
        + enabled            = true
        + frontend_endpoints = [
            + "acctest-FD-001-dev001",
            + "acctest-FD-001-dev001-custom",
          ]
        + id                 = (known after apply)
        + name               = "acctestfd001dev001site03"
        + patterns_to_match  = [
            + "/site03",
            + "/site03/*",
          ]

        + forwarding_configuration {
            + backend_pool_name                     = "acctestfd001dev001site03"
            + cache_enabled                         = false
            + cache_query_parameter_strip_directive = "StripAll"
            + cache_use_dynamic_compression         = false
            + custom_forwarding_path                = "/"
            + forwarding_protocol                   = "HttpsOnly"
          }
      }
    + routing_rule {
        + accepted_protocols = [
            + "Http",
          ]
        + enabled            = true
        + frontend_endpoints = [
            + "acctest-FD-001-dev001",
            + "acctest-FD-001-dev001-custom",
          ]
        + id                 = (known after apply)
        + name               = "acctestfd001dev001site03RedirectHTTPtoHTTPS"
        + patterns_to_match  = [
            + "/site03",
            + "/site03/*",
          ]

        + redirect_configuration {
            + custom_path       = "/site03/"
            + redirect_protocol = "HttpsOnly"
            + redirect_type     = "Found"
          }
      }
  }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

azurerm_frontdoor.default: Creating...
azurerm_frontdoor.default: Still creating... [10s elapsed]
azurerm_frontdoor.default: Still creating... [20s elapsed]
azurerm_frontdoor.default: Still creating... [30s elapsed]
azurerm_frontdoor.default: Still creating... [40s elapsed]
azurerm_frontdoor.default: Still creating... [50s elapsed]
azurerm_frontdoor.default: Still creating... [1m0s elapsed]
azurerm_frontdoor.default: Still creating... [1m10s elapsed]
azurerm_frontdoor.default: Still creating... [1m20s elapsed]
azurerm_frontdoor.default: Creation complete after 1m22s [id=/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Terraform plan output right after first creation reproducing sorting issue

Click to expand!
azurerm_resource_group.default: Refreshing state... [id=/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001]
azurerm_storage_account.site03: Refreshing state... [id=/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Storage/storageAccounts/acctestfd001dev001site03]
azurerm_storage_account.site01: Refreshing state... [id=/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Storage/storageAccounts/acctestfd001dev001site01]
azurerm_storage_account.site02: Refreshing state... [id=/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Storage/storageAccounts/acctestfd001dev001site02]
azurerm_frontdoor.default: Refreshing state... [id=/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place

Terraform will perform the following actions:

# azurerm_frontdoor.default will be updated in-place
~ resource "azurerm_frontdoor" "default" {
      id                                           = "/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001"
      name                                         = "acctest-FD-001-dev001"
      tags                                         = {}
      # (12 unchanged attributes hidden)

    ~ backend_pool {
          id                  = "/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001/backendPools/acctestfd001dev001site03"
        ~ name                = "acctestfd001dev001site03" -> "acctestfd001dev001site02"
          # (2 unchanged attributes hidden)

        ~ backend {
            ~ address     = "acctestfd001dev001site03.z33.web.core.windows.net" -> "acctestfd001dev001site02.z33.web.core.windows.net"
            ~ host_header = "acctestfd001dev001site03.z33.web.core.windows.net" -> "acctestfd001dev001site02.z33.web.core.windows.net"
              # (5 unchanged attributes hidden)
          }
      }
    ~ backend_pool {
          id                  = "/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001/backendPools/acctestfd001dev001site02"
        ~ name                = "acctestfd001dev001site02" -> "acctestfd001dev001site03"
          # (2 unchanged attributes hidden)

        ~ backend {
            ~ address     = "acctestfd001dev001site02.z33.web.core.windows.net" -> "acctestfd001dev001site03.z33.web.core.windows.net"
            ~ host_header = "acctestfd001dev001site02.z33.web.core.windows.net" -> "acctestfd001dev001site03.z33.web.core.windows.net"
              # (5 unchanged attributes hidden)
          }
      }



    ~ frontend_endpoint {
        ~ host_name                         = "acctest-fd-001-dev001-custom.jaydoubleu.co.uk" -> "acctest-fd-001-dev001.azurefd.net"
          id                                = "/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001/frontendEndpoints/acctest-FD-001-dev001-custom"
        ~ name                              = "acctest-FD-001-dev001-custom" -> "acctest-FD-001-dev001"
          # (3 unchanged attributes hidden)
      }
    ~ frontend_endpoint {
        ~ host_name                         = "acctest-fd-001-dev001.azurefd.net" -> "acctest-fd-001-dev001-custom.jaydoubleu.co.uk"
          id                                = "/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001/frontendEndpoints/acctest-FD-001-dev001"
        ~ name                              = "acctest-FD-001-dev001" -> "acctest-FD-001-dev001-custom"
          # (3 unchanged attributes hidden)
      }

    ~ routing_rule {
        ~ accepted_protocols = [
            - "Http",
            + "Https",
          ]
          id                 = "/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001/routingRules/acctestfd001dev001site01RedirectHTTPtoHTTPS"
        ~ name               = "acctestfd001dev001site01RedirectHTTPtoHTTPS" -> "acctestfd001dev001site01"
          # (3 unchanged attributes hidden)

        + forwarding_configuration {
            + backend_pool_name                     = "acctestfd001dev001site01"
            + cache_enabled                         = false
            + cache_query_parameter_strip_directive = "StripAll"
            + cache_use_dynamic_compression         = false
            + custom_forwarding_path                = "/"
            + forwarding_protocol                   = "HttpsOnly"
          }

        - redirect_configuration {
            - custom_path       = "/" -> null
            - redirect_protocol = "HttpsOnly" -> null
            - redirect_type     = "Found" -> null
          }
      }
    ~ routing_rule {
        ~ accepted_protocols = [
            - "Https",
            + "Http",
          ]
          id                 = "/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001/routingRules/acctestfd001dev001site02"
        ~ name               = "acctestfd001dev001site02" -> "acctestfd001dev001site01RedirectHTTPtoHTTPS"
        ~ patterns_to_match  = [
            - "/site02",
            - "/site02/*",
            + "/*",
          ]
          # (2 unchanged attributes hidden)

        - forwarding_configuration {
            - backend_pool_name                     = "acctestfd001dev001site02" -> null
            - cache_enabled                         = false -> null
            - cache_query_parameter_strip_directive = "StripAll" -> null
            - cache_use_dynamic_compression         = false -> null
            - custom_forwarding_path                = "/" -> null
            - forwarding_protocol                   = "HttpsOnly" -> null
          }

        + redirect_configuration {
            + custom_path       = "/"
            + redirect_protocol = "HttpsOnly"
            + redirect_type     = "Found"
          }
      }
    ~ routing_rule {
          id                 = "/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001/routingRules/acctestfd001dev001site01"
        ~ name               = "acctestfd001dev001site01" -> "acctestfd001dev001site02"
        ~ patterns_to_match  = [
            - "/*",
            + "/site02",
            + "/site02/*",
          ]
          # (3 unchanged attributes hidden)

        ~ forwarding_configuration {
            ~ backend_pool_name                     = "acctestfd001dev001site01" -> "acctestfd001dev001site02"
              # (5 unchanged attributes hidden)
          }
      }
    ~ routing_rule {
          id                 = "/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001/routingRules/acctestfd001dev001site03RedirectHTTPtoHTTPS"
        ~ name               = "acctestfd001dev001site03RedirectHTTPtoHTTPS" -> "acctestfd001dev001site02RedirectHTTPtoHTTPS"
        ~ patterns_to_match  = [
            - "/site03",
            - "/site03/*",
            + "/site02",
            + "/site02/*",
          ]
          # (3 unchanged attributes hidden)

        ~ redirect_configuration {
            ~ custom_path       = "/site03/" -> "/site02/"
              # (2 unchanged attributes hidden)
          }
      }
    ~ routing_rule {
        ~ accepted_protocols = [
            - "Http",
            + "Https",
          ]
          id                 = "/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001/routingRules/acctestfd001dev001site02RedirectHTTPtoHTTPS"
        ~ name               = "acctestfd001dev001site02RedirectHTTPtoHTTPS" -> "acctestfd001dev001site03"
        ~ patterns_to_match  = [
            - "/site02",
            - "/site02/*",
            + "/site03",
            + "/site03/*",
          ]
          # (2 unchanged attributes hidden)

        + forwarding_configuration {
            + backend_pool_name                     = "acctestfd001dev001site03"
            + cache_enabled                         = false
            + cache_query_parameter_strip_directive = "StripAll"
            + cache_use_dynamic_compression         = false
            + custom_forwarding_path                = "/"
            + forwarding_protocol                   = "HttpsOnly"
          }

        - redirect_configuration {
            - custom_path       = "/site02/" -> null
            - redirect_protocol = "HttpsOnly" -> null
            - redirect_type     = "Found" -> null
          }
      }
    ~ routing_rule {
        ~ accepted_protocols = [
            - "Https",
            + "Http",
          ]
          id                 = "/subscriptions/<redacted>/resourceGroups/rg-acctest-fd-001-dev001/providers/Microsoft.Network/frontDoors/acctest-FD-001-dev001/routingRules/acctestfd001dev001site03"
        ~ name               = "acctestfd001dev001site03" -> "acctestfd001dev001site03RedirectHTTPtoHTTPS"
          # (3 unchanged attributes hidden)

        - forwarding_configuration {
            - backend_pool_name                     = "acctestfd001dev001site03" -> null
            - cache_enabled                         = false -> null
            - cache_query_parameter_strip_directive = "StripAll" -> null
            - cache_use_dynamic_compression         = false -> null
            - custom_forwarding_path                = "/" -> null
            - forwarding_protocol                   = "HttpsOnly" -> null
          }

        + redirect_configuration {
            + custom_path       = "/site03/"
            + redirect_protocol = "HttpsOnly"
            + redirect_type     = "Found"
          }
      }
      # (3 unchanged blocks hidden)
  }

Plan: 0 to add, 1 to change, 0 to destroy.

Ps. I have temporarily pre-created cnames under my domain for prefix 001..005 (001 already used by me)
Feel free to flip local.prefix between 002 and 005 to replicate issue straight away if you don't have custom domain by hand.

Even if I would move code blocks around to satisfy terraform plan it will not be consistent across after destroy and re-create as it seems to decide ordering completely randomly. In this case it tries to move around all blocks (frontend endpoints, routing rules and backend pools ). Sometimes I get lucky and it gets it "just right" so I am unable to reproduce issue but after destroy and recreate usually can be replicated. One time it will move some blocks around and not the other etc.

cname conffig
frontdoor 001 conffig

@favoretti
Copy link
Collaborator

Will do tomorrow

Thanks so much!

@timja
Copy link
Contributor

timja commented Apr 26, 2021

When I run this on the existing frontdoor that I was using to test: #10571

I just get the wrong ssl cert picked:

│ Error: updating Custom HTTPS configuration for Frontend Endpoint "timj3" (Front Door "timjfd-sbox" / Resource Group "timj-acme-test"): unable to enable/update Custom Domain HTTPS for Frontend Endpoint "timj3" (Resource Group "timj-acme-test"): waiting to enable Custom Domain HTTPS for Frontend Endpoint: Code="CustomDomainSecureDeliveryKeyVaultCustomDomainHostnameNotIncluded" Message="The certificate doesn't include the hostname to be secured."
│
│   on .terraform/modules/landing_zone/frontdoor.tf line 1, in resource "azurerm_frontdoor" "main":
│    1: resource "azurerm_frontdoor" "main" {
│

https://github.com/hmcts/terraform-module-frontdoor/tree/hack (commit 838faf9802fba9d8e499480adbb52b97c11d6cd1)

terraform.tfvars

frontends = [
{
product = "timj1"
name = "timj1"
custom_domain = "timj1.sandbox.platform.hmcts.net"
backend_domain = [
"firewall-sbox-int-palo-sbox.uksouth.cloudapp.azure.com"]
certificate_name = "timj1-sandbox-platform-hmcts-net"
},
{
product = "timj2"
name = "timj2"
custom_domain = "timj2.sandbox.platform.hmcts.net"
backend_domain = [
"firewall-sbox-int-palo-sbox.uksouth.cloudapp.azure.com"]
certificate_name = "timj2-sandbox-platform-hmcts-net"
},
{
product = "timj3"
name = "timj3"
www_redirect = true
custom_domain = "timj3.sandbox.platform.hmcts.net"
backend_domain = [
"firewall-sbox-int-palo-sbox.uksouth.cloudapp.azure.com"]
certificate_name = "timj3-sandbox-platform-hmcts-net"
},
{
product = "timj4"
name = "timj4"
custom_domain = "timj4.sandbox.platform.hmcts.net"
backend_domain = [
"firewall-sbox-int-palo-sbox.uksouth.cloudapp.azure.com"]
certificate_name = "timj4-sandbox-platform-hmcts-net"
},
{
product = "timj5"
name = "timj5"
custom_domain = "timj5.sandbox.platform.hmcts.net"
backend_domain = [
"firewall-sbox-int-palo-sbox.uksouth.cloudapp.azure.com"]
certificate_name = "timj5-sandbox-platform-hmcts-net"
},
{
product = "timj6"
name = "timj6"
custom_domain = "timj6.sandbox.platform.hmcts.net"
backend_domain = [
"firewall-sbox-int-palo-sbox.uksouth.cloudapp.azure.com"]
certificate_name = "timj6-sandbox-platform-hmcts-net"
}
]

main.tf

variable "env" {
default = "sbox"
}
data "azurerm_resource_group" "main" {
name = "timj-acme-test"
}

data "azurerm_resource_group" "key_vault" {
name = "timj-acme-test"
}

variable "frontends" {
default = {}
}

data "azurerm_subscription" "current" {}

module "log_analytics_workspace" {
source = "git::https://github.com/hmcts/terraform-module-log-analytics-workspace-id.git?ref=master"
environment = var.env
}

module "landing_zone" {
source = "git::https://github.com/hmcts/terraform-module-frontdoor.git?ref=hack"

providers = {
azurerm = azurerm
azurerm.data = azurerm.data
}

common_tags = {}
env = var.env
subscription = "sandbox"
project = "timjfd"
location = "uksouth"
frontends = var.frontends
ssl_mode = "AzureKeyVault"
resource_group = data.azurerm_resource_group.main.name
subscription_id = data.azurerm_subscription.current.subscription_id
certificate_key_vault_name = "timj-acme-test"
oms_env = "sandbox"
certificate_name_check = false
key_vault_resource_group = data.azurerm_resource_group.key_vault.name
log_analytics_workspace_id = module.log_analytics_workspace.workspace_id

add_access_policy = true
}

@timja
Copy link
Contributor

timja commented Apr 26, 2021

I destroyed the frontdoor and re-created it, it successfully applied.

When I do my next plan I get:

resource_group_name diff:

  ~ resource "azurerm_frontdoor_custom_https_configuration" "https" {
        id                                = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/frontendEndpoints/timj2"
      - resource_group_name               = "timj-acme-test" -> null
        # (2 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

full shuffling of pretty much everything:

plan

 ~ resource "azurerm_frontdoor" "main" {
        id                                           = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox"
        name                                         = "timjfd-sbox"
        tags                                         = {}
        # (13 unchanged attributes hidden)

      ~ backend_pool {
          ~ health_probe_name   = "healthProbeSettings-timj4" -> "healthProbeSettings-timj1"
            id                  = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/backendPools/timj4"
          ~ load_balancing_name = "loadBalancingSettings-timj4" -> "loadBalancingSettings-timj1"
          ~ name                = "timj4" -> "timj1"

          ~ backend {
              ~ host_header = "timj4.sandbox.platform.hmcts.net" -> "timj1.sandbox.platform.hmcts.net"
                # (6 unchanged attributes hidden)
            }
        }
      ~ backend_pool {
          ~ health_probe_name   = "healthProbeSettings-timj1" -> "healthProbeSettings-timj4"
            id                  = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/backendPools/timj1"
          ~ load_balancing_name = "loadBalancingSettings-timj1" -> "loadBalancingSettings-timj4"
          ~ name                = "timj1" -> "timj4"

          ~ backend {
              ~ host_header = "timj1.sandbox.platform.hmcts.net" -> "timj4.sandbox.platform.hmcts.net"
                # (6 unchanged attributes hidden)
            }
        }

      ~ backend_pool_health_probe {
            id                  = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/healthProbeSettings/healthProbeSettings-timj2"
          ~ name                = "healthProbeSettings-timj2" -> "healthProbeSettings-timj1"
            # (5 unchanged attributes hidden)
        }
      ~ backend_pool_health_probe {
            id                  = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/healthProbeSettings/healthProbeSettings-timj1"
          ~ name                = "healthProbeSettings-timj1" -> "healthProbeSettings-timj2"
            # (5 unchanged attributes hidden)
        }

      ~ backend_pool_load_balancing {
            id                              = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/loadBalancingSettings/loadBalancingSettings-timj1"
          ~ name                            = "loadBalancingSettings-timj1" -> "defaultLoadBalancing"
            # (3 unchanged attributes hidden)
        }
      ~ backend_pool_load_balancing {
            id                              = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/loadBalancingSettings/loadBalancingSettings-timj2"
          ~ name                            = "loadBalancingSettings-timj2" -> "loadBalancingSettings-timj1"
            # (3 unchanged attributes hidden)
        }
      ~ backend_pool_load_balancing {
            id                              = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/loadBalancingSettings/loadBalancingSettings-timj4"
          ~ name                            = "loadBalancingSettings-timj4" -> "loadBalancingSettings-timj2"
            # (3 unchanged attributes hidden)
        }
      ~ backend_pool_load_balancing {
            id                              = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/loadBalancingSettings/defaultLoadBalancing"
          ~ name                            = "defaultLoadBalancing" -> "loadBalancingSettings-timj4"
            # (3 unchanged attributes hidden)
        }

      ~ frontend_endpoint {
          ~ host_name                               = "timj1.sandbox.platform.hmcts.net" -> "timjfd-sbox.azurefd.net"
            id                                      = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/frontendEndpoints/timj1"
          ~ name                                    = "timj1" -> "timjfd-sbox-azurefd-net"
          - web_application_firewall_policy_link_id = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoorWebApplicationFirewallPolicies/timj1sbox" -> null
            # (3 unchanged attributes hidden)

            # (1 unchanged block hidden)
        }
      ~ frontend_endpoint {
          ~ host_name                               = "timj4.sandbox.platform.hmcts.net" -> "timj1.sandbox.platform.hmcts.net"
            id                                      = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/frontendEndpoints/timj4"
          ~ name                                    = "timj4" -> "timj1"
          ~ web_application_firewall_policy_link_id = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoorWebApplicationFirewallPolicies/timj4sbox" -> "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoorWebApplicationFirewallPolicies/timj1sbox"
            # (3 unchanged attributes hidden)

            # (1 unchanged block hidden)
        }
      ~ frontend_endpoint {
          ~ host_name                               = "timjfd-sbox.azurefd.net" -> "timj4.sandbox.platform.hmcts.net"
            id                                      = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/frontendEndpoints/timjfd-sbox-azurefd-net"
          ~ name                                    = "timjfd-sbox-azurefd-net" -> "timj4"
          + web_application_firewall_policy_link_id = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoorWebApplicationFirewallPolicies/timj4sbox"
            # (3 unchanged attributes hidden)
        }

      ~ routing_rule {
          ~ accepted_protocols = [
              - "Http",
              + "Https",
            ]
          ~ frontend_endpoints = [
              - "timj1",
              + "timj4",
            ]
            id                 = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/routingRules/timj1HttpsRedirect"
          ~ name               = "timj1HttpsRedirect" -> "timj4"
            # (2 unchanged attributes hidden)

          + forwarding_configuration {
              + backend_pool_name                     = "timj4"
              + cache_enabled                         = true
              + cache_query_parameter_strip_directive = "StripNone"
              + cache_use_dynamic_compression         = false
              + forwarding_protocol                   = "HttpOnly"
            }

          - redirect_configuration {
              - redirect_protocol = "HttpsOnly" -> null
              - redirect_type     = "Moved" -> null
            }
        }
      ~ routing_rule {
          ~ frontend_endpoints = [
              - "timj2",
              + "timj1",
            ]
            id                 = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/routingRules/timj2HttpsRedirect"
          ~ name               = "timj2HttpsRedirect" -> "timj1HttpsRedirect"
            # (3 unchanged attributes hidden)

            # (1 unchanged block hidden)
        }
      ~ routing_rule {
          ~ accepted_protocols = [
              - "Https",
              + "Http",
            ]
          ~ frontend_endpoints = [
              - "timj4",
              + "timj2",
            ]
            id                 = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/timj-acme-test/providers/Microsoft.Network/frontDoors/timjfd-sbox/routingRules/timj4"
          ~ name               = "timj4" -> "timj2HttpsRedirect"
            # (2 unchanged attributes hidden)

          - forwarding_configuration {
              - backend_pool_name                     = "timj4" -> null
              - cache_enabled                         = true -> null
              - cache_query_parameter_strip_directive = "StripNone" -> null
              - cache_use_dynamic_compression         = false -> null
              - forwarding_protocol                   = "HttpOnly" -> null
            }

          + redirect_configuration {
              + redirect_protocol = "HttpsOnly"
              + redirect_type     = "Moved"
            }
        }
        # (9 unchanged blocks hidden)
    }

@WodansSon
Copy link
Collaborator Author

WodansSon commented Apr 27, 2021

@JayDoubleu, this is really weird... I used your exact config (except I used my own domain) and I get:

XXXXX@XXXXX-XXX-XXX:~/frontdoor/customerRepro$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

azurerm_resource_group.test: Refreshing state... [id=/subscriptions/XXXXX/resourceGroups/XXXXX-frontdoor-order-customer-repro]
azurerm_storage_account.site01: Refreshing state... [id=/subscriptions/XXXXX/resourceGroups/XXXXX-frontdoor-order-customer-repro/providers/Microsoft.Storage/storageAccounts/acctestfd001testsite01]
azurerm_storage_account.site02: Refreshing state... [id=/subscriptions/XXXXX/resourceGroups/XXXXX-frontdoor-order-customer-repro/providers/Microsoft.Storage/storageAccounts/acctestfd001testsite02]
azurerm_storage_account.site03: Refreshing state... [id=/subscriptions/XXXXX/resourceGroups/XXXXX-frontdoor-order-customer-repro/providers/Microsoft.Storage/storageAccounts/acctestfd001testsite03]
azurerm_frontdoor.default: Refreshing state... [id=/subscriptions/XXXXX/resourceGroups/XXXXX-frontdoor-order-customer-repro/providers/Microsoft.Network/frontDoors/acctest-FD-001-test]

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

At this point the only difference between our two environments is the terraform core runtime version:

XXXXXX@XXXXXX-XXX-XXX:~/frontdoor/customerRepro$ terraform -version
Terraform v0.12.29
+ provider.azurerm (unversioned)

CORRECTION:
I was just able to reproduce this and I see what is going on, the API is returning the JSON in random order. I think I will have a fix for this shortly.

@ghost
Copy link

ghost commented May 7, 2021

This has been released in version 2.58.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.58.0"
}
# ... other configuration ...

@favoretti
Copy link
Collaborator

@WodansSon @alec-pinson, a colleague of mine is also testing with their FD instance, seems so far so good.
@alec-pinson if you run into issues - can you please post here?

@sebader
Copy link
Contributor

sebader commented May 7, 2021

I tested so far with new as well as existing instances: works like a charm! No more random changes in terraform plan after the explicit_resource_order was built once! Need to test yet with a FD that has a custom domain.

But looks great, thanks a ton @WodansSon for this much needed work!

@ryanbowden
Copy link

@WodansSon, @favoretti we got an issue,

getting this on our plan:

Error: Invalid resource instance data in state

on frontdoor.tf line 1:
1: resource "azurerm_frontdoor" "redacted" {
 
Instance azurerm_frontdoor.redacted data could not be decoded from the
state: unsupported attribute "custom_https_configuration".

Guessing the state file needs to be cleaned up? Might try an import with new state file to see if that will make it work.

@WodansSon
Copy link
Collaborator Author

WodansSon commented May 7, 2021

@WodansSon, @favoretti we got an issue,

getting this on our plan:

Error: Invalid resource instance data in state

on frontdoor.tf line 1:
1: resource "azurerm_frontdoor" "redacted" {
 
Instance azurerm_frontdoor.redacted data could not be decoded from the
state: unsupported attribute "custom_https_configuration".

Guessing the state file needs to be cleaned up? Might try an import with new state file to see if that will make it work.

ARGH! Yep... that was my bad... that was not a scenario I had thought of, but it makes sense... I removed the support for that code block and field from the resource. I might have to implement a state migration function to remove this from the the state file. That is a great catch @ryanbowden... thanks for the heads up... now I just need to figure out a way to repro it... 🙃

If you go into your state file and remove that code block(custom_https_configuration) along with the custom_https_provisioning_enabled attribute it should work again... 🤞

This is actually way harder than it sounds, since I made the 3.0 hard cut between the front door resource and the custom https resource ahead of the 3.0 release... meaning, from now on, if you want a custom https domain you will also need to include a custom https configuration resource in your configuration file. So, with the new resource, if I remove the block and field from the main resource to avoid this error you will not be in the state you expect yourself to be in after the apply. I would have to detect this change and error out during the state migration because I did not see the blocks I deleted in the migration in the new configuration file, if that makes sense. It is doable, but I would need a bit more time to figure out the best way to detect this and correct it. Thanks again for reporting this issue @ryanbowden.

@favoretti
Copy link
Collaborator

favoretti commented May 7, 2021

I think you won't be able to statemigrate it. TF code needs to be refactored @ryanbowden so that you convert inline custom_https_configuration into azurerm_frontdoor_custom_https_configuration separate resources and import them unfortunately. That was pretty much the breaking change behind this.

@favoretti
Copy link
Collaborator

Or... Hmm... What you could do, if you have a test env, is generally remove FD from the state with terraform state rm and import it anew. Disclaimer: I have NOT tried this, YMMV.

@JayDoubleu
Copy link

Another way to go about it would be going back to previous provider version, moving custom_https_configuration into azurerm_frontdoor_custom_https_configuration block and bumping to 2.58.0 after apply.

@alec-pinson
Copy link

Hi, firstly thanks for this mine is now working perfectly!

One thing i've already mentioned to @favoretti regarding the import section of the doc
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/frontdoor_custom_https_configuration#import

currently:

terraform import azurerm_frontdoor_custom_https_configuration.example_custom_https_1 /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/mygroup1/providers/Microsoft.Network/frontDoors/frontdoor1/frontendEndpoints/endpoint1

however it looks like the new resource id type is

terraform import azurerm_frontdoor_custom_https_configuration.example_custom_https_1 /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/mygroup1/providers/Microsoft.Network/frontDoors/frontdoor1/customHttpsConfiguration/endpoint1

Although that being said I get the error below when trying to import:-

Error: parsing Resource ID "/subscriptions/xxxxxx/resourceGroups/xxxxx/providers/Microsoft.Network/frontDoors/xxxxx/customHttpsConfiguration/xxxx": ID was missing the `frontendEndpoints` element

... I've just remembered I think there was maybe an issue around casing or something for this, anyways instead of importing I allowed terraform to 'overwrite' changes in azure and im now getting the beautiful

No changes. Infrastructure is up-to-date.

@WodansSon
Copy link
Collaborator Author

Ok.... can I get a general consensus around these changes as you have experienced them thus far, I know there is some pain around what I have done, but over all would you view this as a positive change? I know there are some scenarios I overlooked, and I am sorry for that, but I did the best I could with the current issue, well not just issue many issues... lol... I feel this was the best case and I will continue to work on fixing the edge cases, but I believe for new front doors everything should be resolved. The existing front doors may and will find issues depending on their configurations and I think that a one time pain point might be acceptable, granted not ideal, but acceptable. Thoughts?

@JayDoubleu
Copy link

I definitely share the same view. Fixing already existing issues was already a huge challenge and it enables us to work with the provider in general. Left over edge cases at least have known workarounds while the main issue before this fix did not have any.
Great work @WodansSon , having this fixed and shipped is really appreciated.

@JayDoubleu
Copy link

Ps. One thing I just noticed is that front end endpoint needs to be created for azurerm_frontdoor_custom_https_configuration to reference it properly, if exampleFrontendEndpoint2 doesn't yet exist however, its being created as the same terraform plan it complains about invalid index and wants explicit number [0], [1] etc.

Error: Invalid index                                                                                                                                   
                                                                                                                                                       
 179:   frontend_endpoint_id              = azurerm_frontdoor.default.frontend_endpoints[local.custom_frontend_name]                                     
    |----------------                                                                                                                                  
    | azurerm_frontdoor.default.frontend_endpoint is list of object with 2 elements                                                                     
    | local.custom_frontend_name is "exampleFrontendEndpoint2"                                                                                                              
                                                                                                                                                       
The given key does not identify an element in this collection value: a number                                                                          
is required.     
resource "azurerm_frontdoor_custom_https_configuration" "example_custom_https_1" {
  frontend_endpoint_id              = azurerm_frontdoor.example.frontend_endpoints["exampleFrontendEndpoint2"]
  custom_https_provisioning_enabled = true

  custom_https_configuration {
    certificate_source                      = "AzureKeyVault"
    azure_key_vault_certificate_secret_name = "examplefd1"
    azure_key_vault_certificate_vault_id    = data.azurerm_key_vault.vault.id
  }
}

@favoretti
Copy link
Collaborator

@JayDoubleu depends_on in https configuration resource should alleviate that?

@JayDoubleu
Copy link

JayDoubleu commented May 7, 2021

@favoretti tried it but without success unfortunately. Seems like some sort of bug in dependency graph ?
I even tried azurerm_frontdoor.example.frontend_endpoint["exampleFrontendEndpoint2"].id with the same results.

@timja
Copy link
Contributor

timja commented May 7, 2021

@favoretti tried it but without success unfortunately.

I ended up not accessing through the set and just building the ID, was the only way I could get it to work in all scenarios

https://github.com/hmcts/terraform-module-frontdoor/blob/96b0d387b848c292385696a832e2494fe936b39b/frontdoor.tf#L210

@favoretti
Copy link
Collaborator

favoretti commented May 7, 2021

Hmm, this is interesting. Regardless of the fact that frontend_endpoints is something I added as a computed map exactly to prep for TypeSet conversion that never happened... You still should be able to reference them by index, like azurerm_frontdoor.frontend_endpoint[0].id. Potentially, since TypeSet conversion isn't going to happen we can even deprecate/remove those maps if they aren't working as expected.

Funny thing is - I did write an acceptance test checking exactly that and it passes..

@JayDoubleu
Copy link

@timja I ended up with something similar "${azurerm_frontdoor.example.id}/frontendEndpoints/${local.custom_frontend_name}"
This way I don't need depends_on as referencing main front door adds it to dependency graph.

@ryanbowden
Copy link

@WodansSon so after making a few changes in the state files ours is working correctly with no issues now

We had another issues around a log analytics but that's because it had part created it and then state file was messed up we remove it from state file and removed it in portal and now all created with no issues.

Thank you so much to you and everyone else fixing this issue. ❤️

@timja
Copy link
Contributor

timja commented May 7, 2021

I ended up with something similar "${azurerm_frontdoor.example.id}/frontendEndpoints/${local.custom_frontend_name}"
This way I don't need depends_on as referencing main front door adds it to dependency graph.

oo nice, I can move to that.

Hmm, this is interesting. Regardless of the fact that frontend_endpoints is something I added as a computed map exactly to prep for TypeSet conversion that never happened... You still should be able to reference them by index, like azurerm_frontdoor.frontend_endpoint[0].id. Potentially, since TypeSet conversion isn't going to happen we can even deprecate/remove those maps if they aren't working as expected.

Referencing by index seems too risky, as indexes will get shuffled when you add/remove endpoints

@alec-pinson
Copy link

alec-pinson commented May 7, 2021

Hi some snippets of my code which is working:-

resource "azurerm_frontdoor" "this" {
......
  // Frontend Endpoint
  dynamic "frontend_endpoint" {
    for_each = local.configs

    content {
      name                                    = frontend_endpoint.key
      host_name                               = frontend_endpoint.value.waf.frontend_domain
      web_application_firewall_policy_link_id = try(frontend_endpoint.value.waf.waf_policy_id, var.default_waf_policy)
      session_affinity_enabled                = false
      session_affinity_ttl_seconds            = 0
    }
  }

}
// apply certificate secret
resource "azurerm_frontdoor_custom_https_configuration" "this" {
  for_each                          = local.configs
  frontend_endpoint_id              = azurerm_frontdoor.this.frontend_endpoints[each.key]
  custom_https_provisioning_enabled = true

  custom_https_configuration {
    certificate_source                         = "AzureKeyVault"
    azure_key_vault_certificate_secret_name    = data.azurerm_key_vault_secret.this[each.key].name
    azure_key_vault_certificate_secret_version = data.azurerm_key_vault_secret.this[each.key].version
    azure_key_vault_certificate_vault_id       = var.certificate_keyvault_id
  }
}

@timja
Copy link
Contributor

timja commented May 7, 2021

@alec-pinson make sure you validate deleting endpoints and then creating a new one with that. I think I had an issue around there.

@alec-pinson
Copy link

alec-pinson commented May 7, 2021

@alec-pinson make sure you validate deleting endpoints and then creating a new one with that. I think I had an issue around there.

ah ok sorry my bad i haven't created/deleted any endpoints

@JayDoubleu
Copy link

JayDoubleu commented May 7, 2021

I'm assuming referencing frontendEndpoints/ directly will make azurerm_frontdoor_custom_https_configuration freak out when the endpoint goes missing. Definitely not a long term solution but hey, Rome wasn't built in a day ;)
**probably not an issue if azurerm_frontdoor_custom_https_configuration is dynamically created tho.

@mihma
Copy link

mihma commented May 7, 2021

First of all big thanks @WodansSon for this work! I've just run the latest provider release against my front doors with various dynamic blocks and custom https configurations and it worked like a charm. I have faced the same issue though with the azurerm_frontdoor_custom_https_configuration which I create dynamically. Had to hard-code the frontend_endpoint_id similar to what people have been doing here. Keen to know if it's just a temporary solution or not.

@github-actions
Copy link

github-actions bot commented Jun 7, 2021

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.