resource/aws_cloudfront_distribution: Remove viewer_certificate configuration block argument ConflictsWith usage and fix various related issues with deployment timing

[bflad]

Closes #7773
Closes #3077
Closes #1074
Closes #260

Here we remove the problematic viewer_certificate configuration block argument ConflictsWith schema configuration as it interferes with Terraform Module usage until Terraform 0.12 is more prevalent. More details: #7773 (comment)

When writing acceptance testing to cover setting both the viewer_certificate configuration block acm_certificate_arn and cloudfront_default_certificate arguments being defined, the below error was consistently happening when the test configuration included enabled = false:

--- FAIL: TestAccAWSCloudFrontDistribution_ViewerCertificate_AcmCertificateArn (1935.57s)
    testing.go:599: Error destroying resource! WARNING: Dangling resources
        may exist. The full state and error is shown below.

        Error: Error applying: 1 error occurred:
          * aws_cloudfront_distribution.test (destroy): 1 error occurred:
          * aws_cloudfront_distribution.test: CloudFront Distribution E3GDAPNU6UPO0O cannot be deleted: PreconditionFailed: The request failed because it didn't meet the preconditions in one or more request-header fields.
          status code: 412, request id: 4e73a086-3c33-11e9-832f-7732257f45e8

While debugging this the following further issues were encountered:

• The resource did not wait for deployment to complete on creation and updates so in the acceptance testing the deletion function was always handling InProgress operations.
• Disabled distributions would always update the distribution on deletion to disable them without checking if it was necessary, causing unnecessary delays.
• The PreconditionFailed error seemed to be related to some eventual consistency issue within CloudFront right after disabling the distribution, which was always done. Retrying was a sufficient workaround for the error.
• The deletion process did not ignore NoSuchDistribution errors such as the below:

--- FAIL: TestAccAWSCloudFrontDistribution_ViewerCertificate_AcmCertificateArn (1835.97s)
    testing.go:599: Error destroying resource! WARNING: Dangling resources
        may exist. The full state and error is shown below.

        Error: Error applying: 1 error occurred:
          * aws_cloudfront_distribution.test (destroy): 1 error occurred:
          * aws_cloudfront_distribution.test: CloudFront Distribution E2HQM77NFHV9T cannot be deleted: NoSuchDistribution: The specified distribution does not exist.

This changeset bundles all these fixes together as they are related.

Previous output for ConflictsWith acceptance testing:

--- FAIL: TestAccAWSCloudFrontDistribution_ViewerCertificate_AcmCertificateArn_ConflictsWithCloudFrontDefaultCertificate (1.64s)
    testing.go:538: Step 0 error: config is invalid: 2 problems:

        - aws_cloudfront_distribution.test: "viewer_certificate.0.acm_certificate_arn": conflicts with viewer_certificate.0.cloudfront_default_certificate
        - aws_cloudfront_distribution.test: "viewer_certificate.0.cloudfront_default_certificate": conflicts with viewer_certificate.0.acm_certificate_arn

Output from acceptance testing:

--- PASS: TestAccAWSCloudFrontDistribution_Origin_EmptyDomainName (2.15s)
--- PASS: TestAccAWSCloudFrontDistribution_Origin_EmptyOriginID (2.19s)
--- PASS: TestAccAWSCloudFrontDistribution_ViewerCertificate_AcmCertificateArn_ConflictsWithCloudFrontDefaultCertificate (1915.48s)
--- PASS: TestAccAWSCloudFrontDistribution_ViewerCertificate_AcmCertificateArn (1958.08s)
--- PASS: TestAccAWSCloudFrontDistribution_noCustomErrorResponseConfig (2121.58s)
--- PASS: TestAccAWSCloudFrontDistribution_HTTP11Config (2123.60s)
--- PASS: TestAccAWSCloudFrontDistribution_orderedCacheBehavior (2126.73s)
--- PASS: TestAccAWSCloudFrontDistribution_noOptionalItemsConfig (2126.82s)
--- PASS: TestAccAWSCloudFrontDistribution_IsIPV6EnabledConfig (2176.09s)
--- PASS: TestAccAWSCloudFrontDistribution_customOrigin (2178.92s)
--- PASS: TestAccAWSCloudFrontDistribution_S3Origin (2178.98s)
--- PASS: TestAccAWSCloudFrontDistribution_multiOrigin (2179.08s)
--- PASS: TestAccAWSCloudFrontDistribution_S3OriginWithTags (3251.14s)

[nywilken]

This looks good to me. Nicely down with the added checks and detailed PR description.

[bflad]

This has been released in version 2.1.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[m-dunbar]

#7794 introduces a significant workflow problem with timing for cloudfront distribution updates.

Prior to this change, updating an origin path, for example, took seconds to complete. Now, however, the TF apply sits and waits until propagation throughout the cloudfront edge network completes.

This in turn destroys automated workflow (timing changes from taking 2 seconds on average to complete the distribution update to an average of 20 plus minutes, with peak times of over an hour and 20 plus minutes.

A significant change like this needs to have a boolean attribute added to allow users to choose whether or not to wait for distribution propagation.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!