From 09596a08631497a9f552090f04032cbbe02830b7 Mon Sep 17 00:00:00 2001 From: Brian Flad Date: Wed, 2 Oct 2019 13:36:30 -0400 Subject: [PATCH] service/ec2: Updated handling for Lambda Hyperplane ENIs Reference: https://github.com/terraform-providers/terraform-provider-aws/issues/10044 Reference: https://github.com/terraform-providers/terraform-provider-aws/pull/10114 Reference: https://github.com/terraform-providers/terraform-provider-aws/issues/10329 The introduction of [improved VPC networking for Lambda]() brought some welcome enhancements to Lambda functionality, but initially has some unintentional consequences when working with Terraform due to the underlying infrastructure changes. The main issue is that these new Hyperplane ENIs associated with Lambda take additional time currently to detach/delete and that the Lambda service itself is the owner of these ENIs, which prevents early detachment. In working with the AWS Lambda service team, we have received some confirmation on expected detachment/deletion timeframes for Lambda Hyperplane ENIs. Using this information, we set the Lambda ENI timeout to be at a minimum the expected deletion time to match the service expectations without adjusting the overall default `aws_security_group` or `aws_subnet` resource deletion timeouts. This is to ensure legitimate `DependencyViolation` errors return to operators in a fairly timely manner (left as 10 minutes and 20 minutes respectfully). Output from AWS Commerical (us-east-2 - Hyperplane enabled) ``` --- PASS: TestAccAWSLambdaFunction_basic (23.37s) --- PASS: TestAccAWSLambdaFunction_concurrency (30.76s) --- PASS: TestAccAWSLambdaFunction_concurrencyCycle (43.12s) --- PASS: TestAccAWSLambdaFunction_DeadLetterConfig (42.40s) --- PASS: TestAccAWSLambdaFunction_DeadLetterConfigUpdated (41.70s) --- PASS: TestAccAWSLambdaFunction_EmptyVpcConfig (22.99s) --- PASS: TestAccAWSLambdaFunction_encryptedEnvVariables (51.21s) --- PASS: TestAccAWSLambdaFunction_envVariables (45.14s) --- PASS: TestAccAWSLambdaFunction_expectFilenameAndS3Attributes (10.90s) --- PASS: TestAccAWSLambdaFunction_importLocalFile (31.12s) --- PASS: TestAccAWSLambdaFunction_importLocalFile_VPC (1422.82s) --- PASS: TestAccAWSLambdaFunction_importS3 (22.66s) --- PASS: TestAccAWSLambdaFunction_Layers (34.75s) --- PASS: TestAccAWSLambdaFunction_LayersUpdate (54.60s) --- PASS: TestAccAWSLambdaFunction_localUpdate (31.40s) --- PASS: TestAccAWSLambdaFunction_localUpdate_nameOnly (24.21s) --- PASS: TestAccAWSLambdaFunction_nilDeadLetterConfig (12.71s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_java8 (23.05s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_NodeJs10x (26.99s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_NodeJs810 (26.53s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_noRuntime (0.72s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_provided (18.66s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_python27 (27.62s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_python36 (22.87s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_python37 (27.09s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_ruby25 (27.87s) --- PASS: TestAccAWSLambdaFunction_s3 (22.59s) --- PASS: TestAccAWSLambdaFunction_s3Update_basic (32.58s) --- PASS: TestAccAWSLambdaFunction_s3Update_unversioned (31.07s) --- PASS: TestAccAWSLambdaFunction_tags (42.41s) --- PASS: TestAccAWSLambdaFunction_tracingConfig (39.12s) --- PASS: TestAccAWSLambdaFunction_updateRuntime (29.16s) --- PASS: TestAccAWSLambdaFunction_versioned (28.09s) --- PASS: TestAccAWSLambdaFunction_versionedUpdate (47.13s) --- PASS: TestAccAWSLambdaFunction_VPC (1331.55s) --- PASS: TestAccAWSLambdaFunction_VPC_withInvocation (1376.24s) --- PASS: TestAccAWSLambdaFunction_VpcConfig_ProperIamDependencies (1327.69s) --- PASS: TestAccAWSLambdaFunction_VPCRemoval (1490.19s) --- PASS: TestAccAWSLambdaFunction_VPCUpdate (1685.40s) ``` Output from AWS Commercial (us-west-2 - Hyperplane not deployed) ``` --- PASS: TestAccAWSLambdaFunction_basic (40.50s) --- PASS: TestAccAWSLambdaFunction_concurrency (47.79s) --- PASS: TestAccAWSLambdaFunction_concurrencyCycle (62.65s) --- PASS: TestAccAWSLambdaFunction_DeadLetterConfig (55.95s) --- PASS: TestAccAWSLambdaFunction_DeadLetterConfigUpdated (50.23s) --- PASS: TestAccAWSLambdaFunction_EmptyVpcConfig (37.47s) --- PASS: TestAccAWSLambdaFunction_encryptedEnvVariables (73.66s) --- PASS: TestAccAWSLambdaFunction_envVariables (80.88s) --- PASS: TestAccAWSLambdaFunction_expectFilenameAndS3Attributes (22.59s) --- PASS: TestAccAWSLambdaFunction_importLocalFile (42.78s) --- PASS: TestAccAWSLambdaFunction_importLocalFile_VPC (39.40s) --- PASS: TestAccAWSLambdaFunction_importS3 (36.62s) --- PASS: TestAccAWSLambdaFunction_Layers (53.78s) --- PASS: TestAccAWSLambdaFunction_LayersUpdate (89.78s) --- PASS: TestAccAWSLambdaFunction_localUpdate (54.31s) --- PASS: TestAccAWSLambdaFunction_localUpdate_nameOnly (56.10s) --- PASS: TestAccAWSLambdaFunction_nilDeadLetterConfig (26.12s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_java8 (46.49s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_NodeJs10x (52.25s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_NodeJs810 (43.59s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_noRuntime (2.71s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_provided (43.88s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_python27 (47.91s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_python36 (45.95s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_python37 (41.40s) --- PASS: TestAccAWSLambdaFunction_runtimeValidation_ruby25 (50.32s) --- PASS: TestAccAWSLambdaFunction_s3 (35.28s) --- PASS: TestAccAWSLambdaFunction_s3Update_basic (57.89s) --- PASS: TestAccAWSLambdaFunction_s3Update_unversioned (58.81s) --- PASS: TestAccAWSLambdaFunction_tags (75.77s) --- PASS: TestAccAWSLambdaFunction_tracingConfig (55.61s) --- PASS: TestAccAWSLambdaFunction_updateRuntime (57.19s) --- PASS: TestAccAWSLambdaFunction_versioned (33.52s) --- PASS: TestAccAWSLambdaFunction_versionedUpdate (58.25s) --- PASS: TestAccAWSLambdaFunction_VPC (56.81s) --- PASS: TestAccAWSLambdaFunction_VPC_withInvocation (86.81s) --- PASS: TestAccAWSLambdaFunction_VpcConfig_ProperIamDependencies (42.99s) --- PASS: TestAccAWSLambdaFunction_VPCRemoval (80.28s) --- PASS: TestAccAWSLambdaFunction_VPCUpdate (81.84s) --- PASS: TestAccAWSSecurityGroup_basic (10.14s) --- PASS: TestAccAWSSecurityGroup_Change (19.36s) --- PASS: TestAccAWSSecurityGroup_CIDRandGroups (31.78s) --- PASS: TestAccAWSSecurityGroup_DefaultEgress_Classic (6.53s) --- PASS: TestAccAWSSecurityGroup_DefaultEgress_VPC (25.29s) --- PASS: TestAccAWSSecurityGroup_drift (7.55s) --- PASS: TestAccAWSSecurityGroup_drift_complex (31.62s) --- PASS: TestAccAWSSecurityGroup_Egress_ConfigMode (23.76s) --- PASS: TestAccAWSSecurityGroup_egressWithPrefixList (24.51s) --- PASS: TestAccAWSSecurityGroup_failWithDiffMismatch (12.13s) --- PASS: TestAccAWSSecurityGroup_forceRevokeRules_false (1228.05s) --- PASS: TestAccAWSSecurityGroup_forceRevokeRules_true (1242.70s) --- PASS: TestAccAWSSecurityGroup_generatedName (25.26s) --- PASS: TestAccAWSSecurityGroup_importBasic (12.91s) --- PASS: TestAccAWSSecurityGroup_importIPRangeAndSecurityGroupWithSameRules (14.68s) --- PASS: TestAccAWSSecurityGroup_importIPRangesWithSameRules (12.19s) --- PASS: TestAccAWSSecurityGroup_importIpv6 (30.08s) --- PASS: TestAccAWSSecurityGroup_importPrefixList (25.01s) --- PASS: TestAccAWSSecurityGroup_importSelf (31.64s) --- PASS: TestAccAWSSecurityGroup_importSourceSecurityGroup (30.19s) --- PASS: TestAccAWSSecurityGroup_Ingress_ConfigMode (23.47s) --- PASS: TestAccAWSSecurityGroup_ingressWithCidrAndSGs (31.60s) --- PASS: TestAccAWSSecurityGroup_ingressWithCidrAndSGs_classic (9.86s) --- PASS: TestAccAWSSecurityGroup_ingressWithPrefixList (44.12s) --- PASS: TestAccAWSSecurityGroup_invalidCIDRBlock (1.28s) --- PASS: TestAccAWSSecurityGroup_ipv4andipv6Egress (11.90s) --- PASS: TestAccAWSSecurityGroup_ipv6 (12.77s) --- PASS: TestAccAWSSecurityGroup_MultiIngress (12.33s) --- PASS: TestAccAWSSecurityGroup_namePrefix (6.47s) --- PASS: TestAccAWSSecurityGroup_RuleDescription (26.52s) --- PASS: TestAccAWSSecurityGroup_ruleGathering (24.55s) --- PASS: TestAccAWSSecurityGroup_ruleLimitCidrBlockExceededAppend (48.89s) --- PASS: TestAccAWSSecurityGroup_ruleLimitExceededAllNew (53.89s) --- PASS: TestAccAWSSecurityGroup_ruleLimitExceededAppend (50.48s) --- PASS: TestAccAWSSecurityGroup_ruleLimitExceededPrepend (54.09s) --- PASS: TestAccAWSSecurityGroup_rulesDropOnError (22.40s) --- PASS: TestAccAWSSecurityGroup_self (11.93s) --- PASS: TestAccAWSSecurityGroup_tags (40.86s) --- PASS: TestAccAWSSecurityGroup_vpc (10.39s) --- PASS: TestAccAWSSecurityGroup_vpcNegOneIngress (10.55s) --- PASS: TestAccAWSSecurityGroup_vpcProtoNumIngress (11.84s) --- PASS: TestAccAWSSubnet_availabilityZoneId (26.56s) --- PASS: TestAccAWSSubnet_basic (26.69s) --- PASS: TestAccAWSSubnet_enableIpv6 (42.97s) --- PASS: TestAccAWSSubnet_ipv6 (69.30s) ``` --- aws/resource_aws_lambda_function_test.go | 104 +++++++++++++++++++ aws/resource_aws_network_interface.go | 32 +++--- aws/resource_aws_security_group.go | 30 ++++-- aws/resource_aws_subnet.go | 2 +- website/docs/r/lambda_function.html.markdown | 2 + website/docs/r/security_group.html.markdown | 6 +- website/docs/r/subnet.html.markdown | 6 +- 7 files changed, 154 insertions(+), 28 deletions(-) diff --git a/aws/resource_aws_lambda_function_test.go b/aws/resource_aws_lambda_function_test.go index 370d219ad068..4b8102955776 100644 --- a/aws/resource_aws_lambda_function_test.go +++ b/aws/resource_aws_lambda_function_test.go @@ -828,6 +828,33 @@ func TestAccAWSLambdaFunction_VPC_withInvocation(t *testing.T) { }) } +// Reference: https://github.com/terraform-providers/terraform-provider-aws/issues/10044 +func TestAccAWSLambdaFunction_VpcConfig_ProperIamDependencies(t *testing.T) { + var function lambda.GetFunctionOutput + + rName := acctest.RandomWithPrefix("tf-acc-test") + resourceName := "aws_lambda_function.test" + vpcResourceName := "aws_vpc.test" + + resource.ParallelTest(t, resource.TestCase{ + PreCheck: func() { testAccPreCheck(t) }, + Providers: testAccProviders, + CheckDestroy: testAccCheckLambdaFunctionDestroy, + Steps: []resource.TestStep{ + { + Config: testAccAWSLambdaConfigVpcConfigProperIamDependencies(rName), + Check: resource.ComposeTestCheckFunc( + testAccCheckAwsLambdaFunctionExists(resourceName, rName, &function), + resource.TestCheckResourceAttr(resourceName, "vpc_config.#", "1"), + resource.TestCheckResourceAttr(resourceName, "vpc_config.0.subnet_ids.#", "1"), + resource.TestCheckResourceAttr(resourceName, "vpc_config.0.security_group_ids.#", "1"), + resource.TestCheckResourceAttrPair(resourceName, "vpc_config.0.vpc_id", vpcResourceName, "id"), + ), + }, + }, + }) +} + func TestAccAWSLambdaFunction_EmptyVpcConfig(t *testing.T) { var conf lambda.GetFunctionOutput @@ -1873,6 +1900,83 @@ resource "aws_lambda_function" "lambda_function_test" { `, fileName, funcName) } +func testAccAWSLambdaConfigVpcConfigProperIamDependencies(rName string) string { + return fmt.Sprintf(` +data "aws_partition" "current" {} + +resource "aws_iam_role_policy_attachment" "test" { + policy_arn = "arn:${data.aws_partition.current.partition}:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole" + role = "${aws_iam_role.test.id}" +} + +resource "aws_iam_role" "test" { + name = %[1]q + + assume_role_policy = < **NOTE:** Due to [AWS Lambda improved VPC networking changes that began deploying in September 2019](https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/), EC2 subnets and security groups associated with Lambda Functions can take up to 45 minutes to successfully delete. Terraform AWS Provider version 2.31.0 and later automatically handles this increased timeout, however prior versions require setting the customizable deletion timeouts of those Terraform resources to 45 minutes (`delete = "45m"`). AWS and HashiCorp are working together to reduce the amount of time required for resource deletion and updates can be tracked in this [GitHub issue](https://github.com/terraform-providers/terraform-provider-aws/issues/10329). + ## Example Usage ### Basic Example diff --git a/website/docs/r/security_group.html.markdown b/website/docs/r/security_group.html.markdown index 4a52124ff41b..32614e78a396 100644 --- a/website/docs/r/security_group.html.markdown +++ b/website/docs/r/security_group.html.markdown @@ -18,6 +18,8 @@ a conflict of rule settings and will overwrite rules. ~> **NOTE:** Referencing Security Groups across VPC peering has certain restrictions. More information is available in the [VPC Peering User Guide](https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-security-groups.html). +~> **NOTE:** Due to [AWS Lambda improved VPC networking changes that began deploying in September 2019](https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/), security groups associated with Lambda Functions can take up to 45 minutes to successfully delete. Terraform AWS Provider version 2.31.0 and later automatically handles this increased timeout, however prior versions require setting the [customizable deletion timeout](#timeouts) to 45 minutes (`delete = "45m"`). AWS and HashiCorp are working together to reduce the amount of time required for resource deletion and updates can be tracked in this [GitHub issue](https://github.com/terraform-providers/terraform-provider-aws/issues/10329). + ## Example Usage Basic usage @@ -184,8 +186,8 @@ In addition to all arguments above, the following attributes are exported: `aws_security_group` provides the following [Timeouts](/docs/configuration/resources.html#timeouts) configuration options: -- `create` - (Default `10 minutes`) How long to wait for a security group to be created. -- `delete` - (Default `30 minutes`) How long to wait for a security group to be deleted. +- `create` - (Default `10m`) How long to wait for a security group to be created. +- `delete` - (Default `10m`) How long to retry on `DependencyViolation` errors during security group deletion from lingering ENIs left by certain AWS services such as Elastic Load Balancing. NOTE: Lambda ENIs can take up to 45 minutes to delete, which is not affected by changing this customizable timeout (in version 2.31.0 and later of the Terraform AWS Provider) unless it is increased above 45 minutes. ## Import diff --git a/website/docs/r/subnet.html.markdown b/website/docs/r/subnet.html.markdown index 0199b4baa18a..e83bebf28e6e 100644 --- a/website/docs/r/subnet.html.markdown +++ b/website/docs/r/subnet.html.markdown @@ -9,6 +9,8 @@ description: |- Provides an VPC subnet resource. +~> **NOTE:** Due to [AWS Lambda improved VPC networking changes that began deploying in September 2019](https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/), subnets associated with Lambda Functions can take up to 45 minutes to successfully delete. Terraform AWS Provider version 2.31.0 and later automatically handles this increased timeout, however prior versions require setting the [customizable deletion timeout](#timeouts) to 45 minutes (`delete = "45m"`). AWS and HashiCorp are working together to reduce the amount of time required for resource deletion and updates can be tracked in this [GitHub issue](https://github.com/terraform-providers/terraform-provider-aws/issues/10329). + ## Example Usage ### Basic Usage @@ -73,8 +75,8 @@ In addition to all arguments above, the following attributes are exported: `aws_subnet` provides the following [Timeouts](/docs/configuration/resources.html#timeouts) configuration options: -- `create` - (Default `10 minutes`) How long to wait for a subnet to be created. -- `delete` - (Default `30 minutes`) How long to wait for a subnet to be deleted. +- `create` - (Default `10m`) How long to wait for a subnet to be created. +- `delete` - (Default `20m`) How long to retry on `DependencyViolation` errors during subnet deletion from lingering ENIs left by certain AWS services such as Elastic Load Balancing. NOTE: Lambda ENIs can take up to 45 minutes to delete, which is not affected by changing this customizable timeout (in version 2.31.0 and later of the Terraform AWS Provider) unless it is increased above 45 minutes. ## Import