Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Setup Region/Partition Agnostic aws_instance Acceptance Testing #4987

Closed
bflad opened this issue Jun 26, 2018 · 5 comments
Closed

Setup Region/Partition Agnostic aws_instance Acceptance Testing #4987

bflad opened this issue Jun 26, 2018 · 5 comments
Assignees
@YakDriver
Labels
stale Old or inactive issues managed by automation, if no further action taken these will get closed. technical-debt Addresses areas of the codebase that need refactoring or redesign. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.

Comments

@bflad
Copy link
Member

bflad commented Jun 26, 2018

Its not currently possible to run the aws_instance resource and data source acceptance testing outside us-west-2 and get meaningful results. For example, setting AWS_DEFAULT_REGION=us-east-1 while running the tests today returns: Tests failed: 61 (59 new), passed: 6

Many of the failures are similar to:

--- FAIL: TestAccAWSInstance_basic (8.76s)
    testing.go:518: Step 0 error: Check failed: InvalidZone.NotFound: The zone 'us-west-2a' does not exist.

And:

--- FAIL: TestAccAWSInstance_disableApiTermination (17.22s)
    testing.go:518: Step 0 error: Error applying: 1 error(s) occurred:
        
        * aws_instance.foo: 1 error(s) occurred:
        
        * aws_instance.foo: Error launching source instance: InvalidAMIID.NotFound: The image id '[ami-4fccb37f]' does not exist

We can switch to using the aws_region, aws_availability_zones, and aws_partition data sources as appropriate to remove hardcoded us-west-2 pieces. For acceptance tests that cannot occur outside of us-west-2 (due to very specific AMIs, etc.), we can set them up with t.Skip().
@bflad bflad added tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. technical-debt Addresses areas of the codebase that need refactoring or redesign. labels Jun 26, 2018
bflad added a commit that referenced this issue Apr 17, 2019
@bflad
resource/aws_instance: Always refresh associate_public_ip_address and…
2a23d08 
… ipv6_address_count into Terraform state

In the Terraform 0.12 Provider SDK acceptance testing framework, attributes that do not call `ResourceData.Set()` are no longer automatically ignored during `ImportStateVerify` testing. Here we opt to always refresh the `associate_public_ip_address` and `ipv6_address_count` attributes into the Terraform state. This change is backwards compatible.

This changeset is the bare minimum to ensure the existing acceptance testing is in a better passing state with the Terraform 0.12 Provider SDK, however there are likely many more additional changes required once the `ImportStateVerify` test step is applied to all acceptance tests for the resource as part of some upcoming technical debt work (#4987).

Previous output from Terraform 0.12 Provider SDK acceptance testing:

```
--- FAIL: TestAccAWSInstance_importBasic (93.87s)
    testing.go:568: Step 1 error: ImportStateVerify attributes not equivalent. Difference is shown below. Top is actual, bottom is expected.

        (map[string]string) {
        }

        (map[string]string) (len=1) {
         (string) (len=18) "ipv6_address_count": (string) (len=1) "0"
        }

--- FAIL: TestAccAWSInstance_importInDefaultVpcBySgId (193.29s)
    testing.go:568: Step 1 error: ImportStateVerify attributes not equivalent. Difference is shown below. Top is actual, bottom is expected.

        (map[string]string) {
        }

        (map[string]string) (len=1) {
         (string) (len=18) "ipv6_address_count": (string) (len=1) "0"
        }

--- FAIL: TestAccAWSInstance_importInDefaultVpcBySgName (183.91s)
    testing.go:568: Step 1 error: ImportStateVerify attributes not equivalent. Difference is shown below. Top is actual, bottom is expected.

        (map[string]string) {
        }

        (map[string]string) (len=1) {
         (string) (len=18) "ipv6_address_count": (string) (len=1) "0"
        }

--- FAIL: TestAccAWSInstance_importInEc2Classic (108.87s)
    testing.go:568: Step 1 error: ImportStateVerify attributes not equivalent. Difference is shown below. Top is actual, bottom is expected.

        (map[string]string) {
        }

        (map[string]string) (len=2) {
         (string) (len=27) "associate_public_ip_address": (string) (len=5) "false",
         (string) (len=18) "ipv6_address_count": (string) (len=1) "0"
        }
```

Output from Terraform 0.11 Provider SDK acceptance testing:

```
--- PASS: TestAccAWSInstance_addSecondaryInterface (85.27s)
--- PASS: TestAccAWSInstance_addSecurityGroupNetworkInterface (227.80s)
--- PASS: TestAccAWSInstance_associatePublic_defaultPrivate (183.36s)
--- PASS: TestAccAWSInstance_associatePublic_defaultPublic (51.53s)
--- PASS: TestAccAWSInstance_associatePublic_explicitPrivate (52.10s)
--- PASS: TestAccAWSInstance_associatePublic_explicitPublic (51.22s)
--- PASS: TestAccAWSInstance_associatePublic_overridePrivate (51.22s)
--- PASS: TestAccAWSInstance_associatePublic_overridePublic (71.58s)
--- PASS: TestAccAWSInstance_associatePublicIPAndPrivateIP (50.40s)
--- PASS: TestAccAWSInstance_basic (317.85s)
--- PASS: TestAccAWSInstance_blockDevices (101.23s)
--- PASS: TestAccAWSInstance_changeInstanceType (328.94s)
--- PASS: TestAccAWSInstance_creditSpecification_isNotAppliedToNonBurstable (151.69s)
--- PASS: TestAccAWSInstance_creditSpecification_standardCpuCredits (178.74s)
--- PASS: TestAccAWSInstance_creditSpecification_standardCpuCredits_t2Tot3Taint (340.18s)
--- PASS: TestAccAWSInstance_creditSpecification_unlimitedCpuCredits (56.89s)
--- PASS: TestAccAWSInstance_creditSpecification_unlimitedCpuCredits_t2Tot3Taint (320.30s)
--- PASS: TestAccAWSInstance_creditSpecification_unspecifiedDefaultsToStandard (183.49s)
--- PASS: TestAccAWSInstance_creditSpecification_updateCpuCredits (188.24s)
--- PASS: TestAccAWSInstance_creditSpecificationT3_standardCpuCredits (290.36s)
--- PASS: TestAccAWSInstance_creditSpecificationT3_unlimitedCpuCredits (291.21s)
--- PASS: TestAccAWSInstance_creditSpecificationT3_unspecifiedDefaultsToUnlimited (284.34s)
--- PASS: TestAccAWSInstance_creditSpecificationT3_updateCpuCredits (116.93s)
--- PASS: TestAccAWSInstance_disableApiTermination (257.56s)
--- PASS: TestAccAWSInstance_disappears (258.64s)
--- PASS: TestAccAWSInstance_forceNewAndTagsDrift (277.56s)
--- PASS: TestAccAWSInstance_getPasswordData_falseToTrue (154.93s)
--- PASS: TestAccAWSInstance_getPasswordData_trueToFalse (145.14s)
--- PASS: TestAccAWSInstance_GP2IopsDevice (79.88s)
--- PASS: TestAccAWSInstance_GP2WithIopsValue (89.27s)
--- PASS: TestAccAWSInstance_importBasic (201.64s)
--- PASS: TestAccAWSInstance_importInDefaultVpcBySgId (190.99s)
--- PASS: TestAccAWSInstance_importInDefaultVpcBySgName (78.33s)
--- PASS: TestAccAWSInstance_importInEc2Classic (209.77s)
--- PASS: TestAccAWSInstance_instanceProfileChange (143.04s)
--- PASS: TestAccAWSInstance_ipv6_supportAddressCount (202.10s)
--- PASS: TestAccAWSInstance_ipv6_supportAddressCountWithIpv4 (218.22s)
--- PASS: TestAccAWSInstance_ipv6AddressCountAndSingleAddressCausesError (9.72s)
--- PASS: TestAccAWSInstance_keyPairCheck (78.74s)
--- PASS: TestAccAWSInstance_multipleRegions (280.08s)
--- PASS: TestAccAWSInstance_NetworkInstanceRemovingAllSecurityGroups (73.45s)
--- PASS: TestAccAWSInstance_NetworkInstanceSecurityGroups (96.32s)
--- PASS: TestAccAWSInstance_NetworkInstanceVPCSecurityGroupIDs (68.35s)
--- PASS: TestAccAWSInstance_noAMIEphemeralDevices (53.25s)
--- PASS: TestAccAWSInstance_placementGroup (174.29s)
--- PASS: TestAccAWSInstance_primaryNetworkInterface (153.92s)
--- PASS: TestAccAWSInstance_primaryNetworkInterfaceSourceDestCheck (184.91s)
--- PASS: TestAccAWSInstance_privateIP (72.60s)
--- PASS: TestAccAWSInstance_rootBlockDeviceMismatch (172.17s)
--- PASS: TestAccAWSInstance_rootInstanceStore (66.92s)
--- PASS: TestAccAWSInstance_sourceDestCheck (114.61s)
--- PASS: TestAccAWSInstance_tags (265.95s)
--- PASS: TestAccAWSInstance_UserData_EmptyStringToUnspecified (79.90s)
--- PASS: TestAccAWSInstance_UserData_UnspecifiedToEmptyString (78.14s)
--- PASS: TestAccAWSInstance_userDataBase64 (276.05s)
--- PASS: TestAccAWSInstance_volumeTags (95.11s)
--- PASS: TestAccAWSInstance_volumeTagsComputed (110.97s)
--- PASS: TestAccAWSInstance_vpc (109.50s)
--- PASS: TestAccAWSInstance_withIamInstanceProfile (128.98s)
```

Output from Terraform 0.12 Provider SDK acceptance testing (test failure related to EC2 eventual consistency):

```
--- FAIL: TestAccAWSInstance_associatePublic_defaultPublic (199.26s)
    testing.go:568: Step 0 error: Check failed: Check 2/3 error: aws_instance.foo: Attribute 'associate_public_ip_address' expected "true", got "false"

--- PASS: TestAccAWSInstance_addSecondaryInterface (87.28s)
--- PASS: TestAccAWSInstance_addSecurityGroupNetworkInterface (228.83s)
--- PASS: TestAccAWSInstance_associatePublic_defaultPrivate (72.65s)
--- PASS: TestAccAWSInstance_associatePublic_explicitPrivate (193.33s)
--- PASS: TestAccAWSInstance_associatePublic_explicitPublic (52.31s)
--- PASS: TestAccAWSInstance_associatePublic_overridePrivate (72.60s)
--- PASS: TestAccAWSInstance_associatePublic_overridePublic (71.89s)
--- PASS: TestAccAWSInstance_associatePublicIPAndPrivateIP (218.20s)
--- PASS: TestAccAWSInstance_basic (333.50s)
--- PASS: TestAccAWSInstance_blockDevices (183.67s)
--- PASS: TestAccAWSInstance_changeInstanceType (328.80s)
--- PASS: TestAccAWSInstance_creditSpecification_isNotAppliedToNonBurstable (91.65s)
--- PASS: TestAccAWSInstance_creditSpecification_standardCpuCredits (80.29s)
--- PASS: TestAccAWSInstance_creditSpecification_standardCpuCredits_t2Tot3Taint (373.38s)
--- PASS: TestAccAWSInstance_creditSpecification_unlimitedCpuCredits (89.86s)
--- PASS: TestAccAWSInstance_creditSpecification_unlimitedCpuCredits_t2Tot3Taint (352.61s)
--- PASS: TestAccAWSInstance_creditSpecification_unspecifiedDefaultsToStandard (71.34s)
--- PASS: TestAccAWSInstance_creditSpecification_updateCpuCredits (88.96s)
--- PASS: TestAccAWSInstance_creditSpecificationT3_standardCpuCredits (294.12s)
--- PASS: TestAccAWSInstance_creditSpecificationT3_unlimitedCpuCredits (107.09s)
--- PASS: TestAccAWSInstance_creditSpecificationT3_unspecifiedDefaultsToUnlimited (305.43s)
--- PASS: TestAccAWSInstance_creditSpecificationT3_updateCpuCredits (314.59s)
--- PASS: TestAccAWSInstance_disableApiTermination (186.23s)
--- PASS: TestAccAWSInstance_disappears (239.69s)
--- PASS: TestAccAWSInstance_forceNewAndTagsDrift (277.17s)
--- PASS: TestAccAWSInstance_getPasswordData_falseToTrue (136.94s)
--- PASS: TestAccAWSInstance_getPasswordData_trueToFalse (135.95s)
--- PASS: TestAccAWSInstance_GP2IopsDevice (84.83s)
--- PASS: TestAccAWSInstance_GP2WithIopsValue (77.64s)
--- PASS: TestAccAWSInstance_importBasic (114.84s)
--- PASS: TestAccAWSInstance_importInDefaultVpcBySgId (88.80s)
--- PASS: TestAccAWSInstance_importInDefaultVpcBySgName (85.34s)
--- PASS: TestAccAWSInstance_importInEc2Classic (92.19s)
--- PASS: TestAccAWSInstance_instanceProfileChange (288.48s)
--- PASS: TestAccAWSInstance_ipv6_supportAddressCount (80.62s)
--- PASS: TestAccAWSInstance_ipv6_supportAddressCountWithIpv4 (84.83s)
--- PASS: TestAccAWSInstance_ipv6AddressCountAndSingleAddressCausesError (12.15s)
--- PASS: TestAccAWSInstance_keyPairCheck (185.46s)
--- PASS: TestAccAWSInstance_multipleRegions (253.31s)
--- PASS: TestAccAWSInstance_NetworkInstanceRemovingAllSecurityGroups (87.59s)
--- PASS: TestAccAWSInstance_NetworkInstanceSecurityGroups (189.14s)
--- PASS: TestAccAWSInstance_NetworkInstanceVPCSecurityGroupIDs (186.27s)
--- PASS: TestAccAWSInstance_noAMIEphemeralDevices (57.35s)
--- PASS: TestAccAWSInstance_placementGroup (92.35s)
--- PASS: TestAccAWSInstance_primaryNetworkInterface (54.41s)
--- PASS: TestAccAWSInstance_primaryNetworkInterfaceSourceDestCheck (186.46s)
--- PASS: TestAccAWSInstance_privateIP (71.72s)
--- PASS: TestAccAWSInstance_rootBlockDeviceMismatch (174.52s)
--- PASS: TestAccAWSInstance_rootInstanceStore (71.25s)
--- PASS: TestAccAWSInstance_sourceDestCheck (118.67s)
--- PASS: TestAccAWSInstance_tags (245.43s)
--- PASS: TestAccAWSInstance_UserData_EmptyStringToUnspecified (127.46s)
--- PASS: TestAccAWSInstance_UserData_UnspecifiedToEmptyString (91.24s)
--- PASS: TestAccAWSInstance_userDataBase64 (229.08s)
--- PASS: TestAccAWSInstance_volumeTags (98.22s)
--- PASS: TestAccAWSInstance_volumeTagsComputed (102.21s)
--- PASS: TestAccAWSInstance_vpc (206.67s)
--- PASS: TestAccAWSInstance_withIamInstanceProfile (222.62s)
```
@bflad bflad mentioned this issue Apr 17, 2019
Merged
@bflad bflad mentioned this issue Jun 20, 2019
Merged
@ewbankkit ewbankkit mentioned this issue Nov 10, 2019
Merged
@ewbankkit
Copy link
Contributor

Look also at acceptance tests for:

  • aws_ami_from_instance
  • aws_autoscaling_attachment
  • aws_autoscaling_lifecycle_hook
  • aws_autoscaling_notification
  • aws_autoscaling_schedule
  • aws_eip_association
  • aws_elb_attachment
  • aws_launch_configuration
  • aws_launch_template
  • aws_lb_target_group_attachment
  • aws_network_interface_attachment
  • aws_network_interface
  • aws_route_table
  • aws_route
  • aws_spot_fleet_request
  • aws_spot_instance_request
  • aws_volume_attachment

as there are some with hard-coded AMIs (look for ami or image_id attributes).

This was referenced Dec 3, 2019
Closed
Closed
@ewbankkit
Copy link
Contributor

ewbankkit commented Dec 4, 2019 •
edited

Not all instance types are available in all regions/partitions so there will need to be cleanup of instance_type also.

We can use the new EC2 DescribeInstanceTypeOfferings API to add a PreCheck that the test's instance type is supported.

@PatMyron
Copy link
Contributor

We can use the new EC2 DescribeInstanceTypeOfferings API to add a PreCheck that the test's instance type is supported.

Unfortunately instance type regional availability varies in other AWS services beyond EC2 as well. The CloudFormation Linter gathers instance type regional availability for over a dozen services from the pricing API here

@ewbankkit ewbankkit mentioned this issue Feb 10, 2022
Closed
@ewbankkit ewbankkit mentioned this issue Apr 22, 2022
Closed
@github-actions
Copy link

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

@github-actions github-actions bot added the stale Old or inactive issues managed by automation, if no further action taken these will get closed. label Sep 11, 2022
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 11, 2022
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 11, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@YakDriver YakDriver

Labels
stale Old or inactive issues managed by automation, if no further action taken these will get closed. technical-debt Addresses areas of the codebase that need refactoring or redesign. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[WIP] Region/partition-agnostic aws_instance acceptance tests ewbankkit/terraform-provider-aws
4 participants
@bflad @ewbankkit @PatMyron @YakDriver