ec2_instance: wait on instance state, rather than status check #481

tremble · 2021-08-28T19:09:24Z

Attempt to revive ansible-collections/community.aws#461

SUMMARY

The await_instances() function was calling module.client('ec2').get_waiter('instance_status_ok') by default, which causes boto3 to wait on the DescribeInstanceStatus api to return "Ok". The status API is linked to EC2's instance status checks, which lag the instance's real state by 3-5 minutes.

It's unlikely that most Ansible users who set wait: true want their playbooks to delay until EC2's status checks catch up. The more efficient approach is to use wait_for or wait_for_connection. What is useful, however, is to wait the short period of time necessary for EC2 to move an instance from pending to running.

Before this change, the ec2_instance module would wait for an "Ok" status check for newly-created instances before waiting for any other state change. This change makes the default state: present wait for instance_exists, and state: running wait for instance_running. state: started (previously just an alias for state: running) now will wait for instance_status_ok if wait: true.

The only, IMHO unlikely, scenario that this is a breaking change for is if an Ansible user had combined state: present/running/started and wait: true and is actually depending on the play to halt until EC2's status checks return OK. Based on my own informal survey of playbooks on GitHub, I did not see anyone using ec2_instance in this way. Most playbooks follow ec2_instance with wait_for or wait_for_connection.

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

ec2_instance

ADDITIONAL INFORMATION

ansibullbot · 2021-08-28T19:20:56Z

cc @jillr @ryansb @s-hertel @wimnat
click here for bot help

overhacked · 2021-09-03T02:07:17Z

Hello, again, everyone! It appears that there are no blockers for this PR other than waiting on a major release because of the potential (but unlikely) breaking change. See ansible-collections/community.aws#461 (comment).

@markuman also asked if this PR might address #469. I think that changing ec2 instance type is potentially fraught with enough testing issues that I'd rather tackle it in a separate PR. If there's agreement that this PR is ready to merge, then I could stack another PR on it to address the changing instance type. Thanks @tremble, for lighting a fire under me by reopening this. That's great maintainership!

tremble · 2021-09-03T08:52:21Z

Hi @overhacked,

when I checked the integration tests weren't passing. I need to dig into why but haven't had much free time this week.

The `await_instances()` function was calling `module.client('ec2').get_waiter('instance_status_ok')` by default, which causes boto3 to wait on the `DescribeInstanceStatus` api to return "Ok". The status API is linked to [EC2's instance status checks](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html#reporting_status), which lag the instance's real state by 3-5 minutes. It's unlikely that most Ansible users who set `wait: true` want their playbooks to delay until EC2's status checks catch up. The more efficient approach is to use `wait_for` or `wait_for_connection`. What is useful, however, is to wait the short period of time necessary for EC2 to move an instance from `pending` to `running`. Before this change, the `ec2_instance` module would wait for an "Ok" status check for newly-created instances *before* waiting for any other state change. This change makes the default `state: present` wait for `instance_exists`, and `state: running` wait for `instance_running`. `state: started` (previously just an alias for `state: running`) now will wait for `instance_status_ok` if `wait: true`. The only, IMHO unlikely, scenario that this is a breaking change for is if an Ansible user had combined `state: present/running/started` and `wait: true` and is actually depending on the play to halt until EC2's status checks return OK. Based on my own informal survey of playbooks on GitHub, I did not see anyone using `ec2_instance` in this way. Most playbooks follow `ec2_instance` with `wait_for` or `wait_for_connection`.

In some cases, we need to branch based on desired EC2 instance state. In others, on desired module state. Some CI checks were failing due to not checking the right value. Renamed and/or assigned local variables in a few critical functions here to be clear on which state we are referring to.

Added a `force_wait` parameter to `await_instances()` Using it to await `instance_exists` when creating new instances before modifying them, because `diff_instance_and_params()` can fail if it runs too soon before the EC2 API is consistent.

…o longer guarantees running)

…nstance to exist rather than start

alinabuzachis · 2021-09-06T15:47:14Z

recheck

tremble · 2021-09-06T17:39:05Z

recheck

jillr

LGTM with the changelog nit. This is a really nice improvement, thanks @tremble!

changelogs/fragments/481-ec2_instance-wait_sanity.yml

tremble · 2021-09-07T19:42:26Z

Thanks for the initial work on this @overhacked. If CI plays ball, hopefully we'll be able to get this in place for the 2.0.0 release (RealSoonNow...)

tremble · 2021-09-08T07:15:27Z

recheck

ansible-zuul

LGTM!

Wait for instance to start before attempting to attach a volume SUMMARY Following #481 ec2_vol tests failed because the instances hadn't finished starting before trying to attach botocore.exceptions.ClientError: An error occurred (IncorrectState) when calling the AttachVolume operation: Instance 'i-0d740b81c23f11a4c' is not 'running'. ISSUE TYPE Bugfix Pull Request COMPONENT NAME ec2_vol ADDITIONAL INFORMATION https://7860a9269d0955476c03-81788054a9528b940bcdbd064c8a3746.ssl.cf1.rackcdn.com/488/b02aebad5940f5500e4ccb93c093bc00339b2426/check/ansible-test-cloud-integration-aws-py36_1/110bed5/job-output.txt Reviewed-by: None <None>

ec2_asg: Add purge_tags to AutoScalingGroups. SUMMARY Add purge_tags to ec2_asg module. Fixes ansible-collections#481. ISSUE TYPE Feature Pull Request COMPONENT NAME ec2_asg ADDITIONAL INFORMATION There was another PR (currently closed) ansible-collections#482 - with similar functionality but I'm not sure if having modules to handle tags for individual services/modules is a good way to have this functionality. It will certainly cause increase in number of modules. Hence tried modifying existing ec2_asg module to be able to do this. This utilizes underlying API calls to: https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_DescribeTags.html https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_CreateOrUpdateTags.html https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_DeleteTags.html Reviewed-by: Alina Buzachis <None> Reviewed-by: Jill R <None> Reviewed-by: Mandar Kulkarni <mandar242@gmail.com>

tremble force-pushed the ec2_instance/state_present branch from 50f7e36 to 557c840 Compare August 28, 2021 19:10

tremble mentioned this pull request Aug 28, 2021

inventory_aws_ec2/test: switch to ec2_instance #480

Merged

ansibullbot added WIP Work in progress bug This issue/PR relates to a bug module module needs_triage owner_pr PR created by owner/maintainer plugins plugin (any type) labels Aug 28, 2021

goneri removed the needs_triage label Aug 31, 2021

ansibullbot added integration tests/integration tests tests and removed owner_pr PR created by owner/maintainer labels Sep 3, 2021

overhacked and others added 6 commits September 3, 2021 11:04

ec2_instance: await new instance existence

29357ac

Added a `force_wait` parameter to `await_instances()` Using it to await `instance_exists` when creating new instances before modifying them, because `diff_instance_and_params()` can fail if it runs too soon before the EC2 API is consistent.

ec2_instance: add changelog fragment for ansible-collections#461

33ec0f2

Re-enable ec2_instance tests

a3caf23

Update ec2_instance tests to use new shared code

41e66f1

tremble force-pushed the ec2_instance/state_present branch from b680792 to cf206df Compare September 4, 2021 17:02

tremble added 6 commits September 4, 2021 19:26

Update ec2_instance block dev tests to allow for 'pending' (present n…

da5a877

…o longer guarantees running)

Minor fixups

bb1fae9

Minor test tweaks

6294332

Don't create IGW/Route tables, we don't need them

6191164

Make ec2 client a global and rename to 'client'

923b80c

Move filter building into a dedicated function

dc537ce

tremble force-pushed the ec2_instance/state_present branch from cf206df to 3cda932 Compare September 4, 2021 17:41

tremble changed the title ~~[WIP] ec2_instance: wait on instance state, rather than status check~~ ec2_instance: wait on instance state, rather than status check Sep 4, 2021

tremble added 2 commits September 5, 2021 08:22

Rework existing vs present logic

79a0d75

Update some tests to account for 'present' now only waiting for the i…

1e45fd8

…nstance to exist rather than start

tremble force-pushed the ec2_instance/state_present branch from 3cda932 to 1e45fd8 Compare September 5, 2021 06:22

jillr approved these changes Sep 7, 2021

View reviewed changes

changelogs/fragments/481-ec2_instance-wait_sanity.yml Outdated Show resolved Hide resolved

changelog tweak from review

28ebe00

This comment has been minimized.

Sign in to view

alinabuzachis approved these changes Sep 8, 2021

View reviewed changes

tremble added the gate label Sep 8, 2021

ansible-zuul bot approved these changes Sep 8, 2021

View reviewed changes

tremble added gate and removed gate labels Sep 8, 2021

ansible-zuul bot merged commit ad7a0eb into ansible-collections:main Sep 8, 2021

tremble mentioned this pull request Sep 11, 2021

Wait for instance to start before attempting to attach a volume #497

Merged

tremble mentioned this pull request Sep 27, 2021

ec2_instance waits until timeout when creating new EC2 instance ansible-collections/community.aws#736

Closed

1 task

tremble deleted the ec2_instance/state_present branch November 26, 2021 12:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ec2_instance: wait on instance state, rather than status check #481

ec2_instance: wait on instance state, rather than status check #481

tremble commented Aug 28, 2021 •

edited

Loading

ansibullbot commented Aug 28, 2021

overhacked commented Sep 3, 2021

tremble commented Sep 3, 2021

alinabuzachis commented Sep 6, 2021

tremble commented Sep 6, 2021

jillr left a comment

tremble commented Sep 7, 2021

This comment has been minimized.

This comment has been minimized.

tremble commented Sep 8, 2021

ansible-zuul bot left a comment

ec2_instance: wait on instance state, rather than status check #481

ec2_instance: wait on instance state, rather than status check #481

Conversation

tremble commented Aug 28, 2021 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

ansibullbot commented Aug 28, 2021

overhacked commented Sep 3, 2021

tremble commented Sep 3, 2021

alinabuzachis commented Sep 6, 2021

tremble commented Sep 6, 2021

jillr left a comment

Choose a reason for hiding this comment

tremble commented Sep 7, 2021

This comment has been minimized.

This comment has been minimized.

tremble commented Sep 8, 2021

ansible-zuul bot left a comment

Choose a reason for hiding this comment

tremble commented Aug 28, 2021 •

edited

Loading