elb-target - Block/wait fails when target_status is healthy

[nsushain]

SUMMARY

target_status does not block wait for given value for module elb_target

ISSUE TYPE

• Bug Report

COMPONENT NAME

elb_target.py

ANSIBLE VERSION

version 2.6.2

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

    - name: ELB | adding instance back to target group
      elb_target:
        target_group_name: "{{ target_group_name }}"
        region: "{{ aws_region }}"
        target_id: "{{ ansible_ec2_instance_id }}"
        target_status: healthy
        target_port: 3000
        state: present
      register: ec2_target_add
      until: ec2_target_add|success
      retries: 10

EXPECTED RESULTS

Block wait till target_status is healthy, fail if it does not find target_status as healthy beyond 60 seconds

ACTUAL RESULTS

ok: [a.b.c.d] => {
    "attempts": 2, 
    "changed": false, 
    "invocation": {
        "module_args": {
            "aws_access_key": null, 
            "aws_secret_key": null, 
            "deregister_unused": false, 
            "ec2_url": null, 
            "profile": null, 
            "region": "us-east-1", 
            "security_token": null, 
            "state": "present", 
            "target_az": null, 
            "target_group_arn": null, 
            "target_group_name": "<removed>", 
            "target_id": "<removed>", 
            "target_port": 3000, 
            "target_status": "healthy", 
            "target_status_timeout": 60, 
            "validate_certs": true
        }
    }, 
    "target_group_arn": "arn:aws:elasticloadbalancing:us-east-1:<removed>", 
    "target_health_descriptions": {
        "health_check_port": "3000", 
        "target": {
            "id": "<removed>", 
            "port": 3000
        }, 
        "target_health": {
            "description": "Health checks failed with these codes: [404]", 
            "reason": "Target.ResponseCodeMismatch", 
            "state": "unhealthy"
        }
    }
}

[ansibot]

Files identified in the description:

• lib/ansible/modules/cloud/amazon/elb_target.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibot]

cc @wimnat
click here for bot help

[ansibot]

Hi @nsushain,

Thank you for the issue, just so you are aware we have a dedicated Working Group for aws.
You can find other people interested in this in #ansible-aws on Freenode IRC
For more information about communities, meetings and agendas see https://github.com/ansible/community

click here for bot help

[calebsmith]

Hi folks,

Thanks for reporting this issue. I wanted to chime in with some of my findings as I encountered this same problem.

As I encountered it, not only does the module not always wait for the given state as intended, but it can also fail to register a target in a target group as well. I think that both of these issues have the same root cause. Looking at the register_target function in the implementation, this conditional will short-circuit registration as well as the logic for awaiting a healthy state:

https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/elb_target.py#L187

These two conditionals are only both true if the target is not registered with the target group. On the surface, this seems like the correct logic, especially since we only want changed to be set to true when an actual change has occurred. However, this logic does not account for an extra bit of stateful behavior on the part of the load balancer/target group. Namely, the "draining" state.

When a target is deregistered from a target group, it first enters a "draining" state for a period of time. During this time, it will not be given new requests but it will attempt to fulfill existing ones. The time spent in the draining state is configurable on the target group attributes. Once this time is over, the target will leave the target group and will then be "deregistered".

Essentially, I suspect that this module needs to account for this "draining" state and still reregister the target when it is found to be in this state.

If my observation is correct, I would expect a workaround to be that registering a target will only work reliably if it comes after a deregistration triggered by this module, in which the target_status is used to await an 'unused' state. However, I'm too cautious to rely on this for my needs and have chosen to fallback to using awscli commands for now.

Hopefully the above will help diagnose and solve this issue or provide some guidance for others stumbling on this problem.

[ansibot]

cc @jillr @s-hertel @tremble
click here for bot help

[ansibot]

Thank you very much for your interest in Ansible. Ansible has migrated much of the content into separate repositories to allow for more rapid, independent development. We are closing this issue/PR because this content has been moved to one or more collection repositories.

• lib/ansible/modules/cloud/amazon/elb_target.py -> https://galaxy.ansible.com/community/aws

For further information, please see:
https://github.com/ansible/ansibullbot/blob/master/docs/collection_migration.md