Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ec2_elb: times out on de-register #3849

Closed
bhang opened this issue Aug 14, 2013 · 28 comments
Closed

ec2_elb: times out on de-register #3849

bhang opened this issue Aug 14, 2013 · 28 comments
Labels
bug This issue/PR relates to a bug.

Comments

@bhang
Copy link

bhang commented Aug 14, 2013

Ansible version: git checkout of devel - 10a0f03
Module: ec2_elb
Version: 8171b7b

I have a playbook that deals with a common scenario - software deploys to instances behind an ELB. The playbook runs serially and performs the following steps:

  • deregister instance
  • upgrade software
  • register instance

This works well with the ec2_elb module until the commit prior to the one I listed above. With the latest commit, I get an error: operation timed out.

invalid output was:

Traceback (most recent call last):
  File "/Users/shubhang/.ansible/tmp/ansible-1376500488.44-71167606737633/ec2_elb", line 1188, in <module>
    main()
  File "/Users/shubhang/.ansible/tmp/ansible-1376500488.44-71167606737633/ec2_elb", line 227, in main
    response = urllib2.urlopen('http://169.254.169.254/latest/meta-data/placement/availability-zone')
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 394, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 412, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1199, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1174, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 60] Operation timed out>

Thinking that perhaps I had to specify the region (even though it's not required), I added the ec2_region argument but then I got a new error:

msg: ELB [u'<ELB-NAME>'] does not exist

The previous version of this module d644971 works just fine.

@lwade
Copy link
Contributor

lwade commented Aug 14, 2013

Which region does your elb exist in? Does it exist in the same region as
passed to the ec2_region parameter?

On 14 August 2013 18:41, Shubhang Mani notifications@github.com wrote:

Ansible version: git checkout of devel - 10a0f0310a0f03cdc
Module: ec2_elb
Version: 8171b7b 8171b7ba48

I have a playbook that deals with a common scenario - software deploys to
instances behind an ELB. The playbook runs serially and performs the
following steps:

  • deregister instance
  • upgrade software
  • register instance

This works well with the ec2_elb module until the commit prior to the one
I listed above. With the latest commit, I get an error: operation timed out.

invalid output was:

Traceback (most recent call last):
File "/Users/shubhang/.ansible/tmp/ansible-1376500488.44-71167606737633/ec2_elb", line 1188, in
main()
File "/Users/shubhang/.ansible/tmp/ansible-1376500488.44-71167606737633/ec2_elb", line 227, in main
response = urllib2.urlopen('http://169.254.169.254/latest/meta-data/placement/availability-zone')
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 394, in open
response = self._open(req, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 412, in _open
'_open', req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1199, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1174, in do_open
raise URLError(err)urllib2.URLError: <urlopen error [Errno 60] Operation timed out>

Thinking that perhaps I had to specify the region (even though it's not
required), I added the ec2_region argument but then I got a new error:

msg: ELB [u''] does not exist

The previous version of this module d644971d6449711aeworks just fine.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3849
.

@bhang
Copy link
Author

bhang commented Aug 14, 2013

Yes, us-east-1

Looking at the code, I think it might be assuming that the module is being run within EC2 since it tries to query the metadata service if region is not specified. That might be why it's timing out (the first error). Not sure yet why the ELB isn't being found.

@lwade
Copy link
Contributor

lwade commented Aug 14, 2013

Sorry, I missed that. Yes, that's correct. BruceP has removed that in his
latest PR for this, found here:
#3832

On 14 August 2013 19:10, Shubhang Mani notifications@github.com wrote:

Yes, us-east-1

Looking at the code, I think it might be assuming that the module is being
run within EC2 since it tries to query the metadata service if region
is not specified. That might be why it's timing out (the first error). Not
sure yet why the ELB isn't being found.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3849#issuecomment-22655176
.

@lwade
Copy link
Contributor

lwade commented Aug 14, 2013

I should have expanded my last comment a little more. As you have probably
noticed, this part of the code should only be called if ec2_region isn't
set (either via param or env variable on the host). So you should be able
to specify ec2_region to the region in which your ELB exists and then
happily continue in your work :)

On 14 August 2013 19:22, Lester Wade lestertron@gmail.com wrote:

Sorry, I missed that. Yes, that's correct. BruceP has removed that in
his latest PR for this, found here:
#3832

On 14 August 2013 19:10, Shubhang Mani notifications@github.com wrote:

Yes, us-east-1

Looking at the code, I think it might be assuming that the module is
being run within EC2 since it tries to query the metadata service if
region is not specified. That might be why it's timing out (the first
error). Not sure yet why the ELB isn't being found.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3849#issuecomment-22655176
.

@bhang
Copy link
Author

bhang commented Aug 14, 2013

So, I guess I'll temporarily use the older version until the PR makes it in. Should I close this issue ?

@mpdehaan
Copy link
Contributor

Sounds like it needs to stay open but I'll reference it

#3832

@lwade
Copy link
Contributor

lwade commented Aug 14, 2013

Or better still, try out the changes which will soon be merged by grabbing
the module from #3832

On 14 August 2013 21:01, Shubhang Mani notifications@github.com wrote:

So, I guess I'll temporarily use the older version until the PR makes it
in. Should I close this issue ?


Reply to this email directly or view it on GitHubhttps://github.com//issues/3849#issuecomment-22662976
.

@jimi-c
Copy link
Member

jimi-c commented Aug 23, 2013

Can we confirm if this is still an issue? The PR above was merged.

@bhang
Copy link
Author

bhang commented Aug 23, 2013

I pulled the latest from github but I don't see any modifications for this module.

@jimi-c
Copy link
Member

jimi-c commented Aug 23, 2013

Yeah my bad, I thought the above had been merged but it had just been closed. I'll take a look into this today. Can you share an example of your playbook to help replicate this?

@lwade
Copy link
Contributor

lwade commented Aug 23, 2013

Once Bruce's changes get merged, specifying the region will resolve this I
feel.

On 23 August 2013 14:10, James Cammarata notifications@github.com wrote:

Yeah my bad, I thought the above had been merged but it had just been
closed. I'll take a look into this today. Can you share an example of your
playbook to help replicate this?


Reply to this email directly or view it on GitHubhttps://github.com//issues/3849#issuecomment-23162591
.

@bhang
Copy link
Author

bhang commented Aug 23, 2013

@jimi1283 - here's an example of what I'm trying to do: https://gist.github.com/bhang/6320004

@lwade - looking forward to the merge

@jimi-c
Copy link
Member

jimi-c commented Aug 23, 2013

@bhang I've been testing Bruce's patch, and it does appear that this resolves the error, so as soon as that's merged in I'll close this as being fixed. The one gotcha is you'll have to add "region: " to all of your ec2_elb tasks (in fact region is going to be require for most if not all of the ec2* modules going forward).

@bhang
Copy link
Author

bhang commented Aug 23, 2013

@jimi1283 - sounds good, thanks!

@jimi-c
Copy link
Member

jimi-c commented Aug 24, 2013

Merged, give it a test and let me know if you're still having problems.

@bhang
Copy link
Author

bhang commented Aug 26, 2013

So, after pulling the latest, the de-register works fine when I add the region argument, but the register step fails (even after adding the region argument). The error is:

failed: [ec2-xxx-xx-xx-xx.compute-1.amazonaws.com] => {"failed": true}
msg: ELB [u'APP-ELB'] does not exist

FATAL: all hosts have already failed -- aborting

Of course, if I don't supply the region argument to the register step, I get the (expected) error:

msg: Either region or EC2_REGION environment variable must be set.

@jimi-c
Copy link
Member

jimi-c commented Aug 26, 2013

That's odd, it appears the name is being interpreted as a list. Can you look in the library/cloud/ec2_elb code and tell me if you see something like this towards the end of the file?

    for elb in ec2_elbs:
        if not elb_man.exists(elb):
            msg="ELB %s does not exist" % elb
            module.fail_json(msg=msg)

@bhang
Copy link
Author

bhang commented Aug 26, 2013

Yep, I see it. Here's the end of the file on my machine:

    for elb in ec2_elbs:
        if not elb_man.exists(elb):
            msg="ELB %s does not exist" % elb
            module.fail_json(msg=msg)

    if module.params['state'] == 'present':
        elb_man.register(wait)
    elif module.params['state'] == 'absent':
        elb_man.deregister(wait)

    ansible_facts = {'ec2_elbs': [lb.name for lb in elb_man.lbs]}
    ec2_facts_result = dict(changed=elb_man.changed, ansible_facts=ansible_facts)

    module.exit_json(**ec2_facts_result)

# this is magic, see lib/ansible/module_common.py
#<<INCLUDE_ANSIBLE_MODULE_COMMON>>

main()

@jimi-c
Copy link
Member

jimi-c commented Aug 26, 2013

And how are you specifying the ELB's? Is it like this:

      ec2_elbs:
        - testing

Or like this:

      ec2_elbs: testing

If it's the later, could you try the top syntax?

@bhang
Copy link
Author

bhang commented Aug 26, 2013

It's the former i.e.

ec2_elbs:
  - testing

@jimi-c
Copy link
Member

jimi-c commented Aug 27, 2013

That is very odd, that's how I'm testing it and not running into that problem.

@lwade
Copy link
Contributor

lwade commented Aug 27, 2013

Can we see your playbook for registration and deregistration?
On 27 Aug 2013 01:16, "James Cammarata" notifications@github.com wrote:

That is very odd, that's how I'm testing it and not running into that
problem.


Reply to this email directly or view it on GitHubhttps://github.com//issues/3849#issuecomment-23305054
.

@bhang
Copy link
Author

bhang commented Aug 27, 2013

@lwade - I've updated the gist that I shared earlier to include the region arg. This playbook fails at the "Register instance with ELB" task.

https://gist.github.com/bhang/6320004

@jimi-c
Copy link
Member

jimi-c commented Aug 27, 2013

@bhang actually I forgot that you were using a variable for your register step. Could you make sure that variable is an array? I think that may be where the code is getting confused.

@bhang
Copy link
Author

bhang commented Aug 27, 2013

I'm confused. The de-register is supposed to return a fact ec2_elbs which is the list of ELB names (also provided as a list to the de-register). Shouldn't the fact be interpolated as a list when I register? This was working just fine earlier.

@jimi-c
Copy link
Member

jimi-c commented Sep 3, 2013

So it looks like the issue here is that the fact being passed in as an argument results in it being interpreted as a string, despite the fact that it is an array. If you use with_items as follows, it works perfectly, albeit with multiple iterations:

  - name: Deregister instance from ELB
    local_action:
      module: ec2_elb
      ec2_elbs: "{{ item }}"
      instance_id: '{{ ansible_ec2_instance_id }}'
      state: absent
      region: us-east-1
    sudo: false
    with_items: ec2_elbs

This is despite the fact that the module arg is specifically listed as an array, so I'm not quite sure why that's the case here. As a temporary workaround, you can use the syntax above to get this working, and I'll continue debugging.

@jimi-c
Copy link
Member

jimi-c commented Sep 3, 2013

It looks like this is a known issue with the way array facts are interpreted (this is being handled in another issue). Right now, I'd recommend continuing to use the with_items method I posted above, or don't use the fact since you know the names of the LB's you're using in the registration step.

@jimi-c jimi-c closed this as completed Sep 3, 2013
jimi-c added a commit that referenced this issue Sep 4, 2013
@bhang
Copy link
Author

bhang commented Sep 4, 2013

@jimi1283 - Thanks, I'll try the options you suggested.

@ansibot ansibot added bug This issue/PR relates to a bug. and removed bug_report labels Mar 6, 2018
@ansible ansible locked and limited conversation to collaborators Apr 24, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug This issue/PR relates to a bug.
Projects
None yet
Development

No branches or pull requests

5 participants