Exception thrown with get_url() when returned HTTP response 422

[ldgeorge85]

ISSUE TYPE

• Bug Report

COMPONENT NAME

module_utils/url.py
open_url()

ANSIBLE VERSION

ansible 2.1.1.0

CONFIGURATION

Only changes in ansible.cfg were to:
forks = 11
host_key_checking = False
Other than that, all is completely default.

OS / ENVIRONMENT

CentOS Linux release 7.2.1511 (Core)
Ansible installed using epel repo.

SUMMARY

I am trying to develop a custom module. Part of the module uses the open_url() function from ansible. I was initially using requests module, but I wanted to try to make my ansible module dependent on only ansible itself. Anyway, I have ran into a problem with error handling using open_url(). It seems to go back to the urllib2 code. Basically, when I get back a HTTP response code of 422, it is giving an exception and dying. I have code in my module to catch the response codes and try to fail gracefully, but it does not even seem to reach that point, at least from what the trace shows. When I was using the requests module, this worked fine, the module itself just passed the response back to my code to handle what to do.

I am not sure if there is something I can pass it that would change this or if this is truly a bug. I have tried looking at dev versions of this, but there seems to be a lot of changes, though nothing that mentions this.

Just to note, I am getting no errors and it is passing things back fine when the HTTP response is 201(the expected return).

STEPS TO REPRODUCE

I am using this is a custom ansible module I am building. Here are the lines that call that function and then try to catch the HTTP responses.

    # runs the API call
    result = open_url(url, method="POST", headers=headers, url_username=api_user, url_password=api_key, data=json.dumps(virtual_machine))

    result_json = json.loads(result.read())

    # catches the results
    if result.code == 201:
        return False, True, result_json
    if result.code == 422:
        return True, False, result_json

EXPECTED RESULTS

I expect it to fail gracefully with ansible, dumping the failure meta data, like:

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "meta": {"errors": {"base": [""], "initial_root_password": ["should include letters and digits"], "primary_network_group_id": ["could not be found."]}}, "msg": "Error creating VM"}

ACTUAL RESULTS

Instead, I am getting it dying, with a trace. It does show that there was a 422, but it does not fail gracefully at all and does not return any of the meta data from the failure.

An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_IoGaXE/ansible_module_onapp_vs.py", line 255, in <module>
    main()
  File "/tmp/ansible_IoGaXE/ansible_module_onapp_vs.py", line 246, in main
    is_error, has_changed, result = choice_map.get(module.params['state'])(module.params)
  File "/tmp/ansible_IoGaXE/ansible_module_onapp_vs.py", line 105, in onapp_vs_create
    result = open_url(url, method="POST", headers=headers, url_username=api_user, url_password=api_key, data=json.dumps(virtual_machine))
  File "/tmp/ansible_IoGaXE/ansible_modlib.zip/ansible/module_utils/urls.py", line 839, in open_url
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.7/urllib2.py", line 469, in error
    result = self._call_chain(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 926, in http_error_401
    url, req, headers)
  File "/usr/lib64/python2.7/urllib2.py", line 901, in http_error_auth_reqed
    response = self.retry_http_basic_auth(host, req, realm)
  File "/usr/lib64/python2.7/urllib2.py", line 914, in retry_http_basic_auth
    return self.parent.open(req, timeout=req.timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 422: Unprocessable Entity

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_name": "onapp_vs"}, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_IoGaXE/ansible_module_onapp_vs.py\", line 255, in <module>\n    main()\n  File \"/tmp/ansible_IoGaXE/ansible_module_onapp_vs.py\", line 246, in main\n    is_error, has_changed, result = choice_map.get(module.params['state'])(module.params)\n  File \"/tmp/ansible_IoGaXE/ansible_module_onapp_vs.py\", line 105, in onapp_vs_create\n    result = open_url(url, method=\"POST\", headers=headers, url_username=api_user, url_password=api_key, data=json.dumps(virtual_machine))\n  File \"/tmp/ansible_IoGaXE/ansible_modlib.zip/ansible/module_utils/urls.py\", line 839, in open_url\n  File \"/usr/lib64/python2.7/urllib2.py\", line 154, in urlopen\n    return opener.open(url, data, timeout)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 437, in open\n    response = meth(req, response)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 550, in http_response\n    'http', request, response, code, msg, hdrs)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 469, in error\n    result = self._call_chain(*args)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 409, in _call_chain\n    result = func(*args)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 926, in http_error_401\n    url, req, headers)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 901, in http_error_auth_reqed\n    response = self.retry_http_basic_auth(host, req, realm)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 914, in retry_http_basic_auth\n    return self.parent.open(req, timeout=req.timeout)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 437, in open\n    response = meth(req, response)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 550, in http_response\n    'http', request, response, code, msg, hdrs)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 475, in error\n    return self._call_chain(*args)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 409, in _call_chain\n    result = func(*args)\n  File \"/usr/lib64/python2.7/urllib2.py\", line 558, in http_error_default\n    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)\nurllib2.HTTPError: HTTP Error 422: Unprocessable Entity\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}

[vikas-t]

@ldgeorge85 The urllib2.urlopen() will return the response (result) only if the response code is between 200 to 300. You can check it here https://github.com/python-git/python/blob/master/Lib/urllib2.py
This is the snippet

if not (200 <= code < 300):
            response = self.parent.error(
                'http', request, response, code, msg, hdrs)
return response

[GriffithLea]

@vikas-t, thanks for your comment - it quickly got to the crux of the problem - but then shouldn't open_url() handle the exception from urllib2 and return to the caller? I understand what you said, but it feels like there's more to say. I'd expect you to add either:

• Yeah, this is a bug (maybe this is implied since the issue is still open).
• open_url() is being used incorrectly, here is a pointer to an example of proper usage.

I suppose one might suggest that the caller should handle the exception from urllib2, but that doesn't make sense since presumably the point of ansible.module_utils.urls is to encapsulate what is underneath it.

I'm on Ansible 2.4 and I just ran into this frustrating problem. I didn't have it with requests (when my code was in a standalone script), and I thought I was doing a good thing by going to open_url().

Like requests, curl is similarly friendly:

{
  "status" : 400,
  "reason" : "Bad Request",
  "message" : "resource name 'testdupe' already used"
}

How do I get this same functionality with open_url()? If I do the exact same request as I did in curl or requests that produced the above example result, the wheels come off.

[sivel]

open_url does not handle the exceptions, it is up to the caller to handle the exception. In your onapp_vs.py module, you will need to wrap the open_url call in a try/except block.

Should you want automatic exception handling, please utilize fetch_url instead of open_url

If you have further questions please stop by IRC or the mailing list:

• IRC: #ansible on irc.freenode.net
• mailing list: https://groups.google.com/forum/#!forum/ansible-project

[GriffithLea]

BTW, the example at the top of the fetch_url() def says:

    :returns: A tuple of (**response**, **info**). Use ``response.body()`` to read the data.

Shouldn't that be response.read() ?