ansible: CookieJar / LWPCookieJar class incompatibily while using the uri module. #113

kmARC · 2018-03-07T14:34:55Z

Environment:

$ ansible --version
ansible 2.4.3.0
  config file = /builds/project-0/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.12+ (default, Sep 17 2016, 12:08:02) [GCC 6.2.0 20160914]
$ uname -a
Linux runner--project-0-concurrent-0 4.15.1-041501-generic #201802031831 SMP Sat Feb 3 18:32:13 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

OS: Ubuntu 16:10 (Docker image)

Task:

- name: Get auth token
  uri:
    url: "https://keycloak.example.com/auth/realms/master/protocol/openid-connect/token"
    method: POST
    body: "client_id=admin-cli&username=admin&\
           password={{ keycloak_admin_password }}&grant_type=password"
    return_content: true
    validate_certs: false
  register: r_token
  no_log: false
  run_once: true
  delegate_to: localhost

Example failure

TASK [Get auth token] **********************************************************
task path: /builds/project-0/setup/client.yml:4
The full traceback is:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 130, in run
    res = self._execute()
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 528, in _execute
    result = self._handler.run(task_vars=variables)
  File "/tmp/mitogen-master/ansible_mitogen/mixins.py", line 121, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/action/normal.py", line 45, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars, wrap_async=wrap_async))
  File "/tmp/mitogen-master/ansible_mitogen/mixins.py", line 279, in _execute_module
    env=cast(env),
  File "/tmp/mitogen-master/ansible_mitogen/mixins.py", line 130, in call
    return self._connection.call(func, *args, **kwargs)
  File "/tmp/mitogen-master/ansible_mitogen/connection.py", line 231, in call
    return self.call_async(func, *args, **kwargs).get().unpickle()
  File "/tmp/mitogen-master/mitogen/core.py", line 314, in unpickle
    raise obj
CallError: exceptions.TypeError: unbound method __init__() must be called with CookieJar instance as first argument (got LWPCookieJar instance instead)
  File "<stdin>", line 1368, in _dispatch_calls
  File "<stdin>", line 1363, in _dispatch_one
  File "master:/tmp/mitogen-master/ansible_mitogen/helpers.py", line 128, in run_module
    mod.main()
  File "/usr/local/lib/python2.7/dist-packages/ansible/modules/net_tools/basics/uri.py", line 439, in main
    dict_headers, socket_timeout)
  File "/usr/local/lib/python2.7/dist-packages/ansible/modules/net_tools/basics/uri.py", line 352, in uri
    method=method, timeout=socket_timeout)
  File "/usr/local/lib/python2.7/dist-packages/ansible/module_utils/urls.py", line 1040, in fetch_url
    cookies = cookiejar.LWPCookieJar()
  File "/usr/lib/python2.7/cookielib.py", line 1758, in __init__
    CookieJar.__init__(self, policy)

dw · 2018-03-07T14:39:54Z

Which OS and ansible --version? Thanks

…

On 7 March 2018 at 20:22, Mark Korondi ***@***.***> wrote: Task: - name: Get auth token uri: url: "https://keycloak.example.com/auth/realms/master/protocol/openid-connect/token" method: POST body: "client_id=admin-cli&username=admin&\ password={{ keycloak_admin_password }}&grant_type=password" return_content: true validate_certs: false register: r_token no_log: false run_once: true delegate_to: localhost Example failure TASK [Get auth token] ********************************************************** task path: /builds/project-0/setup/client.yml:4 The full traceback is: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 130, in run res = self._execute() File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 528, in _execute result = self._handler.run(task_vars=variables) File "/tmp/mitogen-master/ansible_mitogen/mixins.py", line 121, in run return super(ActionModuleMixin, self).run(tmp, task_vars) File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/action/normal.py", line 45, in run results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars, wrap_async=wrap_async)) File "/tmp/mitogen-master/ansible_mitogen/mixins.py", line 279, in _execute_module env=cast(env), File "/tmp/mitogen-master/ansible_mitogen/mixins.py", line 130, in call return self._connection.call(func, *args, **kwargs) File "/tmp/mitogen-master/ansible_mitogen/connection.py", line 231, in call return self.call_async(func, *args, **kwargs).get().unpickle() File "/tmp/mitogen-master/mitogen/core.py", line 314, in unpickle raise obj CallError: exceptions.TypeError: unbound method __init__() must be called with CookieJar instance as first argument (got LWPCookieJar instance instead) File "<stdin>", line 1368, in _dispatch_calls File "<stdin>", line 1363, in _dispatch_one File "master:/tmp/mitogen-master/ansible_mitogen/helpers.py", line 128, in run_module mod.main() File "/usr/local/lib/python2.7/dist-packages/ansible/modules/net_tools/basics/uri.py", line 439, in main dict_headers, socket_timeout) File "/usr/local/lib/python2.7/dist-packages/ansible/modules/net_tools/basics/uri.py", line 352, in uri method=method, timeout=socket_timeout) File "/usr/local/lib/python2.7/dist-packages/ansible/module_utils/urls.py", line 1040, in fetch_url cookies = cookiejar.LWPCookieJar() File "/usr/lib/python2.7/cookielib.py", line 1758, in __init__ CookieJar.__init__(self, policy) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#113>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAJC4xb0sKeRF6B6PRZYv5e48PU-IfYks5tb_AggaJpZM4SgjYK> .

dw · 2018-03-07T14:45:43Z

Looks like some trickery involving six.py in module_utils/urls.py. Thanks for the report, will look at this tomorrow

kmARC · 2018-03-07T14:47:04Z

I just updated the original report. Thanks!

mlaurense · 2018-03-07T15:42:37Z

I can confirm the same issue when using apt_key:

fatal: [production999.local]: FAILED! => {"changed": false, "msg": "error getting key id from url: https://deb.goaccess.io/gnugpg.key", "traceback": "Traceback (most recent call last):\n File \"/usr/lib/python2.7/dist-packages/ansible/modules/packaging/os/apt_key.py\", line 205, in download_key\n rsp, info = fetch_url(module, url)\n File \"/usr/lib/python2.7/dist-packages/ansible/module_utils/urls.py\", line 1040, in fetch_url\n cookies = cookiejar.LWPCookieJar()\n File \"/usr/lib/python2.7/cookielib.py\", line 1758, in __init__\n CookieJar.__init__(self, policy)\nTypeError: unbound method __init__() must be called with CookieJar instance as first argument (got LWPCookieJar instance instead)\n"}

This appears when running ansible 2.4.3.0 / python 2.7.10 on OSX to a Ubuntu 16.04 machine.

dw · 2018-03-07T15:46:28Z

Thanks for letting me know! Will treat this with priority, it's a bug affecting a helper module, so many more modules are likely impacted.

…

On 7 March 2018 at 21:27, mlaurense ***@***.***> wrote: I can confirm the same issue when using apt_key: fatal: [production999.local]: FAILED! => {"changed": false, "msg": "error getting key id from url: https://deb.goaccess.io/gnugpg.key", "traceback": "Traceback (most recent call last):\n File \"/usr/lib/python2.7/dist-packages/ansible/modules/packaging/os/apt_key.py\", line 205, in download_key\n rsp, info = fetch_url(module, url)\n File \"/usr/lib/python2.7/dist-packages/ansible/module_utils/urls.py\", line 1040, in fetch_url\n cookies = cookiejar.LWPCookieJar()\n File \"/usr/lib/python2.7/cookielib.py\", line 1758, in __init__\n CookieJar.__init__(self, policy)\nTypeError: unbound method __init__() must be called with CookieJar instance as first argument (got LWPCookieJar instance instead)\n"} This appears when running ansible 2.4.3.0 / python 2.7.10 on OSX to a Ubuntu 16.04 machine. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#113 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAJC6Q8k-6VfpNTv2CHorqpHg1pNY0Xks5tb__6gaJpZM4SgjYK> .

dw · 2018-03-08T00:39:30Z

What a fantastic catch! Thanks for reporting this, it was a huge bug that's gone unnoticed for a very long time :)

This actually addresses multiple problems: * Single-file programs were broken, since the fix introduced in 6931cc1 caused builtin_find_module() to start indicating __main__ can always be loaded locally. That's broken, and there might be more cases where the same problem will crop up. Since it was indicated __main__ could be loaded locally, the built-in import machinery was allowed to attempt that (since we remove __main__ from sys.modules during bootstrap), which caused a safety check to fire in the bowels of Python: "Cannot re-init internal module %.200s" * The check for presence of the whitelist was totally broken, since the whitelist is never an empty list. Therefore 'self' was being returned for every module, including extension modules like 'termios'. I have hand-verified this does not break the fix for issue #113. I looked at writing a test for that, but it requires a Docker container (or similar) with an ancient version of Ansible installed. Will open a separate ticket tracking this.

Amazed this one managed to scrape through for so long. Calling __import__ from within find_module() was causing the target module, in this case cookielib, to be loaded *then overwritten* by a subsequent duplicate load higher in the stack. The result is that cookielib was loaded twice, and, per usual Python import semantics, a reference to the partially initialized first cookielib was installed in sys.modules while its code executed. At the end of cookielib on 2.x, it imports _LWPCookieJar, which in turn imports the partially built cookielib from sys.modules, then subclasses the CookieJar from /that/ module. Everything is wonderful. Then the call returns back up into the import mechanism which restarts the entire process -- only this time, _LWPCookieJar is /not/ reinitialized, so the copy in sys.modules is still left with types pointing at the old module! So the duplicate import creates a new CookieJar which is not the base class of LWPCookieJar. Tada! 3 hours debugging. This is probably a performance fix in disguise, didn't realize things were so broken. It may also be a regression elsewhere. Urgently need to finish the tests.

This actually addresses multiple problems: * Single-file programs were broken, since the fix introduced in 6931cc1 caused builtin_find_module() to start indicating __main__ can always be loaded locally. That's broken, and there might be more cases where the same problem will crop up. Since it was indicated __main__ could be loaded locally, the built-in import machinery was allowed to attempt that (since we remove __main__ from sys.modules during bootstrap), which caused a safety check to fire in the bowels of Python: "Cannot re-init internal module %.200s" * The check for presence of the whitelist was totally broken, since the whitelist is never an empty list. Therefore 'self' was being returned for every module, including extension modules like 'termios'. I have hand-verified this does not break the fix for issue #113. I looked at writing a test for that, but it requires a Docker container (or similar) with an ancient version of Ansible installed. Will open a separate ticket tracking this.

add received dtmf instead of NULL

dw added a commit that referenced this issue Mar 7, 2018

issue #113: import crash reproduction

1a2a5b0

dw added bug Code feature that hinders desired execution outcome ansible Issues relating to Mitogen for Ansible labels Mar 7, 2018

dw closed this as completed in 6931cc1 Mar 8, 2018

dw mentioned this issue Mar 12, 2018

ImportError: Master does not have 'termios' #121

Closed

dw added the NeedsTest Label for in-progress and closed bugs that do not yet have a corresponding regression test label Mar 15, 2018

dw added a commit that referenced this issue Mar 19, 2018

issue #113: import crash reproduction

57b9d59

PatrickCoakley23 pushed a commit to cyara/mitogen that referenced this issue Nov 10, 2023

Merge pull request mitogen-hq#113 from spearlineltd/TT55_dialplan_change

c803348

add received dtmf instead of NULL

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ansible: CookieJar / LWPCookieJar class incompatibily while using the uri module. #113

ansible: CookieJar / LWPCookieJar class incompatibily while using the uri module. #113

kmARC commented Mar 7, 2018 •

edited

dw commented Mar 7, 2018 via email

dw commented Mar 7, 2018

kmARC commented Mar 7, 2018

mlaurense commented Mar 7, 2018

dw commented Mar 7, 2018 via email

dw commented Mar 8, 2018

ansible: CookieJar / LWPCookieJar class incompatibily while using the uri module. #113

ansible: CookieJar / LWPCookieJar class incompatibily while using the uri module. #113

Comments

kmARC commented Mar 7, 2018 • edited

Environment:

Task:

Example failure

dw commented Mar 7, 2018 via email

dw commented Mar 7, 2018

kmARC commented Mar 7, 2018

mlaurense commented Mar 7, 2018

dw commented Mar 7, 2018 via email

dw commented Mar 8, 2018

kmARC commented Mar 7, 2018 •

edited