New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ansible: CookieJar / LWPCookieJar class incompatibily while using the uri module. #113

Closed
kmARC opened this Issue Mar 7, 2018 · 6 comments

Comments

Projects
None yet
3 participants
@kmARC
Copy link

kmARC commented Mar 7, 2018

Environment:

$ ansible --version
ansible 2.4.3.0
  config file = /builds/project-0/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.12+ (default, Sep 17 2016, 12:08:02) [GCC 6.2.0 20160914]
$ uname -a
Linux runner--project-0-concurrent-0 4.15.1-041501-generic #201802031831 SMP Sat Feb 3 18:32:13 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

OS: Ubuntu 16:10 (Docker image)

Task:

- name: Get auth token
  uri:
    url: "https://keycloak.example.com/auth/realms/master/protocol/openid-connect/token"
    method: POST
    body: "client_id=admin-cli&username=admin&\
           password={{ keycloak_admin_password }}&grant_type=password"
    return_content: true
    validate_certs: false
  register: r_token
  no_log: false
  run_once: true
  delegate_to: localhost

Example failure

TASK [Get auth token] **********************************************************
task path: /builds/project-0/setup/client.yml:4
The full traceback is:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 130, in run
    res = self._execute()
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 528, in _execute
    result = self._handler.run(task_vars=variables)
  File "/tmp/mitogen-master/ansible_mitogen/mixins.py", line 121, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/action/normal.py", line 45, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars, wrap_async=wrap_async))
  File "/tmp/mitogen-master/ansible_mitogen/mixins.py", line 279, in _execute_module
    env=cast(env),
  File "/tmp/mitogen-master/ansible_mitogen/mixins.py", line 130, in call
    return self._connection.call(func, *args, **kwargs)
  File "/tmp/mitogen-master/ansible_mitogen/connection.py", line 231, in call
    return self.call_async(func, *args, **kwargs).get().unpickle()
  File "/tmp/mitogen-master/mitogen/core.py", line 314, in unpickle
    raise obj
CallError: exceptions.TypeError: unbound method __init__() must be called with CookieJar instance as first argument (got LWPCookieJar instance instead)
  File "<stdin>", line 1368, in _dispatch_calls
  File "<stdin>", line 1363, in _dispatch_one
  File "master:/tmp/mitogen-master/ansible_mitogen/helpers.py", line 128, in run_module
    mod.main()
  File "/usr/local/lib/python2.7/dist-packages/ansible/modules/net_tools/basics/uri.py", line 439, in main
    dict_headers, socket_timeout)
  File "/usr/local/lib/python2.7/dist-packages/ansible/modules/net_tools/basics/uri.py", line 352, in uri
    method=method, timeout=socket_timeout)
  File "/usr/local/lib/python2.7/dist-packages/ansible/module_utils/urls.py", line 1040, in fetch_url
    cookies = cookiejar.LWPCookieJar()
  File "/usr/lib/python2.7/cookielib.py", line 1758, in __init__
    CookieJar.__init__(self, policy)
@dw

This comment has been minimized.

Copy link
Owner

dw commented Mar 7, 2018

dw added a commit that referenced this issue Mar 7, 2018

@dw

This comment has been minimized.

Copy link
Owner

dw commented Mar 7, 2018

Looks like some trickery involving six.py in module_utils/urls.py. Thanks for the report, will look at this tomorrow

@dw dw added bug ansible labels Mar 7, 2018

@kmARC

This comment has been minimized.

Copy link
Author

kmARC commented Mar 7, 2018

I just updated the original report. Thanks!

@mlaurense

This comment has been minimized.

Copy link

mlaurense commented Mar 7, 2018

I can confirm the same issue when using apt_key:

fatal: [production999.local]: FAILED! => {"changed": false, "msg": "error getting key id from url: https://deb.goaccess.io/gnugpg.key", "traceback": "Traceback (most recent call last):\n File \"/usr/lib/python2.7/dist-packages/ansible/modules/packaging/os/apt_key.py\", line 205, in download_key\n rsp, info = fetch_url(module, url)\n File \"/usr/lib/python2.7/dist-packages/ansible/module_utils/urls.py\", line 1040, in fetch_url\n cookies = cookiejar.LWPCookieJar()\n File \"/usr/lib/python2.7/cookielib.py\", line 1758, in __init__\n CookieJar.__init__(self, policy)\nTypeError: unbound method __init__() must be called with CookieJar instance as first argument (got LWPCookieJar instance instead)\n"}

This appears when running ansible 2.4.3.0 / python 2.7.10 on OSX to a Ubuntu 16.04 machine.

@dw

This comment has been minimized.

Copy link
Owner

dw commented Mar 7, 2018

@dw dw closed this in 6931cc1 Mar 8, 2018

@dw

This comment has been minimized.

Copy link
Owner

dw commented Mar 8, 2018

What a fantastic catch! Thanks for reporting this, it was a huge bug that's gone unnoticed for a very long time :)

dw added a commit that referenced this issue Mar 12, 2018

issue #121: fix breakage caused by a9c6c13
This actually addresses multiple problems:

* Single-file programs were broken, since the fix introduced in
  6931cc1 caused builtin_find_module()
  to start indicating __main__ can always be loaded locally. That's
  broken, and there might be more cases where the same problem will crop
  up.

  Since it was indicated __main__ could be loaded locally, the built-in
  import machinery was allowed to attempt that (since we remove __main__
  from sys.modules during bootstrap), which caused a safety check to
  fire in the bowels of Python:

      "Cannot re-init internal module %.200s"

* The check for presence of the whitelist was totally broken, since the
  whitelist is never an empty list. Therefore 'self' was being returned
  for every module, including extension modules like 'termios'.

I have hand-verified this does not break the fix for issue #113. I
looked at writing a test for that, but it requires a Docker container
(or similar) with an ancient version of Ansible installed. Will open a
separate ticket tracking this.

@dw dw added the NeedsTest label Mar 15, 2018

dw added a commit that referenced this issue Mar 19, 2018

dw added a commit that referenced this issue Mar 19, 2018

importer: avoid duplicate module load(!); closes #113.
Amazed this one managed to scrape through for so long. Calling
__import__ from within find_module() was causing the target module, in
this case cookielib, to be loaded *then overwritten* by a subsequent
duplicate load higher in the stack.

The result is that cookielib was loaded twice, and, per usual Python
import semantics, a reference to the partially initialized first
cookielib was installed in sys.modules while its code executed.

At the end of cookielib on 2.x, it imports _LWPCookieJar, which in turn
imports the partially built cookielib from sys.modules, then subclasses
the CookieJar from /that/ module.

Everything is wonderful. Then the call returns back up into the import
mechanism which restarts the entire process -- only this time,
_LWPCookieJar is /not/ reinitialized, so the copy in sys.modules is
still left with types pointing at the old module!

So the duplicate import creates a new CookieJar which is not the base
class of LWPCookieJar. Tada! 3 hours debugging.

This is probably a performance fix in disguise, didn't realize things
were so broken. It may also be a regression elsewhere. Urgently need to
finish the tests.

dw added a commit that referenced this issue Mar 19, 2018

issue #121: fix breakage caused by a9c6c13
This actually addresses multiple problems:

* Single-file programs were broken, since the fix introduced in
  6931cc1 caused builtin_find_module()
  to start indicating __main__ can always be loaded locally. That's
  broken, and there might be more cases where the same problem will crop
  up.

  Since it was indicated __main__ could be loaded locally, the built-in
  import machinery was allowed to attempt that (since we remove __main__
  from sys.modules during bootstrap), which caused a safety check to
  fire in the bowels of Python:

      "Cannot re-init internal module %.200s"

* The check for presence of the whitelist was totally broken, since the
  whitelist is never an empty list. Therefore 'self' was being returned
  for every module, including extension modules like 'termios'.

I have hand-verified this does not break the fix for issue #113. I
looked at writing a test for that, but it requires a Docker container
(or similar) with an ancient version of Ansible installed. Will open a
separate ticket tracking this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment