ImportError: No module named lxml on OS X

[ssbarnea]

Please enable OS X tests on travis because it seems kinda impossible to make the xml module to work on it, even when this module is installed and I am running the task against localhost.

Have a look here on how to enable it, is two lines of code https://github.com/pycontribs/tendo/blob/master/.travis.yml

[cmprescott]

I'll enable os x build this weekend. In the meantime, what Ansible version are using? Did you install lxml with pip install lxml?

EDIT: PS -- no go on enabling os x build. travis-ci/travis-ci#4729 & https://travis-ci.org/cmprescott/ansible-xml/builds/117309833

[ssbarnea]

INFO:   === Version Report ===
Python 2.7.10
ansible (2.0.1.0)
lxml (3.6.0)
pip (8.1.1)

But there is something strange, executing the tests works on my machine but executing another very simple playbook fails complaining about the missing lxml.

- hosts: localhost
  tasks:
    - debug: msg="Running a set of ansible operations in order to validate that your have a proper setup. This tests only localhost so don't be surprised if same actions fail on another machine."

    - xml:
        file=data/books.xml
        xpath=note/from
        value=ansible

Error

ASK [xml] *********************************************************************
task path: /Users/sorins/bmll/devops/tests/test.yml:5
ESTABLISH LOCAL CONNECTION FOR USER: sorins
localhost EXEC /bin/sh -c '( umask 22 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1458318526.62-237035020845880 `" && echo "` echo $HOME/.ansible/tmp/ansible-tmp-1458318526.62-237035020845880 `" )'
localhost PUT /var/folders/5s/866_p9c16k59ygvyqnkmpv000000gn/T/tmp1u_Wf2 TO /Users/sorins/.ansible/tmp/ansible-tmp-1458318526.62-237035020845880/xml
localhost EXEC /bin/sh -c 'LANG=en_GB.UTF-8 LC_ALL=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8 /usr/bin/python /Users/sorins/.ansible/tmp/ansible-tmp-1458318526.62-237035020845880/xml; rm -rf "/Users/sorins/.ansible/tmp/ansible-tmp-1458318526.62-237035020845880/" > /dev/null 2>&1'
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/Users/sorins/.ansible/tmp/ansible-tmp-1458318526.62-237035020845880/xml", line 109, in <module>
    from lxml import etree
ImportError: No module named lxml

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_name": "xml"}, "parsed": false}

[tbielawa]

Osx can have weeeird library logic sometimes. Because you said one method works, but a full test doesn't, this may be related to how the specific python binary is called.

This may be an edge case if @ssbarnea has an unusual python package/library configuration. I know on my osx box i get issues because i have several different python versions installed.

Id double check the module writing docs for any specific notes apropos the she-bang. This module explicitly says #!/usr/bin/python, that may be ill advised.

To the OP, if you run /usr/bin/python and then try to import lxml? Maybe your ansible-playbook script is calling a different python binary. If so, try the import test again with THAT python binary

head which ansible-playbook``

[ssbarnea]

Here is the issue with OS X, because is too dangerous to do anything with the OS provided python, people are installing a local copy using brew, which resides in /usr/local/bin/python2. More than 9/10 people that use the command line on linux are doing this. On this machines running which python will return /usr/local/bin/python.

Assuming a hardcoded path for python interpreter is clearly a big mistake, the correct approach is to point to the default one using one of these shebang lines:

#!/bin/env python
#!/bin/env python2
#!/bin/env python3

Please note that the first one should be used only by software that works on both pyhon2 and python3, ansible is supposed to use #!/bin/env python2 in order to assure that is using the py2 even on machines that have python3 as the default python interpreter.

[ssbarnea]

Here is a funny fact, it seems that on my machine

$ head `which ansible-playbook`
#!/usr/local/opt/python/bin/python2.7

This tells me that ansible installer makes a decision to point there. Good part is that's a symlink to brew version of python, one identical with the one inside /usr/local/bin/python2.7.

So which was the module that had the wrong path in the shebang line?

$ /usr/bin/python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import lxml
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named lxml

Yes, the native python does not have lxml and it should never get it, the problem is not the python itself, the problem is that ansible ends up trying to use that one instead of default one (env/which approach)

[tbielawa]

@ssbarnea thank you for confirming that using /usr/bin/python you get an error trying to import lxml. @ssbarnea I have a request for you. Please try this again using the path to the python binary that you found in ansible-playbook.

$ /usr/local/opt/python/bin/python2.7
>>> import lxml

If that succeeds, then see what happens if you do this:

$ export ansible_python_interpreter=/usr/local/opt/python/bin/python2.7
$ ansible-playbook -i .... foo/your_playbook.yaml

If those fail, I advise you attempt to install lxml using the package manager that you installed ansible with. This reference may be useful

• http://stackoverflow.com/questions/1277124/how-do-you-install-lxml-on-os-x-leopard-without-using-macports-or-fink

Here's some other information I was able to figure out which may be helpful (or just good to know):

I just cloned the modules-core and modules-extras repos and did a quick scan of the shebangs. BY FAR #!/usr/bin/python is the preferred shebang for Ansible modules.

$ find . -name '*.py' | xargs head -n2 | grep python | sort | uniq -c | sort -n
      2 #!/usr/bin/env python
      4 # -*- mode: python -*-
     12 #!/usr/bin/python -tt
    490 #!/usr/bin/python

However, note the following quote from the docs http://docs.ansible.com/ansible/faq.html#how-do-i-handle-python-pathing-not-having-a-python-2-x-in-usr-bin-python-on-a-remote-machine

Do not replace the shebang lines of your python modules. Ansible will do this
for you automatically at deploy time.

[ssbarnea]

I was able to bypass the problem by adding this to the playbook:

  vars:
    ansible_python_interpreter: python2

As you can guess, this will tell ansible to use python2 executable from PATH, which will solve the missing lxml issue. It seems to be working fine on OS X and Linux too.

I am sure that this is not a problem of the ansible-xml module itself, but it seems that this one is affected the most. Imagine that without this hack even including a pip install lxml on the target machine will not solve the problem, because the pip install will eventually install lxml on another interpreter.

My impression is that's an ansible deployment decision issue, the decision of replacing the shebang with /usr/bin/python instead of #!/usr/bin/env python even if I would argue that probably the best would be one like #!/usr/bin/env python2.

Now, I think that it would be in our interest to resolve this bad user experience, so people would be able to use xml module without having to spend hours wondering why it does not work on some machines.

[cmprescott]

Closing as won't fix. However, keep in mind I have a rainy day project to create an ansible-role-lxml. Hopefully, that solves a lot of the headaches associated with installing the dependency on the controller.

[ssbarnea]

I am ok with this, meanwhile I changed jobs and I did not need this anymore.

I think that ideally lxml should become optional and ansible-xml should use a pure python xml library (maybe with fallbacks) in order to avoid all these issues. I know that lxml is performant but I put more value on portability than speed, especially when it comes to Ansible.

Still, I am fully aware that I will need XML support again and I would like to see XML as part of ansible core distribution. If you also support the idea you should vote/comment on ansible/ansible-modules-core#4112

I am not sure if this could be of you to you but at least in one case I was able to remove the need of lxml by replacing it with defusedxml.

BTW, Support for Python 3 in Ansible is almost here, the latest version has experiment support for it.