v1.1.1 gevent/ares.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

[woozyking]

Note: I think this is a bit different from #549

After updating from version 1.1.0 to 1.1.1, under the same environment:

Python (virtualenv from OS Python 2.7):

$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2

OS:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:    14.04
Codename:   trusty

It gives me this error:

Traceback (most recent call last):
  File "/home/overlord/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 515, in spawn_worker
    worker.init_process()
  File "/home/overlord/venv/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 198, in init_process
    hub.reinit()
  File "/home/overlord/venv/local/lib/python2.7/site-packages/gevent/hub.py", line 353, in reinit
    if hasattr(hub.resolver, '_on_fork'):
  File "/home/overlord/venv/local/lib/python2.7/site-packages/gevent/hub.py", line 728, in _get_resolver
    self.resolver_class = _import(self.resolver_class)
  File "/home/overlord/venv/local/lib/python2.7/site-packages/gevent/hub.py", line 412, in _import
    return _import(path[-1])
  File "/home/overlord/venv/local/lib/python2.7/site-packages/gevent/hub.py", line 424, in _import
    x = __import__(module)
  File "/home/overlord/venv/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "/home/overlord/venv/local/lib/python2.7/site-packages/gevent/resolver_ares.py", line 11, in <module>
    from gevent.ares import channel, InvalidIP
  File "/home/overlord/venv/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
ImportError: /home/overlord/venv/local/lib/python2.7/site-packages/gevent/ares.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

We use gevent as gunicorn's async worker to run a flask web app. Reverting to 1.1.0 resolves this issue.

[jamadden]

That looks pretty much the same as #549 to me.

How did you install gevent? Specifically, did you use a manylinux binary wheel from PyPI or did you build it from source? There are manylinux wheels posted for both wide and narrow builds on PyPI, and it looks like you got the narrow build but needed the wide build. Perhaps your pip needs upgrading to get the right wheel (you may also need to clear your local pip cache).

A workaround might be to compile from scratch using the --no-binary argument of pip.

[woozyking]

We did pip install gevent -U --no-cache-dir to update gevent, bypassing pip cache. I suppose it used the binary wheel from PyPi.

Output:

$ pip install gevent -U --no-cache-dir
Collecting gevent
  Downloading gevent-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 32.9MB/s 
Requirement already up-to-date: greenlet>=0.4.9 in ./venv/lib/python2.7/site-packages (from gevent)
Installing collected packages: gevent
  Found existing installation: gevent 1.1.0
    Uninstalling gevent-1.1.0:
      Successfully uninstalled gevent-1.1.0
Successfully installed gevent-1.1.1

pip version:

$ pip --version
pip 8.1.1 from /home/overlord/venv/local/lib/python2.7/site-packages (python 2.7)

Not sure whether mu wheel is the narrow or wide one.

[jamadden]

The 'mu' builds are the most commonly used builds; gevent 1.1.0 only uploaded 'mu' builds. These were built with the official pypa/manylinux docker image, which was updated between 1.1.0 and 1.1.1; in the 1.1.0 time frame it didn't include the 'm' version (hence why they're missing).

I'm not sure what to tell you about how to investigate further. My two recommendations at this point would be to use the --no-binary option to build from source (to fix your immediate problem), and to open up an issue with the manylinux people (and CC me) who should have a much better idea about how their official builds could result in a situation like this.

[woozyking]

Thank you @jamadden

I'm not entirely sure what to report on at manylinux repo, as I'm not familiar with the whole build process and this tool/platform.

[jamadden]

Ok, I opened pypa/manylinux#49

[woozyking]

@jamadden thank you again, very much!

[jamadden]

I've pulled the effected wheel from PyPI (since having linux wheels is a very new thing, I suspect most linux users are used to compiling gevent and so that shouldn't be a problem). The manylinux guys are working on toolchain improvements to help prevent this from happening again.