>=eventlet-0.20.0: 'import six' imports the bundled eventlet six version

[hwoarang]

Hi,

We are seeing a very strange problem in one of our CentOS7 boxes

When using eventlet-0.19.0 with 'six' then importing 'six' works a expected

python -c 'import eventlet; import six; print eventlet.__version__; print six.__version__'
0.19.0
1.10.0

But with >=eventlet-0.20.0 something strange is happening

python -c 'import eventlet; import six; print eventlet.__version__; print six.__version__'
0.20.0
1.8.0

python -c 'import eventlet; import six; print eventlet.__version__; print six.__version__'
0.20.1
1.8.0

This is really wrong since 'import six' shouldn't import the bundled six module from eventlet but use the system's 'six' version

[temoto]

python -V

[hwoarang]

~$ python -V
Python 2.7.5

[temoto]

Can you reproduce it on another machine, not CentOS7?

cd github.com/eventlet/eventlet
git checkout v0.19.0
python=python2.7 bash test-381
git checkout v0.20.1
python=python2.7 bash test-381

in theory, this script should test for this issue, but it doesn't trigger error for me.

#!/bin/bash
set -eux

: ${python:=}
package=${1-"-e ."}
venv=./venv-issue-381

echo "testing with pip install $package" >&2
rm -rf "$venv"
virtualenv --python=$python "$venv"
#$venv/bin/pip install -U pip setuptools
$venv/bin/pip install 'six==1.10.0' $package
six_version=$($venv/bin/python -c 'import eventlet ; import six ; print(six.__version__)')
test "$six_version" = "1.10.0"

[hwoarang]

No I can't reproduce it on another host. Other CentOS 7 hosts appear to behave correctly. However, the fact that 0.19.0 works OK on that box suggests that something subtle has changed between 0.19 and 0.20 and triggers this behavior.

[temoto]

Uhm, yeah, but those two facts together mean that some subtle change combined with something on that particular machine triggers the error. Can you try completely purging all files related to eventlet and six and try again? I stress the point of not relying on any package management smartness.

[hwoarang]

Yes I completely removed all eventlet and six files from the host and installed again. It still does not resolve the issue. I even removed the eventlet dependencies and nothing changed. I install both 'six' and 'eventlet' using 'pip' which shouldn't do any weird thing. If I execute this

python -c 'import eventlet; import sys; print sys.modules["six"]'
<module 'six' from '/usr/lib/python2.7/site-packages/eventlet/support/six.pyc'>

you see that for whatever reason the 'six' symbol is binded to the eventlet six module during the 'eventlet' import.

On the other hand, a regular 'import six' works fine

python -c 'import six; import sys; print sys.modules["six"]'
<module 'six' from '/usr/lib/python2.7/site-packages/six.pyc'>

Unfortunately I can't really explain or even understand where the root cause of this could be. But since 0.19 works I believe that looking at the differences between 0.19 and 0.20 is a good start.

[temoto]

Do you have same Python version on different machines?

[temoto]

And what about setuptools versions?

[hwoarang]

Yes it's same python and setuptools (34.0.2)

[hwoarang]

This might also be of interest which is printed when executing

python -vvv -c 'import eventlet'

...
# trying /usr/lib/python2.7/site-packages/ecdsa/six.so
# trying /usr/lib/python2.7/site-packages/ecdsa/sixmodule.so
# trying /usr/lib/python2.7/site-packages/ecdsa/six.py
# trying /usr/lib/python2.7/site-packages/ecdsa/six.pyc
# trying /usr/lib/python2.7/site-packages/eventlet/support/six.so
# trying /usr/lib/python2.7/site-packages/eventlet/support/sixmodule.so
# trying /usr/lib/python2.7/site-packages/eventlet/support/six.py
# /usr/lib/python2.7/site-packages/eventlet/support/six.pyc matches /usr/lib/python2.7/site-packages/eventlet/support/six.py
import six # precompiled from /usr/lib/python2.7/site-packages/eventlet/support/six.pyc
...

[temoto]

Please print(sys.path) on both healthy and problem machines.

[hwoarang]

Sure

broken:

python -c 'import sys; print sys.path'
['', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages']

working:

['', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages']

they look pretty similar

[hwoarang]

Bisection also let me to this as the first bad commit

52b09becacd23f384cf69ae37d70c893c43e3b13 is the first bad commit
commit 52b09becacd23f384cf69ae37d70c893c43e3b13
Author: Sergey Shepelev <temotor@gmail.com>
Date:   Sat Aug 20 19:40:22 2016 +0500

    bundle dnspython in support/; resolving is always green

    Fixes installation issue when older dnspython is present in system packages
    https://github.com/eventlet/eventlet/pull/341

[temoto]

Yeah I know that commit is the only thing that could broke it, but I thought it's crafted smart enough to not break anything.

Can you also install this thing site-packages/gtk-2.0 on healthy machine and try again?

[temoto]

it's not gonna hurt to add clickable reference to breaking commit and issue: 52b09be #341

[hwoarang]

Sadly installing pygtk2 didn't make any difference. The paths are now the same but nothing really changed

[temoto]

Thanks for all your input, I'm out of ideas right now what's going on.

Is it possible you share all python related files, so I try to recreate the conditions in virtual machine and try more thorough step-by-step debugging?

find /usr -xdev -print0 -path '/usr/*/python*' |tar --null -cf - -T - |xz -c0 >eventlet-issue-381-filesystem.tar.xz
Note it will copy any globally installed packages from maybe project you work on. You can share it via private torrent or upload to some cloud storage with link-only access.

Full filesystem image would be easier, but I think it's a reasonable guess that root cause is somewhere among /usr python files since you showed no use of virtualenvs.

[temoto]

Also please show pwd ; ls -lh . before running test script on broken machine.

[temoto]

Also, I know this is going insane road but please, try this version

pip install https://github.com/eventlet/eventlet/archive/wrong-six-381.zip

[hwoarang]

So the wrong-six package didn't fix it. Uploading all the python related files could be a bit problematic but I will see if I can do that.

[temoto]

By the way, if the real problem is eventlet's bundled six is too old for something else in your system, a temporary crutch workaround is to replace eventlet/support/six.py to whatever version you need.

[hwoarang]

Yes, I have already replaced the bundled files with the 1.10 ones. I suspect the problem will be "masked" once you make a new release which will include 1.10 anyway

[rnurgaliyev]

I'm sorry for reopening old and closed thread, but could someone please elaborate, how #479 actually resolves this issue? We still have this problem and cherrypicking this commit doesn't really help.

[temoto]

@rnurgaliyev

• eventlet import failure #479 is an issue not PR
• 81c7cf9 fixes problem with dnspython
• 53b63c6 fixes issue with six by making it external dependency, not bundled in eventlet code