Issues when importing requests with import_patched after importing normally

[graingert]

Not sure if this is requests fault or eventlets

graingert@MS-7681-ubu:~$ python
Python 2.7.3 (default, Sep 26 2012, 21:51:14) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> import eventlet
>>> eventlet.import_patched("requests")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/eventlet/patcher.py", line 110, in import_patched
    *additional_modules + tuple(kw_additional_modules.items()))
  File "/usr/local/lib/python2.7/dist-packages/eventlet/patcher.py", line 84, in inject
    module = __import__(module_name, {}, {}, module_name.split('.')[:-1])
  File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 52, in <module>
    from . import utils
ImportError: cannot import name utils
>>> import urllib2
>>> eventlet.import_patched("urllib2")
<module 'urllib2' from '/usr/lib/python2.7/urllib2.pyc'>
>>> 
graingert@MS-7681-ubu:~$ python
Python 2.7.3 (default, Sep 26 2012, 21:51:14) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import eventlet
>>> eventlet.import_patched("requests")
<module 'requests' from '/usr/local/lib/python2.7/dist-packages/requests/__init__.pyc'>
>>> 

[pjwerneck]

Had the same issue. Temporarily fixed it by converting the relative "from . import utils" in requests/init.py to "import utils".

[temoto]

@pjwerneck Beware that this fix would break importing a "real" package named utils from sys.path.

[pjwerneck]

Indded, that's why I'm not sending a pull request for it, but I can't wait for a proper solution from eventlet now.

[dbrgn]

Same issue here.

[temoto]

Yup, there's no doubt the bug exists. It's just entirely unclear what's the cause and, consequently, how to fix it.

[flub]

Not having checked this but it might be worth looking at replacing the __import__() call by importlib.import_module() to see if that helps the relative import.

Note that there is more wrong with the patcher, it would still be buggy if the above fixes this issue. Problem is that the patcher does not save/restore enough state when importing packages rather then modules and you very easily end up with mixed packages, e.g. after import_patched you'd both the patched and unpatched requests module would have the same unpatched requests.utils module. I think the proper solution is to re-write eventlet.patcher to keep a complete separate patched environment instead of mixing it into the existing sys.modules.

[graingert]

What about using import hooks?
On Jun 10, 2013 4:33 PM, "Floris Bruynooghe" notifications@github.com
wrote:

Not having checked this but it might be worth looking at replacing the
import() call by importlib.import_module() to see if that helps the
relative import.

Note that there is more wrong with the patcher, it would still be buggy if
the above fixes this issue. Problem is that the patcher does not
save/restore enough state when importing packages rather then modules and
you very easily end up with mixed packages, e.g. after import_patched you'd
both the patched and unpatched requests module would have the same
unpatched requests.utils module. I think the proper solution is to
re-write eventlet.patcher to keep a complete separate patched environment
instead of mixing it into the existing sys.modules.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7#issuecomment-19206475
.

[queertypes]

I've investigated this issue further, and reached the following conclusion after reproducing the issue in a patcher unit test.

It appears that the reason patcher.import_patched explodes on requests is because it cannot handle module hoisting as requests does in requests.__init__ in combination with having requests loaded elsewhere in the application. See below:

# file: requests/__init__.py

from . import utils
from .models import Request, Response, PreparedRequest
from .api import request, get, head, post, patch, put, delete, options
from .sessions import session, Session
from .status_codes import codes
from .exceptions import (
    RequestException, Timeout, URLRequired,
    TooManyRedirects, HTTPError, ConnectionError
)

One work around that may work from the application side (I've not extensively tested it) is to do:

import eventlet
requests = eventlet.import_patched('requests.__init__')

However, that's still a workaround. The solution to the problem in general may be to have import_patched load the target module's __init__.py first, then load the module itself.

For the interested, here's a reproducing test case. I'm posting it because I may not be able to look into the issue further for some time:

# file: eventlet/tests/patcher_test.py
class ImportPatched(ProcessBase):
...
    def test_import_requests_patched(self):                                                                                                                   
        child_mod1 = """                                                                                                                                      
from __future__ import print_function                                                                                                                         
import requests                                                                                                                                               
print("requests", requests, requests.utils)                                                                                                                   
"""                                                                                                                                                           
        child_mod2 = """                                                                                                                                      
from __future__ import print_function                                                                                                                         
from eventlet import patcher                                                                                                                                  
requests = patcher.import_patched('requests.__init__')                                                                                                        
print("requests", requests, requests.utils)                                                                                                                   
"""                                                                                                                                                           
        parent_mod = """                                                                                                                                      
import child1                                                                                                                                                 
import child2                                                                                                                                                 
print("requests", child2.requests, child2.requests.utils)                                                                                                     
"""                                                                                                                                                           
        self.write_to_tempfile("child1", child_mod1)                                                                                                          
        self.write_to_tempfile("child2", child_mod2)                                                                                                          
        self.write_to_tempfile("parent", parent_mod)                                                                                                          
        output, lines = self.launch_subprocess('parent.py')                                                                                                   
        print output, lines                                                                                                                                   
        for line in lines:                                                                                                                                    
            self.assert_(not 'ImportError' in line, repr(line))

Thanks!

[temoto]

Thank you very much @cabrera ! Test case worth a thousand words.

[GitHubGeek]

The line requests = eventlet.import_patched('requests') works fine for me. requests 2.0.0 in python 2.7.5.

Tested by modifying the example Producer Consumer Web Crawler to use requests.Session()

Correction: Actually the line works only when I %cpaste it inside IPython.

[frexvahi]

I got another error which is probably related to this one - here is a code sample:

## __init__.py
import eventlet
eventlet.monkey_patch()

## some module that gets imported first
requests_in_other_module = eventlet.import_patched('requests')

## Library code (simplified from https://github.com/influxdata/influxdb-python/blob/master/influxdb/client.py#L270)
import requests
import requests.exceptions

try:
    requests.get('http://localhost:1')  # raises ConnectionError
except requests.exceptions.ConnectionError as e:
    print('Got ConnectionError:', e.args)

Somehow the import requests.exceptions does not add the requests.exceptions module as an attribute of the requests module:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=1): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2dba802e8>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "simple.py", line 13, in <module>
    except requests.exceptions.ConnectionError as e:
AttributeError: module 'requests' has no attribute 'exceptions'

[sang-kevin]

I got another error which is probably related to this one - here is a code sample:

## __init__.py
import eventlet
eventlet.monkey_patch()

## some module that gets imported first
requests_in_other_module = eventlet.import_patched('requests')

## Library code (simplified from https://github.com/influxdata/influxdb-python/blob/master/influxdb/client.py#L270)
import requests
import requests.exceptions

try:
    requests.get('http://localhost:1')  # raises ConnectionError
except requests.exceptions.ConnectionError as e:
    print('Got ConnectionError:', e.args)

Somehow the import requests.exceptions does not add the requests.exceptions module as an attribute of the requests module:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=1): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2dba802e8>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "simple.py", line 13, in <module>
    except requests.exceptions.ConnectionError as e:
AttributeError: module 'requests' has no attribute 'exceptions'

I also met this problem, have you solved it?