Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTPretty breaking other URLs #65

Closed
spulec opened this issue May 24, 2013 · 28 comments
Closed

HTTPretty breaking other URLs #65

spulec opened this issue May 24, 2013 · 28 comments

Comments

@spulec
Copy link
Contributor

spulec commented May 24, 2013

With the following test

@httprettified
def test_httpretty_should_not_override_other_urls():
    u"HTTPretty shouldn't break other URLs"

    HTTPretty.register_uri(HTTPretty.GET, "http://github.com/foo",
                           body="<root><baz /</root>")

    response = requests.get('http://google.com/')
    expect(response.status_code).to.equal(200)

I get

  File "HTTPretty/tests/functional/test_requests.py", line 153, in test_httpretty_should_not_override_other_urls
    response = requests.get('http://google.com/')
  File "HTTPretty/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "HTTPretty/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "HTTPretty/lib/python2.7/site-packages/requests/sessions.py", line 289, in request
    history = [r for r in gen] if allow_redirects else []
  File "HTTPretty/lib/python2.7/site-packages/requests/sessions.py", line 133, in resolve_redirects
    proxies=proxies
  File "HTTPretty/lib/python2.7/site-packages/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "HTTPretty/lib/python2.7/site-packages/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "HTTPretty/lib/python2.7/site-packages/requests/adapters.py", line 222, in send
    r.content
  File "HTTPretty/lib/python2.7/site-packages/requests/models.py", line 550, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "HTTPretty/lib/python2.7/site-packages/requests/models.py", line 496, in generate
    chunk = self.raw.read(chunk_size)
  File "HTTPretty/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 148, in read
    return self._fp.read(amt)
  File "python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "python2.7/httplib.py", line 592, in _read_chunked
    value.append(self._safe_read(amt))
  File "python2.7/httplib.py", line 649, in _safe_read
    raise IncompleteRead(''.join(s), amt)
IncompleteRead: IncompleteRead(532 bytes read)
@spulec
Copy link
Contributor Author

spulec commented Jul 12, 2013

This appears to be caused by a 301 redirect. If I change the code to

response = requests.get('http://google.com/', allow_redirects=False)

it doesn't fail.

@kouk
Copy link
Contributor

kouk commented Sep 26, 2013

I have this problem too. In my case the problem is not solved with disabling redirects. The root cause is that HTTPretty does not understand EAGAIN errors. The problem is in httpretty/core.py

        _d = True
        while _d:
            try:
                _d = self.truesock.recv(16)
                self.truesock.settimeout(0.0)
                self.fd.write(_d)
            except socket.error:
                break

the recv function gets EAGAIN if for some reason the web server is slow to respond, if it is using HTTP keep-alive and/or chunked transfer encoding. In my case one solution seems to be:

  1. patch the above code to continue on EAGAIN
  2. disable keep alive in my client's request headers (Connection: close)

The real solution would require HTTPretty not loading the real server's response in one go.

@andresriancho
Copy link
Contributor

Found this issue too, it's possible to reproduce using:

import requests
import httpretty

httpretty.enable()
httpretty.register_uri(httpretty.GET, "http://yipit.com/",
                       body="Find the best daily deals")

print requests.get('http://www.bing.com/')

That code will trigger the IncompleteRead: IncompleteRead(532 bytes read) error.

@andresriancho
Copy link
Contributor

I downloaded the latest httpretty from the repository (currently at 4691786) and tried the same example posted in the previous comment. Pull request #102 generates 100% CPU usage.

Haven't tested with "Connection: close", but even if that works, I can't modify the source code I'm testing.

@andresriancho
Copy link
Contributor

Experimented with select.select for fixing the issue andresriancho@f6e59f4 and it seems to work, but some tests don't pass.

I assume that the tests that aren't passing are because:

  • I'm reading byte-by-byte: recv(1) instead of using the 16byte buffer
  • select.select has different requirements over the input you send to it, and thus some mocks used during testing fail

The code with select.select passes the simple test I sent before.

@andresriancho
Copy link
Contributor

I don't really have more time to spend on this, if someone else (@gabrielfalcao?) wants to take a look at the tests, that would be awesome. Also, please decide a value for SELECT_TIMEOUT.

@andresriancho
Copy link
Contributor

ping @gabrielfalcao

@simon-weber
Copy link

I'm running into this as well (on 0.8.0):

>>> import httpretty
>>> import requests
>>> assert requests.get('http://www.google.com').status_code == 200
>>> httpretty.enable()
>>> assert requests.get('http://www.google.com').status_code == 200
^CTraceback (most recent call last):
  <snip>
  File "/usr/lib/python2.7/httplib.py", line 790, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python2.7/dist-packages/httpretty/core.py", line 393, in sendall
    self.real_sendall(data)
  File "/usr/local/lib/python2.7/dist-packages/httpretty/core.py", line 336, in real_sendall
    received = self.truesock.recv(self._bufsize)
KeyboardInterrupt
>>> assert requests.get('http://www.google.com', allow_redirects=False).status_code == 200
^CTraceback (most recent call last):
  <same as above>

Has there been any progress since @andresriancho's changes? I can spend some time on this.

@simon-weber
Copy link

Since this was blocking (no pun intended) some work of mine, I decided to just jump in.

A few notes:

  • @andresriancho was right, the test code just needs to be updated. Here's an example: simon-weber@73f534a.
  • re: andresriancho@f6e59f4#commitcomment-4941346: I don't think the reasoning behind reading byte-by-byte is correct. The argument to recv is the maximum amount we want to read -- if there's less than what we want, it'll just return what it has. recv only blocks when there's no data, which is what we're using select to prevent.
  • 16 bytes seems like a pretty small buffer. I'd imagine a typical request returns at least 10s of kilobytes; bumping it up to something like 4096 might speed everyone's tests up =)

@simon-weber
Copy link

Two other things:

  • @andresriancho's code fixes http retrieval, but relies on the timeout to know when to stop receiving data.
  • it does not fix https retrieval:
python -c 'import requests; import httpretty; httpretty.enable(); requests.get("https://www.google.com")'
running select
try to receive
got nothing; bailing
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by <class 'httplib.BadStatusLine'>: '')

I'm kind of confused as to why real_sendall has to recv in the first place. Shouldn't it be possible to just delegate this work to whatever code is expecting a real socket from httpretty?

@simon-weber
Copy link

Oh, I see:

The received bytes are written in this socket's StringIO buffer so that HTTPretty can return it accordingly when necessary.

@andreagrandi
Copy link

I'm having exactly the same problem and I confirm that it doesn't work with https calls. Is there any other way to do this?
p.s: I can't use httmok because our library doesn't use requests.

@clintecker
Copy link

This is still happening and is a pretty big blocker. Is there any movement towards fixing this bug?

@gabrielfalcao
Copy link
Owner

This might have been fixed by @frankamp on #216

@porterjamesj
Copy link

I believe I'm still seeing this on 0.8.8. I've seen both the BadStatusLine error reported by @simon-weber above, as well as just hanging forever.

@GitFree
Copy link

GitFree commented May 25, 2015

seeing this in both 0.8.8 and 0.8.9.
log shows (Caused by <class 'httplib.BadStatusLine'>: '')

@anentropic
Copy link

I am getting Connection aborted.', BadStatusLine("''",) error on non-HTTPretty url when @httpretty.activate is active

when I run this:

    @httpretty.activate
    def test_user_subscribed_to_broadcast(self):
        httpretty.register_uri(httpretty.POST, 'https://broadcast.test/broadcast/v1/user',
                               body='Hello', status=201)

I get BadStatusLine("''",) error in a PUT request to http://localhost:9200/elasticsearch-index/user/1?refresh=true (via pyelasticsearch)

(it sounds a bit like #182 except I get it for a uri not registered with HTTPretty)

the request succeeds if I remove the decorator

if I drop into ipdb I see that I do have allow_net_connect=True

httpretty==0.8.10

@anentropic
Copy link

looks like same issue as #194 #199 #187 ...actually about half a dozen more, they've all been prematurely closed

@anentropic
Copy link

FWIW whatever they're doing differently... https://github.com/patrys/httmock works fine

@boazin
Copy link

boazin commented Sep 10, 2015

Still happening... getting a
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))
Even on http://localhost:8000

Related?
(It works if I wrap the needed URL in httpretty.enable() httpretty.disable() - but it's an ugly solution)

@gabrielfalcao
Copy link
Owner

@boazin is the @httpretty.activate the decorator not working ?

@boazin
Copy link

boazin commented Sep 10, 2015

Haven't tried the decorator. I have a boolean I need to check in a config to decide if I use the mock or the real URL, so I can't use the decorator - need to manually call .enable() .disable()

@m1schka
Copy link

m1schka commented Feb 9, 2016

I wouldn't consider this issue as closed (as of 0.8.14)
Similar to @anentropic, I get a BadStatusLine from inside the boto package even though I don't mock those requests (because I'm using fakes3 for testing).
I'm also using enable/disable manually, because I'm using it in test setup to activate a special api mock, which can occur at any given point in the tests.

@sybrenstuvel
Copy link

I can confirm this issue on both Python 2.7 and 3.5. This is my test script:

from __future__ import print_function

import requests
import httpretty

httpretty.enable()

print('Requesting Google.')
print(requests.get('http://www.google.com/'))
print('Done!')

On Python 2.7.6 it hangs on the requests.get(...) line:

Requesting Google.
^CTraceback (most recent call last):
  File "check_httpretty.py", line 9, in <module>
    print(requests.get('http://www.google.com/'))
  File "/home/sybren/.virtualenvs/pillar/local/lib/python2.7/site-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/home/sybren/.virtualenvs/pillar/local/lib/python2.7/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/sybren/.virtualenvs/pillar/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/sybren/.virtualenvs/pillar/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/home/sybren/.virtualenvs/pillar/local/lib/python2.7/site-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/home/sybren/.virtualenvs/pillar/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/home/sybren/.virtualenvs/pillar/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 805, in send
    self.sock.sendall(data)
  File "/home/sybren/.virtualenvs/pillar/local/lib/python2.7/site-packages/httpretty/core.py", line 456, in sendall
    self.real_sendall(data)
  File "/home/sybren/.virtualenvs/pillar/local/lib/python2.7/site-packages/httpretty/core.py", line 380, in real_sendall
    received = self.truesock.recv(self._bufsize)
KeyboardInterrupt

On Python 3.5.1 it produces:

Requesting Google.
Traceback (most recent call last):
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/python3.5/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/opt/python3.5/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/opt/python3.5/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/opt/python3.5/lib/python3.5/http/client.py", line 911, in _send_output
    self.send(msg)
  File "/opt/python3.5/lib/python3.5/http/client.py", line 885, in send
    self.sock.sendall(data)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/httpretty/core.py", line 456, in sendall
    self.real_sendall(data)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/httpretty/core.py", line 372, in real_sendall
    self.truesock.connect(self._address)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 609, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 247, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/packages/urllib3/packages/six.py", line 309, in reraise
    raise value.with_traceback(tb)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/python3.5/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/opt/python3.5/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/opt/python3.5/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/opt/python3.5/lib/python3.5/http/client.py", line 911, in _send_output
    self.send(msg)
  File "/opt/python3.5/lib/python3.5/http/client.py", line 885, in send
    self.sock.sendall(data)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/httpretty/core.py", line 456, in sendall
    self.real_sendall(data)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/httpretty/core.py", line 372, in real_sendall
    self.truesock.connect(self._address)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError(9, 'Bad file descriptor'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "check_httpretty.py", line 9, in <module>
    print(requests.get('http://www.google.com/'))
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/home/sybren/.virtualenvs/pillar-python-sdk/lib/python3.5/site-packages/requests/adapters.py", line 426, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', OSError(9, 'Bad file descriptor'))

This is all on Ubuntu 14.04, using httpretty 0.8.14.

@m1schka
Copy link

m1schka commented Mar 29, 2016

I went back to using https://github.com/getsentry/responses, which is doing a better job at what I need

Robpol86 added a commit to Robpol86/appveyor-artifacts that referenced this issue May 1, 2016
Using socket directly instead of httpretty to test timeouts, thanks to
gabrielfalcao/HTTPretty#65.

Handling ConnectionError errors.

Pinning lint versions to avoid broken tests caused by bad dependenceis.
@david-caro
Copy link

I'm having this same issue on fedora 25, with python 2.7.13, requests 2.13.1 and httpretty>0.6.4 (tried all the versions to 0.8.4, and since 0.6.4 they just hang). My test

python -c "import httpretty;import requests;httpretty.enable();print requests.get('http://google.com')"

So it seems that a change introduced on 0.6.5 broke it...

@david-caro
Copy link

Yep, definetly the EAGAIN 'fix' broke it for me: 1d6f94e (in the sense that right before that commit it still works, after it, it hangs and breaks)

david-caro added a commit to david-caro/harvesting-kit that referenced this issue Feb 9, 2017
See gabrielfalcao/HTTPretty#65

Signed-off-by: David Caro <david@dcaro.es>
@davidread
Copy link

Downgrading to httpretty 0.6.4 works for me on Linux, but I get a new error on OSX, related to the proxy.

mkvirtualenv -p python3.6 test6
pip install requests
pip install 'httpretty<0.6.5'
python -c "import httpretty;import requests;httpretty.enable();print(requests.get('http://google.com'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/dread/.virtualenvs/test6/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/dread/.virtualenvs/test6/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/dread/.virtualenvs/test6/lib/python3.6/site-packages/requests/sessions.py", line 499, in request
    prep.url, proxies, stream, verify, cert
  File "/Users/dread/.virtualenvs/test6/lib/python3.6/site-packages/requests/sessions.py", line 672, in merge_environment_settings
    env_proxies = get_environ_proxies(url, no_proxy=no_proxy)
  File "/Users/dread/.virtualenvs/test6/lib/python3.6/site-packages/requests/utils.py", line 692, in get_environ_proxies
    if should_bypass_proxies(url, no_proxy=no_proxy):
  File "/Users/dread/.virtualenvs/test6/lib/python3.6/site-packages/requests/utils.py", line 676, in should_bypass_proxies
    bypass = proxy_bypass(netloc)
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 2612, in proxy_bypass
    return proxy_bypass_macosx_sysconf(host)
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 2589, in proxy_bypass_macosx_sysconf
    return _proxy_bypass_macosx_sysconf(host, proxy_settings)
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 2563, in _proxy_bypass_macosx_sysconf
    hostIP = ip2num(hostIP)
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 2542, in ip2num
    parts = list(map(int, parts))
ValueError: invalid literal for int() with base 10: 'google'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests