Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gevent + requests blocks when using socks #3861

Closed
xsren opened this issue Feb 10, 2017 · 3 comments
Closed

gevent + requests blocks when using socks #3861

xsren opened this issue Feb 10, 2017 · 3 comments

Comments

@xsren
Copy link

@xsren xsren commented Feb 10, 2017

I use python(2.7.6),gevnet(1.1.2),requests(2.11.1) to make http requests concurrent, and it works good. But when I add socks proxy to requests, it blocks.

This is my code:

import time
import requests
import logging
import click

import gevent
from gevent import monkey
monkey.patch_all()

FORMAT = '%(asctime)-15s %(message)s'
logging.basicConfig(format=FORMAT)
logger = logging.getLogger('test')

#socks proxy
user = MY_SOCKS_PROXY_USERNAME
password = MY_SOCKS_PROXY_PASSWORD
host = MY_SOCKS_PROXY_HOST
port = MY_SOCKS_PROXY_PORT
proxies = {
        'http': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
        'https': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
}
url = 'https://www.youtube.com/user/NBA'


def fetch_url(i,with_proxy):
    while True:
        logger.warning('thread %s fetch url'%i)
        try:
            if with_proxy:
                res = requests.get(url,proxies=proxies, timeout=5)
            else:
                res = requests.get(url, timeout=5)
        except Exception as e:
            logger.error(str(e))
            continue

        logger.warning(res.status_code)

def do_other_thing():
    while True:
        logger.warning('do other thing...')
        time.sleep(1)

@click.command()
@click.option('--with_proxy/--without_proxy',help='if use proxy', default=True)
def run(with_proxy):
    if with_proxy:
        logger.warning('with proxy......')
    else:
        logger.warning('without proxy......')
    ts = []
    ts.append(gevent.spawn(do_other_thing))
    for i in xrange(3):
        ts.append(gevent.spawn(fetch_url,i,with_proxy))
    gevent.joinall(ts)

if __name__=='__main__':
    run()

These pictures show the result.

run with proxy
run without proxy

With proxy, do_other_thing will blocks before fetch_url done.
Without proxy, it works good. (timeout error occurs, because of the GFW)

Can anyone help me solving this problem? Thanks very much!

@Lukasa

This comment has been minimized.

Copy link
Member

@Lukasa Lukasa commented Feb 10, 2017

My best guess is that you are monkeypatching too late. When using gevent you must monkeypatch first, before importing anything else. Doing it later runs the risk of breaking anything that has done a from import from the things that gevent monkeypatches.

Can you try moving your import and monkeypatch to the very top of your file before doing anything else?

@xsren

This comment has been minimized.

Copy link
Author

@xsren xsren commented Feb 10, 2017

Yeah, ' moving your import and monkeypatch to the very top of your file ' solved my problem! Thank you very much!

This suggestion also solves my project's problem.
I hava 3 files called A,B,C, and run sequence is A->B->C. I use gevent in B, so I add monkeypatch in B, but i blocks with requests+socks. Now I add monkeypatch at the very top of A, it works great now! Hope this will help others!

Thank you again!

@Lukasa

This comment has been minimized.

Copy link
Member

@Lukasa Lukasa commented Feb 10, 2017

Happy to help!

@Lukasa Lukasa closed this Feb 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.