Python library for massive (but controlled) parallel HTTP calls using gevent and requests
Inspired in grequests, this library allows you to do massively parallel HTTP requests, but controlling both the limit of simultaneous connections per target host and globally.
The usage is simple. First, install the library using pip:
pip install ghrequests
Then, your script needs to create the requests that should be sent, send them all and read the responses after:
import ghrequests
urls = [
"https://google.com", "https://canonical.com", "https://yahoo.com",
"https://microsoft.com", "https://facebook.com", "https://python.org"
]
requests = [ghrequests.get(url) for url in urls]
ghrequests.request_all(requests, max_connections=10, max_per_domain=2)
responses = [i.response for i in requests]
By default, max_per_domain
refers to the urlparse(url).netloc
of your
requested URLs, but this can be customized to be anything you want.
To limit the requests per HTTP method, for example, you could subclass
ghrequests.Request
to do what you need:
import ghrequests
class MyRequest(ghrequests.Request):
@property
def domain(self):
return self.method
requests = [
MyRequest("GET", "http://google.com"),
MyRequest("POST", "https://some.url.com"),
MyRequest("HEAD", "https://bla.foo.com"),
# ... several other requests
]
ghrequests.request_all(requests, max_per_domain=2)
responses = [i.response for i in requests]
The above example should limit to have at most 2 POST
, 2 GET
, 2 HEAD
(...) simultaneous requests at a time.
ghrequests can actually be used just as a flow control, regardless of doing HTTP requests or not.
All you need to do is implementing your
Request subclass accordingly, overriding domain
property and run
method:
class MyRunner(ghrequests.Request):
"""This requester runs something on the database, and we want to limit
it per target database."""
def __init__(self, database, other_parameter):
super().__init__() # Don't forget to call super.
self.database = database
self.other_parameter = other_parameter
@property
def domain(self):
# Target domain is the database where we will run the query.
return self.database
def run(self):
# Fetch something from the database, for example.
value = self.database.select("SELECT count FROM table LIMIT 1")[0]
return value * self.other_parameter
requests = [
MyRunner(db1, 5),
MyRunner(db1, 15),
MyRunner(db1, 25),
MyRunner(db1, 35),
MyRunner(db2, 5),
MyRunner(db2, 15),
MyRunner(db2, 25),
MyRunner(db2, 35),
# ...
]
# Send at most 2 request per target database:
ghrequests.request_all(requests, max_per_domain=2)
responses = [i.response for i in requests]