Request Timing

[michaeldnelson]

It would be great if there was a way to get the response time for each individual request. I would like to make concurrent requests for optimization purposes but report on individual request latency. Perhaps there is a way to do this I am unaware of. Great library. Thanks!

edit: Actually I think I might be able to do this with background callbacks. Will test and report back.

[ross]

something like the following should do the trick for something quick and dirty.

#!/usr/bin/env python

from datetime import datetime
from requests_futures.sessions import FuturesSession

session = FuturesSession()
times = {}

def time_it(sess, resp):
    times[resp.url] = datetime.now() - times[resp.url]

def do_request(url):
    times[url] = datetime.now()
    return session.get(url, background_callback=time_it)

futures = []
for url in ('http://httpbin.org/get', 'http://httpbin.org/get?key=val',
            'http://httpbin.org/get?key2=val2'):
    futures.append(do_request(url))
for future in futures:
    future.result() # you'd want to actually do something with the result here
# now that we've waited for everything times should be the durations
for url, delta in times.items():
    dur = delta.days * 24 * 60 * 60 * 1000 + delta.seconds * 1000 + \
        delta.microseconds / 1000.0
    print '%4dms %s' % (dur, url)

if i were doing it in something real/perm i'd probably inherit from FuturesSession and build/add the timing logic in to the request method. something like.

#!/usr/bin/env python

from datetime import datetime
from requests_futures.sessions import FuturesSession

class TimingSession(FuturesSession):

    def __init__(self, *args, **kwargs):
        super(TimingSession, self).__init__(*args, **kwargs)
        self.timing = {}

    def request(self, method, url, *args, **kwargs):
        background_callback = kwargs.pop('background_callback', None)

        # start counting
        self.timing[url] = datetime.now()

        def time_it(sess, resp):
            # here if you want to time the server stuff only
            self.timing[resp.url] = datetime.now() - self.timing[resp.url]
            if background_callback:
                background_callback(sess, resp)
            # here if you want to include any time in the callback

        return super(TimingSession, self).request(method, url, *args,
                                                  background_callback=time_it,
                                                  **kwargs)


session = TimingSession()

futures = []
for url in ('http://httpbin.org/get', 'http://httpbin.org/get?key=val',
            'http://httpbin.org/get?key2=val2'):
    futures.append(session.get(url))
for future in futures:
    future.result()
for url, delta in session.timing.items():
    dur = delta.days * 24 * 60 * 60 * 1000 + delta.seconds * 1000 + \
        delta.microseconds / 1000.0
    print '%4dms %s' % (dur, url)

[michaeldnelson]

Wow! Thanks for the quick detailed answer. This will really help me out.

[ross]

np