reference implementation for concurrent requests w/ rate limits

[perrygeo]

From #13

an ops module that would effectively be a reference implementation for how to do things like concurrent requests that respect rate limits. OTOH maybe the CLI could be that reference implementation.

[ryanbaumann]

@perrygeo

Concurrent reuqests would be useful as a CLI for geocoder-testing.

I made this python script wrapping the Mapbox Python SDK to run large batches of queries using the Python Multiprocessing library. It should be compatible with both Python 2 (and 3 with some minor changes).

• Doesn't solve the rate limiting issue without monitoring each thread for request counts.

   # Multiprocess the API calls to the geocoder API
    THREADS = 4
    pool = Pool(THREADS)
    #Split the dataframe into chunks=#rows, to make one request per row in the dataframe.
    chunks = np.array_split(df, df.shape[0])  
    try:
        df_lst.append(
            pool.map(geocoder.forward(df['query']), chunks)
        )
    except KeyboardInterrupt:
        # Allow ^C to interrupt from any thread.
        sys.stdout.write('\033[0m')
        sys.stdout.write('User Interupt\n')
    pool.close()

[perrygeo]

Using a thread executor with Python 3.5 asyncio and a semaphore to rate limit would be a solid approach. See https://gist.github.com/perrygeo/c287c1b9370611b73b890e9df3e82a29#file-async_get-py-L79-L81 for an example.

Closing for now.