Skip to content

Commit

Permalink
Merge pull request #420 from geopy/v2/add-async-ratelimiter
Browse files Browse the repository at this point in the history
geopy 2.0: RateLimiter: add thread-safety + add AsyncRateLimiter
  • Loading branch information
KostyaEsmukov committed Jun 21, 2020
2 parents 83b371f + b78c377 commit 884206b
Show file tree
Hide file tree
Showing 6 changed files with 522 additions and 175 deletions.
2 changes: 2 additions & 0 deletions docs/changelog_2xx.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ New features
- Added optional asyncio support in all geocoders via
:class:`.AioHTTPAdapter`, see the new :ref:`Async Mode <async_mode>`
doc section.
- :class:`.AsyncRateLimiter` -- an async counterpart of :class:`.RateLimiter`.
- :class:`.RateLimiter` is now thread-safe.

Packaging changes
~~~~~~~~~~~~~~~~~
Expand Down
16 changes: 13 additions & 3 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -55,33 +55,43 @@ Default Options Object
Usage with Pandas
-----------------

It's possible to geocode a pandas DataFrame with geopy, however,
It is possible to geocode a pandas DataFrame with geopy, however,
rate-limiting must be taken into account.

A large number of DataFrame rows might produce a significant amount of
geocoding requests to a Geocoding service, which might be throttled
by the service (e.g. by returning `Too Many Requests` 429 HTTP error
or timing out).

:class:`geopy.extra.rate_limiter.RateLimiter` class provides a convenient
:mod:`geopy.extra.rate_limiter` classes provide a convenient
wrapper, which can be used to automatically add delays between geocoding
calls to reduce the load on the Geocoding service. Also it can retry
failed requests and swallow errors for individual rows.

If you're having the `Too Many Requests` error, you may try the following:

- Use :class:`geopy.extra.rate_limiter.RateLimiter` with non-zero
- Use :mod:`geopy.extra.rate_limiter` with non-zero
``min_delay_seconds``.
- Try a different Geocoding service (please consult with their ToS first,
as some services prohibit bulk geocoding).
- Take a paid plan on the chosen Geocoding service, which provides
higher quota.
- Provision your own local copy of the Geocoding service (such as Nominatim).

Rate Limiter
++++++++++++

.. automodule:: geopy.extra.rate_limiter
:members: __doc__

.. autoclass:: geopy.extra.rate_limiter.RateLimiter

.. automethod:: __init__

.. autoclass:: geopy.extra.rate_limiter.AsyncRateLimiter

.. automethod:: __init__

AlgoliaPlaces
-------------

Expand Down
2 changes: 2 additions & 0 deletions geopy/extra/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Extra modules are intentionally not exported here, to avoid
# them being always imported even when they are not needed.

0 comments on commit 884206b

Please sign in to comment.